The hosting industry is undergoing a dramatic transformation due to increasing demand for self-service cloud based capabilities. To greet this shift, Static.com, a subsidiary of Catalog.com, is leveraging 19 years of hosting experience, and adopting Cloud Foundry as the foundation for its Platform as a Service (PaaS). This move demonstrates the company’s commitment to providing modern, easily-installed and managed cloud-based solutions to hosting companies and enterprises.

To address the demand for self-service cloud hosting, Static has designed the Static Cloud Manager, a robust yet easy-to-use interface. Static’s Cloud Manager leverages Cloud Foundry to allow end users and enterprises to configure, deploy and scale applications and services in seconds. Additionally, Static has expanded on Cloud Foundry’s core platform and developed a growing catalog of OSS applications, frameworks and services for end users to choose from.

With this growing array of options comes a greater need for insight into application metrics and usage patterns. Static has integrated Apache Hadoop with Cloud Foundry as one of the first steps in what it sees as the cloud’s next major offering: Analytics as a Service (AaaS). Thanks to the power and flexibility of Cloud Foundry, these robust new offerings are rapidly evolving the cloud.

Static adds Apache Hadoop to Cloud Foundry

As PaaS and Infrastructure as a Service (IaaS) become the new hosting standard, having the ability to gain insight into usage and the performance of sites and cloud-based applications is essential. Static believes that the AaaS offering will allow users to make educated decisions about their sites and applications based on usage patterns. The first step to achieving this is to be able to store and crunch large amounts of big data into usable metrics, and Apache Hadoop fits this need perfectly.

Adding services like Apache Hadoop to Cloud Foundry allows Static’s users to easily add cost-effective big data solutions to address custom application needs. Adding Apache Hadoop as a pluggable service inside Cloud Foundry is the first step to automating AaaS to any site or application within the PaaS, enabling customizable metrics and immediate insight into usage patterns.

The Shift in Hosting

Traditional hosting offers minimal choices to the user, forcing them into shared pools where sites and applications are not guaranteed any dedicated resources. Shared hosting has worked for many users and hosting companies, but Static has seen an increase in demand from users who are switching to Virtual Private Servers (VPS) solely for the ability to have a dedicated set of resources and the freedom to choose what they run. This use case places the burden on the user to install, setup and administer the entire process, from start to finish.

Static’s goal was to rethink this approach and to allow for users to have the ability to choose:

• An expanding applications list ready to be installed with one click
• All the latest languages and version to choose from
• Latest frameworks ready to be use
• Full control over memory limits
• Ability to scale instances easily
• Choose from array of services
• Sync code between test and production environments
• Backup and rollback applications and services with ease

Static has leveraged Cloud Foundry to help make this a reality with the Static Cloud Manager, an easy-to-use but robust interface which interacts with Cloud Foundry. Users are able to 1-click create a Platform as a Service application with a wide array of frameworks and services while taking advantage of integrated hosting features such as domain registration, DNS, billing, support, Virtual Private Servers (VPS), and email.

The Cloud Network

Having choices does not stop at applications and services. Static is working to create a unified cloud network of datacenters and hosting companies that will:

• Run your application in multiple data centers for high availability
• Migrate applications seamlessly from one datacenter to another while maintaining uptime
• Choose your application and services based on data centers capable of meeting your needs

Static’s Commitment to Open Source Development

Static believes that open source software development drives innovation and leads to higher quality software. We are deeply committed to the open source ecosystem and being a leader in the Cloud Foundry community. Static is proud to contribute back to the Apache Hadoop project, as well as Cloud Foundry, through services such as snapshot functionality to the file system service, chef cookbooks, bug fixes and enhancements. These contributions can also be found on our GitHub account at http://github.com/staticcorp.

About the Author: Jake Farrell is the Chief Technical Officer for Static Corporation. He specializes in scalable distributed back end infrastructure systems. He is actively involved in numerous projects with the Apache Software Foundation and is the Project Management Committee Chair for the Apache Thrift project.

Looking at the Cloud Foundry project as a whole though… where should you start?

As you might imagine, there are a lot of moving parts in a PaaS. At Pivotal, we have teams working on the backend (the main runtime environment for Cloud Foundry, also known as ‘vcap’); the frontend (the tools and APIs for deploying apps, such as the ‘cf’ gem); services (database, message queues, and related resources); and Cloud Foundry BOSH, the installer and deployment tool, which has different plugins targeting infrastructures like AWS, OpenStack and vSphere. In the future, we would love to see even more Cloud Provider Interfaces (CPIs) for BOSH to allow Cloud Foundry to be deployed onto additional clouds.

One of the best ways to get involved with the community and the codebase is to start with the Github repositories and the mailing lists. If you are not sure where particular functionality resides within the project as a whole, and you’ve had a good look at Github already, then we have a number of mailing lists where you can ask for help. vcap-dev is for those working on the PaaS itself; bosh-users is for those who want to understand how to use BOSH to deploy Cloud Foundry; and bosh-dev is for the elite hackers (like Dr Nic Williams!) who want to help us make BOSH even more awesome. If you participate in the mailing lists you will also be able to keep up with what others are working on, and learn who some of the leading developers on the project are. If you want to get an idea of the general direction of development, you’ll find that in our roadmap.

Once you are up-and-running with Cloud Foundry, or if you know to which part of the project  you want to contribute, then consider the size of an initial contribution. If you are new to the project you might want to start with something small (some Open Source projects call these kind of small changes to smooth out any rough edges “paper cuts”). If you have something larger, we are always keen to work with you to understand how we can integrate your code. Before sending a pull request, take a look at our OSS contribution information in the project readme, which includes details on the Contributor License Agreement for individuals and corporations.

We thought it would be good to highlight a recent pull request which was a superb example of a major addition to the project. Recently we received a contribution via a detailed Github pull request, adding Oracle support to some of the major components inside Cloud Foundry: the User Account and Authentication service (UAA) and the Cloud Controller. Originally, these components were built to use Postgres as a backend, and more recently MySQL support was added. Since many enterprises already use Oracle as their main relational data store, it is great to give them this additional storage option.

Taking a look at the specific request for the Cloud Controller, we were really struck by a few things. The first one is that the pull request itself is very detailed, and walks through the various changes required in order to replace the database with Oracle.

This pull request is a lot to digest. If you want you can review the single feature PR that lead to this PR for some more concise context into some of the decisions I made in creating this PR. Here is a summary of the more notable changes I had to make and some context for why I made those changes:

This level of detail is not always required with every patch, of course, but for something this extensive it was very welcome. Another aspect of this patch is that was was made of up several individual changes (things like tweaks to the database schema, differences between Oracle datatypes and those used by other databases, etc), and they were broken down into separate commits which made it easier to consume. It also helped that this set of changes was extensively discussed on the vcap-dev mailing list before this larger pull request was submitted, so the engineering team were aware of some of the other impacts it would have.

Although the resulting pull request was detailed and clear, it has taken a few days to make all of the changes, as the core code was still being worked on. This was actually one of the rarer cases where we had to do more work to get the patches integrated – and that shows how willing we are to work with contributions. Smaller changes are extremely welcome too, and often much quicker to digest. Don’t be put off by this larger example – come and start hacking with us!

We hope this post highlights some of the things that help to make it smoother to contribute to the project. To recap:

1. review the contribution guidelines and complete the CLA;
2. if you want to submit a major code change, discuss it on the relevant mailing list beforehand;
3. think about how the pull request is structured, and submit changes in separate consumable commits (rather than as one monolithic change) where appropriate;
4. the patch should be well-tested, should include tests if possible, and should pass our Travis CI process;
5. the changes should be clearly documented in the request, especially when the updates are more extensive than correcting a few typos.

The past few months have seen the Cloud Foundry Open Source community grow hugely, and mailing list activity continues to be at an all-time high – so we’re really looking forward to continuing to work with other developers to make the platform even more awesome! Thanks for being on this journey with us.

Andy Piper @andypiper

Developer Advocate, Cloud Foundry

Guest Blog by Colin Humphreys, CloudCredo, Inc.

Comic Relief is a well known charity based in the UK which strives to create a just world free from poverty. They raise millions of pounds through two big fundraising campaigns – Red Nose Day and Sport Relief. The money raised is spent in the best possible way to tackle the root causes of poverty and social injustice. All the money raised by the public is spent by Comic Relief to help poor and disadvantaged people in the UK and the world’s poorest countries.

Seven Hours to Maximise Funding For The Entire Year

A year of planning for Red Nose Day 2013 resulted in a 6 week media campaign culminating in 7 hours of prime time television coverage on the 15th March. Their donations platform is required to process in the region of 600,000 transactions in this 7 hour period, handle in excess of 10,000 concurrent call centre operators, and handle peaks of 500 donations completing per second from web, mobile and call centers.

These requirements pose an operational nightmare. A fifty minute outage at peak traffic, allowed within an annual “four nines”(99.99%) uptime SLA, could result in 50% of the charity’s annual income being lost. Comic Relief require a platform delivered with no outages; there won’t be a second chance for another year. Feedback on the performance of the platform under high load from real users is only received on an annual basis. To demonstrate the difficulty of this challenge, JustGiving, one of the largest charity-orientated service providers, had a huge outage beginning early on Red Nose Day 2013.

For the previous eight years the donations applications have been delivered via a monolithic Java application and RDBMS backend, deployed and scaled with the help of twelve partners. This setup suffered from single points of failure, high recovery times from potential failures, and a lack of consistency between environments in the deployment pipeline. Comic Relief were also looking to remove their reliance on any single third-party supplier so as to not be limited in their technology choices.

Working Prototype Delivered in a Single Day on CloudFoundry.com

Moving from traditional development methodologies to a lean, agile approach delivered by a small team required a leap of faith from Comic Relief. Their fears were allayed by delivering a working prototype via CloudFoundry.com on the first day of application development. This was critical to receiving constant feedback from Comic Relief, and gave a starting point from which to iterate on the applications and the platform.

The software that drives the ability to take donations is a series of small micro-applications decoupled via queuing technologies. The service-layer application generates events that are used to mutate an eventually consistent view of the donations domain, enabling failure handling and fault tolerance across the distributed architecture. The applications were all developed to the platform contract represented by Cloud Foundry.

Vendor Neutrality and High Availability are Primary Architectural Goals

Developing the applications to the Cloud Foundry contract brought vendor neutrality to Comic Relief’s donations deployment. Wherever Cloud Foundry ran the donations suite could run too. This commoditised the infrastructure vendors and freed Comic Relief from vendor-related constraints, allowing infrastructure choices to be based on price, performance and efficiency – rather than forced by vendor lock-in.

Developing on the operational capabilities of Cloud Foundry played a major role in meeting Comic Relief’s requirements. The data services were enhanced with redundancy and replication, intelligent connection load-balancing was added to each instance, and a distributed global load-balancing/fault tolerance service added to recover quickly from single or multiple instance outages. The service was distributed across multiple AWS EC2 regions and zones, and also London-based VMware vCloud/vSphere IaaS provider, Carrenza (See Figure).

Service distributed across multiple regions and zones

Enhancing Cloud Foundry’s capabilities involved the same Continuous Delivery-based patterns employed to develop the donations applications. The platform pipeline was based on Jenkins orchestrating Cloud Foundry BOSH, with automated testing using Cucumber and RSpec, automated load testing using Grinder, and automated security testing with Zed Attack Proxy in tandem with a custom suite. The platform pipeline for Cloud Foundry then formed part of a larger integration pipeline for continuous delivery of the donations service as a whole to production-like environments. Frequent and regular deployments following extensive automated testing generated the confidence required to make any necessary changes to such a mission critical system.

Record Earnings With No Outage

So how did it actually perform on the night? Shortly after 01:30 GMT, comedian Russell Brand announced the money raised stood at £75,107,851 – passing the previous high of £74.3m. The new record was delivered by a platform that suffered absolutely no outage in receiving donations – the first time this has ever been achieved by a donations platform for Red Nose Day.

I’ve been delivering platforms for nearly fifteen years; I can’t recall ever being more excited about a new technology than I was on the day Cloud Foundry was released. I’m exceptionally proud to have taken the promise of Cloud Foundry and delivered real value in a mission-critical scenario.

About the Author: Colin Humphreys is CEO of CloudCredo, a team of people with extensive knowledge in running and customizing Cloud Foundry. He led the installation of the first Cloud Foundry to deliver SLA-driven production services, delivers tooling to the Cloud Foundry community, and is a regular conference speaker on PaaS-related topics. He also organizes the London PaaS User Group.

Normal BOSH Deploy Cycle

When you do a bosh deploy of a large scale distributed service such as Cloud Foundry to a cluster, BOSH does a number of things on your behalf:

1. Prepares deployment
2. Compiles packages – Calculates all packages and their dependencies that need to be compiled. It then begins compiling the packages and storing their output in the blobstore. The number of workers specified in the deployment configuration determines how many VMs can be created at once for compiling.
3. Prepares DNS
4. Creates bound missing VMs – Creates new VMs, deletes extra VMs.
5. Binds instance VMs
6. Prepares configuration
7. Updates/deletes jobs – Deletes unneeded instances, creates needed instances, updates existing instances if they are not already updated. This is the step where things get pushed live.

In step #2 BOSH spins up a number of worker VMs to compile your code packages, then installs your compiled code in one or more VMs called stemcells. Normally all of these VMs are running somewhere on an IaaS provider such as VMware vSphere or AWS EC2.

Using Nise BOSH to Speed Up the Feedback Cycle

This is great in production, but when iterating on a BOSH Package, spinning up multiple VMs for compilation and deployment makes for a slow feedback cycle and a lot of ssh’ing into servers. It can also be costly, as you may have to pay for development resources on an IaaS, e.g. on AWS. Nise BOSH is a great help when developing a BOSH package. A Package is a collection of source code along with a script that contains instruction how to compile it to binary format and install it, with optional dependencies on other pre-requisite packages. Without Internet access or an IaaS, Nise BOSH will compile the packages necessary to run the job and install them on the box that you’re on, saving you the time, expense and complexity of using remote resources on AWS or another IaaS. Let’s see how this works.

First, start on a machine that looks like your target Stemcell. A Stemcell is a VM template with an embedded BOSH Agent. The Stemcell used for Cloud Foundry is a standard Ubuntu distribution. For this you can use Vagrant, a handy tool to create easily reproducible environments on your local laptop. Install Vagrant and then in a terminal:

$ vagrant init lucid64 http://files.vagrantup.com/lucid64.box 
$ vagrant up
$ vagrant ssh

You should now be in an Ubuntu lucid64 environment on your local machine. Now install Nise BOSH –

$git clone git@github.com:nttlabs/nise_bosh.git
$cd nise_bosh
$bundle install

Next you need to get a BOSH Cloud Foundry release – a BOSH release is a collection of source code, configuration files and startup scripts used to run services, along with a version number that uniquely identifies the components.

$git clone git@github.com:cloudfoundry/cf-release.git 
$cd cf-release 
$git submodule sync 
$git submodule update --init --recursive

You can now modify the Cloud Foundry open source code in cf-release/src as your project needs require. Once you are ready to test your changes, you need to create the release, but first install the BOSH CLI -

$gem install bosh_cli 
$bosh create release

Before you can deploy your release you’ll need a BOSH manifest file we’ll call deploy.conf. An example of the contents for a Nise BOSH manifest can be found in the Nise BOSH doc.

Now run Nise BOSH from the nise_bosh directory to launch dea_next. Without Internet access or an IaaS, Nise BOSH will compile the packages necessary to run the job and install them on the box that you’re on. Then just run run-job start to start the jobs in /var/vcap/bosh/etc/monitrc locally on your box.

$sudo PATH=$PATH bundle exec ./bin/nise-bosh ~/cf-release ~/deploy.conf dea_next
$./bin/run-job start

Nise BOSH is all set up to help you play with dea_next all by itself, but eventually it might enable to you develop and run Cloud Foundry right on your local machine. We see the many community contributions like NTT Nise BOSH as proof of the power of an Open PaaS ecosystem to advance the project. Next time you’re working on a BOSH release, give it a try.

–Matthew Kocher and Vikram Rana, Cloud Foundry Team

As one of the co-founders of OpenCredo I have been actively working with Cloud Foundry since its early days. We were one of the first companies to launch a production application on Cloud Foundry with the help of our friends at Carrenza. More recently we have established CloudCredo to help both enterprise customers wishing to host Cloud Foundry on their own infrastructure and cloud service providers wishing to offer Platform as a Service. We helped Ospero to bring a Cloud Foundry-based PaaS running on vCloud Director to market to better serve their customers.

With the growing availability of public Cloud Foundry instances, we are seeing an increase in interest from enterprise customers in hybrid cloud as a way to extract maximum value from their existing infrastructure while enabling capacity on demand through expansion into public cloud. In this blog we will discuss why hybrid PaaS with Cloud Foundry is gaining interest, some of the ways people are using it today and how we hope to be using it in the future.

Why Cloud Foundry for Hybrid PaaS

The availability of a PaaS that seamlessly spans in-house data centers and various public cloud offerings makes hybrid cloud a compelling commercial proposition. For example, it serves to drive competition among public cloud providers. Extra capacity can be bought from the most cost competitive provider, or spread across a number of providers, as determined by the best commercial fit for the need. To date the majority of public PaaS offerings have been closed source and can’t be run within the enterprise data center. Combine this form of lock-in with some well-publicized outages and most large enterprises simply aren’t willing to consider betting everything on a single instance cloud provider. We see the open, extensible nature of Cloud Foundry and the increasing number of companies offering hosted Cloud Foundry instances as key to making enterprises comfortable with PaaS.

For us the number of companies offering or intending to offer a Cloud Foundry instance is reaching a critical mass, giving enterprises a competitive set of choices of where to run their applications. The Cloud Foundry Core initiative also means that customers can rely on public Cloud Foundry Core compatible instances to meet certain minimum requirements. Taken alongside improved support for running Cloud Foundry on public IaaS such as EC2, enterprises now have three real options for deploying cloud applications: 1) on a Cloud Foundry instance running in house on existing infrastructure, 2) on a Cloud Foundry instance running on large scale public IaaS such as EC2 or 3) on one of the increasing number of cloud providers offering Cloud Foundry as a service.

Being able to pick a company offering a PaaS hosted within national boundaries is already broadening the applicability of PaaS to regulated industries that have requirements on data sovereignty and how data is managed. There are still many other considerations around technology stacks and non-functional requirements that determine whether Cloud Foundry is suitable for a particular workload.

Simple Hybrid Cloud Use Cases

We are seeing significant interest in hybrid PaaS as a way of adding capacity in peak periods and increasing availability. Currently this only works out of the box for simpler cases where 1) there is no need to massively scale the persistence technology and 2) the application’s database does not need to be shared between services on either side of the public private cloud divide. Where enterprises employ SOA it can also be challenging to find a subset of the services that can be moved out to the public cloud. While services can span public and private cloud, this adds complexity, potential performance challenges and cost where data stored in the public cloud is separately billable.

The factors that determine the simplicity of an application for the purposes of a hybrid cloud scenario are primarily how coupled it is to the rest of the enterprise. This coupling can take the form of persisted data, messaging and application services. The simplest case is an application that has none of these forms of dependencies – with the increasing adoption of SOA it is increasingly rare to find these types of applications. When considering the potential use of hybrid cloud, the cost benefit versus the complexity due to connectedness to the rest of the enterprise will be one of the key trade-offs.

Unless for some reason you have a large number of island-like applications then batch processing applications can often provide a fertile ground in the early days of adopting a hybrid cloud deployment strategy. Batch applications often combine minimum data connectivity to the rest of the application, often running on a data snapshot, with providing cost benefit from scaling up just for the batch run.

Scalable Database Services

The open source Cloud Foundry project provides a number of open data technologies that can be configured to “scale up” as single instance services out of the box. We see incorporation of scale-out, multi-instance database services as key to widening the use cases that can be addressed by Cloud Foundry and are currently working to address this opportunity for our customers.

At CloudCredo we recently open sourced a simple Cassandra service integration for Cloud Foundry on GitHub. The initial implementation was targeted mainly at development and test usage since it was single instance and therefore didn’t provide any way to make the database scale out or highly available. We are now extending that out to a multi-node cluster and we intend that other services such as MongoDB will also follow.

Scalable Databases as a Service can be plugged in to Cloud Foundry very easily by leveraging the Service Broker. This pattern allows the cluster to be managed from outside of Cloud Foundry but for the ‘external’ cluster service to be exposed to clients as a running Cloud Foundry service. The benefits of this model include Cloud Foundry features such as central configuration management and automatic service binding from applications.

While horizontal scaling and clustering of open database services opens up new potential, what really excites us is the potential to use the multi-data center capabilities of persistence technologies in conjunction with hybrid PaaS. Cassandra and other open data technologies such as Riak make it possible to have a logical database which spans multiple data centers and therefore potentially both public and private cloud.

A PaaS which integrates multi-data center databases that are easy to deploy and provide data synchronization would make solving problems such as disaster recovery and transparent scaling of both database and application code much easier. Disaster recovery currently relies on expensive low latency RPO systems that require manual, offsite back-ups be taken to mirror production. A multi-node, multi-data center data service would have the potential to automatically mirror data across data centers as part of the online transaction processing of the application. Having effectively the same database available within both public and private PaaS would also simplify the process of scaling out.

Conclusion

The growing popularity and increasing number of ways to consume Cloud Foundry is making hybrid PaaS an appealing reality for many enterprises. Cloud Foundry provides the Service Broker to facilitate integration of new, clustered open data technologies and legacy clustered databases, easing enterprise adoption in cases where the database services that come as part of the platform are not sufficient. The combination of reliable open source technologies that can deal with databases and the increasing availability of Cloud Foundry is an exciting option for those trying to use PaaS within the enterprise. Being able to solve these sorts of traditionally expensive problems with a PaaS will make PaaS even more compelling for large enterprises.

About the Author: Jonas Partner is CEO of OpenCredo, a software consultancy and delivery firm with offices in the United Kingdom and North America. Recently, Jonas co-founded CloudCredo, a company focused on helping businesses get Cloud Foundry up and running on the infrastructure of their choice. Before establishing OpenCredo in 2009, Jonas was a consultant with Spring Source where he contributed to the Spring Integration project. Jonas maintains interests in Machine Learning and the scalability of highly concurrent enterprise systems.

In this guest blog, Wei-Min Lu, founder and CEO of Anchora, explains why Anchora selected Cloud Foundry as the basis for its MoPaaS platform and why it chose to join Cloud Foundry Core.

Today we are excited to join Cloud Foundry Core as proof of our commitment to cloud application portability across a choice of public and private clouds. Here in China, there is growing demand for an open PaaS that enables developers to build mobile applications and enterprises to build private or dedicated clouds. We built MoPaaS, a cloud application engine based on the Cloud Foundry service platform. Cloud Foundry gives us the flexibility to offer both a public cloud platform with extensions for mobile and data applications and a private/dedicated PaaS solution that integrates with enterprises’ existing infrastructure. We also provide services within Cloud Foundry applications allowing enterprises to take advantage of big data infrastructure for use cases like cloud storage and synchronization, analytics and search.

The Need for Open PaaS in China

It is projected that PaaS will grow at 30% annually over the next few years. Higher growth is expected to occur in China due to 1) a rapidly expanding mobile application market requiring an agile back-end services platform and 2) the need for cloud application portability and the rise of applications taking advantage of big data infrastructure in the enterprise.

China is now the world’s biggest smart phone market with 390 million units projected to be sold this year – more than 1/3 of smart phones to be sold globally in 2013. Businesses of all sizes are looking to take advantage of this trend, increasing the number of mobile apps by 35% annually. There is a growing need for an agile platform with additional services required for mobile applications, such as push notification, geospatial data storage and short message services. Due to the limited resources on the client device, mobile applications require a platform that is able to process large quantities of data in the cloud. A new generation of Open PaaS is needed to handle additional mobile-specific requirements.

With more than 70% of companies implementing PaaS over the next two years, we see emerging needs for PaaS solutions in the China enterprise IT market to consolidate existing applications and infrastructure while building an ecosystem of new enterprise applications in a consistent and unified manner. Enterprise application portability is increasingly becoming a critical need – businesses expect to be able to move their applications for optimal service, regulatory compliance, and data privacy requirements. As enterprises are realizing the impact and starting to build apps to take advantage of big data, there is an urgent need for more sophisticated platform services with big data infrastructure. This includes scalable services for log analysis, cloud storage and synchronization, information clustering, analytics, and customized search.

Building a Public PaaS Using Cloud Foundry

We built MoPaaS to serve developers’ needs to create applications and services without worrying about backend IT tasks. It helps them simplify and automate application lifecycle management, including development, deployment, and operation, significantly reducing IT infrastructure expenditures and devops costs and time. MoPaaS is built on an intelligent cloud service platform infrastructure that consists of two integrated parts:

• Service platform: extends the Cloud Foundry service platform to provide better support for mobile applications, platform monitoring and management, and service extensibility.
• Data platform: based on innovative information chain management (ICM) technology. It offers a simple interface for developers to support data driven applications.

Fig 1. MoPaaS Cloud Application Engine Architecture

MoPaaS provides necessary devops automation, masks complexities from developers and enhances usability by offering additional functions:

• UI: providing ease of use for developers of various levels and a CLI providing more flexibility and control for experienced developers. Developers can use the MoPaaS UI for devops tasks like application deployment and performing resource allocation.
• MDSS (mass data storage service): a high performance distributed storage system that provides cloud storage for user data and application source code management.
• Application monitoring and management: a streamlined visual tool and web console for monitoring and managing applications, services, and environment variables.
• MoPaaS Services: Mobile specific services, such as notifications, short messages, and locations etc. Data services, such as scalable services for log analysis, cloud storage and synchronization, information clustering, and customized search.

Building an Enterprise PaaS Solution Using Cloud Foundry

MoPaaS also provides an intelligent IaaS-PaaS solution for enterprises and service providers to build dedicated clouds to boost service agility, reduce IT expenditures and devops costs while increasing the portability, scalability and reliability of enterprise applications. This solution offers necessary granular controls and customer configuration flexibility provided by IaaS and the devops automation and ease of app creation provided by PaaS. The MoPaas architecture strictly separates PaaS from IaaS services.

Fig 2. MoPaaS Intelligent Cloud Platform

In addition to the services offered by Cloud Foundry, MoPaaS offers a comprehensive array of data services as native Cloud Foundry services and integrates various legacy services.

Fig 3. MoPaaS Services

MoPaaS effectively addresses a wide range of the enterprise cloud platform requirements, including:

• Application consolidation: there is a need for building and running an ecosystem of enterprise applications on a cloud-based platform in a consistent and unified manner. However, many of the applications in enterprises have been around for quite some time and were not designed for the cloud. Among the key requirements, legacy services used by those applications need to be integrated into the cloud platform or upgraded to new services of the cloud platform. MoPaaS addresses the application consolidation and migration issues effectively, as the MoPaaS platform not only supports basic services, but also adds new data services to meet enterprise big data requirements using Cloud Foundry service gateway and seamlessly integrates legacy services used by existing applications, such as Oracle Database, Microsoft SQL Server, and IBM DB2 using Cloud Foundry service broker (Figure 3).

• Data platform: Big data has increasingly become top of mind across organizations, from the data center to the boardroom. Companies are seeking systems and tools capable of tackling the challenges of massive data growth. Integrated with Hadoop-based Information Chain Management (ICM) data services, MoPaaS enables organizations to deploy a central platform to solve end-to-end data problems that involve any combination of information ingestion, processing, storage, exploration, analytics, and retrieval. With MoPaaS enterprise developers can easily integrate sophisticated big data services with their applications as native Cloud Foundry services. These include log analysis, cloud storage and synchronization, information clustering, and customized search. In addition, with ICM data services, MoPaaS provides new capability for monitoring and managing applications on the cloud platform.

• Application portability: MoPaas architecture strictly separates PaaS from IaaS services. MoPaaS furnishes applications with an infrastructure-agnostic execution method, standardizes the method of exposing services such as databases, messaging and queuing, runtime, and management, and standardizes service credentials across runtimes. Thus MoPaaS keeps the applications portable in case they need to move the PaaS layer and application services to another infrastructure provider. Additionally, MoPaaS ICM mechanism enables automatic application data transferring along with the associated applications via its data synchronization service. Developers don’t have to worry about infrastructure-specific requirements for their applications, because the MoPaaS platform takes care of these concerns for them.

Summary

The initial success of Cloud Foundry-based PaaS providers demonstrates the strength of the Cloud Foundry open PaaS solution and the power of its open source ecosystem and community. This is a win for everyone invested in the success of Cloud Foundry.

We are excited to become the first company in China to join the Cloud Foundry Core community. By being a member of the Core ecosystem, Anchora is committed to the open PaaS concept and delivering cloud portability to Chinese application developers and service providers.

About the Author: Wei-Min Lu is the founder and CEO of Anchora. Anchora provides both MoPaaS, a public open cloud platform with extensions for mobile and data applications, and a dedicated PaaS solution that facilitates application consolidation and cloud platform portability for enterprises. Dr. Lu has over 15 years experience in product development and marketing. Before he founded Anchora in 2008, he served in key engineering and management positions at IBM and NASA-JPL. His background is in cloud computing, cybernetics, machine learning, information retrieval and storage technologies. Dr. Lu holds a Ph.D. in Electrical Engineering/Math from CalTech and a B.Sc. from Tsinghua University, China.

An earlier release of the Cloud Foundry plugin (1.1.0) introduced tunneling support for data services, and the command-line application feature in 1.4.0 now extends the support by automatically resolving tunnel values when launching a command-line application for a Cloud Foundry service. In addition, a set of predefined commands for executing commonly used command-line applications are available by default, and ready to use if the external command-line application is installed and set in the operating system’s PATH environment variable.

These commands are launched from within Eclipse or STS, and executed in an external command-line terminal. New commands can be defined either through a new Eclipse preference page for Cloud Foundry, or by opening a Service Tunnel Commands wizard from the Services table in the Cloud Foundry Server editor. The commands are stored as Eclipse preferences and therefore persisted across Eclipse runtime sessions.

In this blog, we will describe the steps to define and launch a new mysqldump command that automatically generates a dump file for a Cloud Foundry MySQL service. This blog assumes the reader has basic understanding of how to create a Cloud Foundry server in Eclipse, as well as create services through the Cloud Foundry Server editor’s Applications tab.

The command-line application feature in 1.4.0 allows users to launch an external application by simply selecting a service in the Services table in the Cloud Foundry Server editor and right-clicking to execute a particular command from the Services table context menu (see Figure below). The launch process then automatically creates a tunnel if one was not already created and executes the command in the operating system’s default terminal application. Terminal applications can also be changed per command definition. For example, in Linux the default terminal used by the feature is xterm, yet a user can change a command to use a different terminal like gnome-terminal.

Predefined Commands

The Cloud Foundry Integration for Eclipse version 1.4.0 provides a list of predefined commands that are ready to use for MySQL, PostgreSQL, MongoDB, and Redis services, provided the corresponding applications are installed. The commands are accessible by right-clicking on a service in the Services table in the Cloud Foundry Server editor:

The list of commands are:

MySQL:

1. mysql
2. mysqldump

MongoDB:

1. mongo
2. mongodump
3. mongorestore

PostgreSQL:

1. psql

Redis:

1. redis-cli

If these applications are already set in the operating system’s PATH environment variable, these commands can be executed right away without changes, and are launched in an external terminal like Terminal.app for Mac OS X, cmd.exe for Windows, and xterm for Linux.

For example, launching mysql for a MySQL service in the Cloud Foundry editor will automatically connect the mysql application to the MySQL service in the Cloud Foundry server through a tunnel, provided mysql is installed and set in the OS PATH environment variable. The end result is an external terminal window opening outside of Eclipse, displaying the mysql application prompt:

Since the tunnel credentials are resolved automatically for the external application by the Cloud Foundry plugin, the mysql application automatically connects to the data service and opens the mysql prompt without asking the user for credentials.

If new commands need to be added, or existing ones edited, they can be done so by either selecting Command Definitions in the Services table context menu, or by opening Eclipse Preferences -> Cloud Foundry -> Service Tunnel External Application Commands

Creating a New mysqldump Command

In our example, we will be adding a new mysqldump command that creates a particular output dump file called myoutputfile.txt. Although a mysqldump command already exists by default as part of the set of predefined commands, we will create a separate one where a dump file is explicitly set.

Commands are associated per service vendor, so the new mysqldump command will apply to any MySQL service regardless of the Cloud Foundry server.

To add a new command, choose Command Definitions from the Services table context menu to open the Service Tunnel Commands wizard, select the desired service, and click “Add”.

This will open the Command Definition wizard:

Two required properties are needed to define a command:

1. A unique display name. Two or more commands with the same name cannot exist for a given service.
2. An executable location, or just the name of the application if already set in the OS PATH environment variable.

Optionally, a user can also define command options that are passed to the application when it is being executed, and environment variables that are set by the Cloud Foundry plugin prior to launching the command. Option variables using the pattern ${variablename} can be used in the options definition.

These variables are grouped into two types:

1. Tunnel-specific variables that are reserved for tunnel values like username, password, port, and database name. These variables are automatically resolved by the Cloud Foundry plugin when the command is launched.
2. Additional variables that will be prompted to the user in a dialogue when the command is executed.

Some applications like psql require that passwords be set as environment variables. As with command options, variables using the pattern ${variablename} can be used in place of environment variable values.

In addition, the terminal definition can also be adjusted for that particular command, and optionally applied to all current and future commands by editing the terminal launch command value in the External Command Line Terminal section.

To facilitate creating custom commands, the list of predefined commands are available via a drop-down control, and can be used as templates for new definitions:

Let’s select mysqldump from the list. It will now populate the wizard with the default definition for myslqdump, including the necessary options required to run the application. Since we already have a default mysqldump command, an error will appear asking the user to set a different display name:

Now, we want to edit the display name and give it a unique name, and also set a specific name for the dump file: myoutputfile.txt. In addition, we are changing the location of the mysql executable as we do not have it set in the OS PATH environment variable.

Clicking “Finish” in the Command Definition wizard will now add the command to the list of commands for the MySQL service:

Clicking “Finish” once more in the Service Tunnel Commands wizard will persist the changes. At this point, clicking “Cancel” will alternately cancel any changes made to any services.

Now let’s right-click on the MySQL service in the Services table, and we’ll see the newly added mysqldump with outputfile command in the context menu. Selecting this command will execute it, and if necessary, automatically create a tunnel connection if one wasn’t already present.

After the command executes, the following dump file is created in the user’s home directory:

If instead of executing the custom mysqldump with outputfile that was created above, a user runs the default mysqldump command where the dump file is defined using an ${output file} variable, a dialogue will open prompting the user for the name of the output file prior to executing the command. The file name can include an absolute path, or if just specifying a name, it will be created relative to the user’s home directory.

Variations in Terminal Launch

The command-line feature for 1.4.0 is supported in Mac OS X, Linux and Windows. However, there are differences in how a command is launched in an external terminal. For Mac OS X, the default terminal is Terminal.app, and the Cloud Foundry plugin executes the command and sets any environment variables in a temporary .sh script file that it creates and then passes as an argument to Terminal.app. This temporary file is deleted after the process running the command is launched.

For Windows and Linux, no temporary script file is created, and instead the command and its options are passed directly to the external process as process arguments.

Installing Cloud Foundry Integration for Eclipse 1.4.0

To install version 1.4.0, follow the documented instructions to install Cloud Foundry Integration for Eclipse. Earlier versions of the integration, starting from 1.0.0 can be updated from within Eclipse or STS. Note that updates for older Cloud Foundry plugins prior to 1.0.0 are not supported, and must first be uninstalled before installing 1.4.0.

Only Eclipse JEE Indigo and Juno, as well as STS 2.9.0 and GGTS 3.0.0 and higher are supported.

Our independent status is quite meaningful and gives us the ability to engage with our large ecosystem in new ways. Adding full-time external committers has always been a goal of the team, and we are engaged with several organizations about putting dedicated resources on the extended engineering team –we believe this to be a very important step forward. The scale of these external investments is significant and a major milestone in our growth. The heart of Cloud Foundry, however, really comes from individual community contributions and users, so of course, we invite you to join us. All you need to do is send a pull-request.

As more and more organizations contribute to Cloud Foundry a transparent roadmap to the community is paramount. In January we began publishing a quarterly plan as part of our community documentation and since then our mailing list activity has hit an all-time high. We will be prolific in communicating the updated roadmap and updates on new work including logging APIs, .NET support, Cloud Foundry BOSH enhancements and more.

Cloud independence and multi-cloud support are important beliefs for us and the broader Pivotal Initiative. Expect open interfaces and support for a variety of clouds, with continued development on AWS, Openstack, vCloud and vSphere environments.

Our own financial and human investment in the project continues to grow. Cloud Foundry is staffed with over forty full time internal engineers, augmented by an additional team of twenty five spread across PM, developer advocacy and community management.

Last week we began actively selling software update and support subscriptions to large customers, another important milestone. Enterprise interest in building large scale internal agile platforms with Cloud Foundry is tremendous and we expect an interesting new backlog of feature requests from these engagements. The only proprietary code in these subscriptions is our web console, so any new enhancements will immediately benefit the larger community.

Finally we have again weighed the best governance model for the project and believe the current corporate sponsored, Apache 2 licensed, pull request driven approach, remains the right one. Adding external committers will also enhance the diversity and strength of the team. That said, the massive growth of the community and ecosystem requires mediating a diverse set of needs and we will always be open to other governance models for the project in the future.

Our mission is to become the most popular platform for new applications and the services used to build them across public and private clouds. If your organization would like to join us in building something great please contact us here.

James Watters, Head of Product

The Cloud Foundry Team

In this guest blog, Mark Prichard, Senior Director of Product Management for CloudBees®, explains how to create a fully automated environment for application build, test and deployment to Cloud Foundry.com using Jenkins in the Cloud via the CloudBees DEV@cloud™ build service.

Spring Framework is the most widely adopted development model for Java-based enterprise applications, used by millions of developers for its powerful abstractions and declarative support for crucial application infrastructure concepts. Jenkins is the world’s number one open-source continuous integration (CI) platform, with deep roots in the Java community and 60,000 installations worldwide. These two technologies are part of the fabric that has made Java a highly productive, agile programming environment for business applications. With the rise of cloud computing and Platform as a Service (PaaS), developers want CI services to build, test and deploy their Spring applications entirely in the cloud, using elastic, on-demand resources. You can now use Jenkins via CloudBees DEV@cloud to continuously deploy to Cloud Foundry.com.

Usage of Continuous Integration and Delivery for Release Management

In our most recent annual Jenkins CI user survey, a couple of facts stood out: first, 74% of users are building Java applications with Jenkins (although we also see significant development in C/C++, Javascript, Python, C#, PHP, Ruby, Scala and Groovy); and second, we are seeing a really big up-tick in usage by large organizations, with 60% growth among the very largest group. Overall, the number of installations is up 66% in the last year, with over 83% of those surveyed using Jenkins for what they consider mission-critical development. Looking at those results, it seems very clear that many of those large, mission-critical applications are using Java with Spring Framework – with CI provided by Jenkins.

Another key trend that is growing in importance daily is Continuous Delivery. More and more organizations are looking to embrace an agile model in which stringent, automated testing allows enhancements or “micro-releases” to go live without the traditional waterfall release cycles. We are seeing a major shift in enterprise software development to cloud-based, continuous delivery, with fully automated quality, coverage, functional and performance tests gating live deployments. This is the new best practice, and it is now available for Cloud Foundry development.

In this blog post, I’m going to show you how easy it is to set up a CI job using Jenkins via CloudBees DEV@cloud service to automatically build, test and deploy a rich Spring Framework application to Cloud Foundry.com.

Using Jenkins in the Cloud to Continuously Deploy Spring Apps to Cloud Foundry

Here’s an overview of the process:

1. Link your Cloud Foundry and CloudBees accounts using OAuth 2.0 for secure and automatic deployment
2. Clone a Git repo and set up a Jenkins job to build automatically when changes are pushed (we’ve provided ClickStarts™ that automatically do these tasks for you)
3. Set up and configure Jenkins Cloud Foundry deployment plugin to push your application to Cloud Foundry if your build succeeds

That’s it: your application and its services are now running live on Cloud Foundry!

Let’s take a quick look at all this cool stuff. First, we need a way to allow secure access from your CloudBees DEV@cloud account to the Cloud Foundry deployment services. CloudBees and Cloud Foundry both support the industry-standard OAuth 2.0 protocol, allowing you to establish a trust relationship between your accounts without the need for either party to store account details like passwords from the other service. Go to cloudfoundry.cloudbees.com, which will redirect you to the Cloud Foundry log in. Log in and authorize your CloudBees account to deploy to your Cloud Foundry account.

Next we want to set up a build job. That’s really easy, even if you’ve never used Jenkins before: we’ve set up ClickStarts that will clone private Git repositories with a couple of fully-featured Cloud Foundry/SpringSource examples from GitHub (springmvc-hibernate and petclinic-grails) and then create Jenkins jobs for those builds. From the Cloud Foundry ClickStarts launch page all you need to do is click on the Spring icon, enter the name you want to use for your build and in a few seconds you’ll be taken to the Jenkins build job, which will start automatically as soon as the repository is available.

At this point, you have everything set up: a private Git repo to use for development, a Jenkins CI build job and automatic deployment to Cloud Foundry. The build job has been configured to use your CloudBees account name as part of the Deployment Hostname (e.g. springmvc-hibernate-< youraccount>.cloudfoundry.com), but of course you can change that if you like. You can now clone a local copy of the source code repository (click on the Repositories tab on the toolbar for full details) and you have a fully automated, cloud-based develop-build-test-deploy Continuous Delivery environment for your Spring/Grails applications. Every time you push an upstream commit, Jenkins will run a complete build/test and deploy to Cloud Foundry.

As you can see by browsing the console output for the build job, this is a substantial application that uses many aspects of Spring Framework, including Hibernate for the mapping between the Java application and two data services, Redis and PostgreSQL, that also need to be provisioned on Cloud Foundry.com and bound to the application. That is all done as part of the deployment; you can see the details in the Services section of the host service configuration.

Once the build and deployment are complete, you can go into the console output for the job and see all the details of the deployment. If you have vmc set up on your workstation, you can use vmc apps and vmc services to verify those deployments, like this:

All you have to do now is browse to those URLs and you’re up and running with Spring/Grails and Jenkins in the cloud – enjoy!

Summary

Cloud Foundry together with CloudBees Jenkins in the Cloud service give you a complete Continuous Deployment solution for your enterprise Spring Framework and Grails projects:

• You have access to all the capabilities of a fully managed Jenkins as a service – you don’t have to set up Jenkins or the build systems.
• You can use Jenkins to deploy your Java Spring applications seamlessly to Cloud Foundry.
• You always have enough build capacity — CloudBees dynamically adds more build machines as you need them.
• It’s free to get started.

Watch the video: Continuous Integration to Cloud Foundry.com Using Jenkins in the Cloud
Docs: https://developer.cloudbees.com/bin/view/DEV/DeployAppsToCloudFoundry
Give it a try: cloudfoundry.cloudbees.com

About the Author: Mark Prichard, Senior Director of Product Management for CloudBees, speaks and blogs regularly as an evangelist for Platform as a Service. He came to CloudBees after 13 years at BEA Systems and Oracle, where he was product manager for the WebLogic Platform. A graduate of St John’s College, Cambridge and the Cambridge University Computer Laboratory, Mark works in Los Altos, CA.

An introduction and details of the UAA from a developer perspective can be found in earlier blog posts. In this post I will focus on the UAA’s integration of three emerging identity standards and what that implies for some specific Cloud Foundry deployment patterns.

Goals

The goal of the UAA is to make it easier, simpler, more secure and more convenient for users to get to the features of Cloud Foundry. The value is in the open PaaS and the applications it enables. So the identity services provided by the UAA should not only be useful for cloudfoundry.com and applications on it, but for pure open source deployments, managed private clouds, and even the micro cloud running on a thumb drive.

To achieve this goal, we have implemented and integrated new standards in a way that supports various deployment scenarios, such as multiple identity providers and external authentication mechanisms.

Fusing Identity Standards for a Stronger Platform

We have woven identity services into the platform using three standard protocols: OAuth2 for delegated authorization, OpenID Connect for session and authentication information, SCIM for user and group management. Of course we provide support for other standards as well, such as JSON Web tokens (JWT), SAML2, and OpenID 2.0. Even support for the legacy identity stalwart LDAP is in process. But what is particularly interesting is how we have combined the three primary protocols.

1. OAuth2 — this is the foundational protocol and provides delegated authorization. It provides a way for web applications to ask users to allow access to cloudfoundry on the user’s behalf. It provides authentication and SSO as a side effect. The app gets one or more secure tokens — some that the app can use to identify the user, and some that it can present to the cloudfoundry API to act on behalf of the user. But OAuth2 does not specify the token format. We chose another standard, JWT, for that.
2. OpenID Connect — provides identity information for the authenticated user. OAuth2 does not specify how the user’s information can be obtained by an app, just an opaque token. OpenID connect adds the ability for an app to efficiently and safely get authenticated user attributes like user name or email address.
3. SCIM — User and group provisioning. User account management is out of scope for OAuth2, but SCIM specifies simple user provisioning as well as groups, and it’s extensible to new types. Since OAuth2 requires that web applications preregister with the authorization server (UAA), but does not specify how these registrations are done, we use a SCIM extension for that.

Imagine two web applications: – one manages users and groups for access to cloudfoundry.com – one displays results of monitors a user’s running applications as a service — like our partner Appsecute provides.

You could diagram the interactions of the apps and three protocols like this:

The three protocols complement each other well so far, but there is a piece still missing.

Connecting the Missing Link

For delegated authorization to work, the user needs a unit of delegation — some handle so they can say: I want to allow application X to read my application list on cloudfoundry.com, but not actually modify anything. This notion of specifying names for roles or permissions is called the token ‘scope’ in OAuth2. We needed some way to support management of these scopes, their existence, what users have access to them, what web applications can ask for them — and do it all in a way that naturally supports their use in authorization decisions.

Therefore, we mapped SCIM groups to OAuth2 scopes. In other words, you can create a group in the UAA and register a web app with that group name as a scope. When a user logs into the web app, it’s redirected to the UAA. If the user is in the group and they choose to authorize the web app with that scope, the web app gets a signed token that contains that scope.

We use this in our own cloudfoundry dashboard application. The system reliability engineers access the dashboard app, which is a UAA client app and always requests a token with dashboard.user scope. The UAA denies the scope unless the user is in the dashboard.user group.

What this gets us is that the dashboard app does not have any user account code, and it never gets the user’s credentials. It merely redirects to the UAA if there is no access token in the authorization header. If there is an access token it validates the token, and then checks if it has dashboard.user in the scope. For the system monitors, this means no need for shared accounts or passwords and better control of which users have access.

This is particularly useful when there are multiple such applications. Users get SSO among the applications, the applications are each more secure, access can be easily controlled by groups in the UAA.

Extensible Authentication Services

A somewhat recent addition is that we have separated the authentication service from the rest of the UAA. In OAuth2 terms this is separating the authorize and token endpoints. We can have different implementations of the authentication service for different needs. So far implementations are available for username and password (on cloudfoundry.com), for various flavors of openid2, and for SAML2 identity providers.

Why this is Cool

This is all quite useful, but how it can be used in a cloudfoundry deployment is best illustrated by some examples:

Suppose cloudfoundry is running in a managed instance for an enterprise. The enterprise has a suite of apps that can use the full advantage of the PaaS — easy deployment, scale up, scale down, with a myriad of frameworks and services. All the apps can use the UAA at the core for SSO and role (group) based authorization. Add a user to a group via SCIM, and the corresponding scope will be available at next login to all apps.

Cloudfoundry.com includes various web apps to help developers. Some of them only need to know who the user is for SSO. The openid scope is perfect for them. Suppose a new site wants to also provide personalized content based on the user’s running apps. We can create a registration for that app that restricts its scope to listing the user’s apps. If that app is compromised, nothing in the data of the user’s apps can be changed.

Suppose you are a developer running a private cloud in an enterprise. With Cloud Foundry you get these features:

• You can create and deploy applications exactly the same as apps on cloudfoundry.com with access to the multitude of frameworks and services, exactly the same as apps on cloudfoundry.com.
• Your apps can register with the UAA in your cloudfoundry instance, which means they don’t have to manage users or authentication. All apps in your instance can have SSO and common groups for authorization.
• Further, you can configure the UAA in your instance to include a SAML Login server, which gets user authentication from the corporate SAML IdP in front of an Active Directory instance. All apps now have SSO with the corporate IdP.
• Eventually you’d like a user’s AD group membership for be available to your app as an OAuth2 scope. Put the group membership in the SAML assertion and it can be mapped to a scope by the SAML Login Server — this part isn’t actually implemented yet, but is here to show future possibilities.

Cloud Foundry identity services are built on open standards and open source. They increase security and reduce friction so users can take advantage of the open PaaS.