Java and Cloud Architecture Rambings

How to Start Developing with Cloud Foundry for JVM based Languages - Part 1

2018-05-23T10:43:00.000-04:00

There really is a very slim barrier to entry to start developing for JVM based languages on Cloud Foundry (CF). In fact, one of the easiest ways to start developing is to use the CF-DEV project . Development teams can proceed pretty far in the development life-cycle without having to actually think about the eventual deployment architecture and, once those decisions are made it will not affect your code

There are some things to consider when you start developing such as how you are going to deal with data storage, and how you will access resources that are not on Cloud Foundry. In this set of blog posts, there will be instructions on how to make sure that these factors do not throw your development process into disorder. Also, in order to not have issues later on when you figure out your production deployment, there are some architectural considerations as well

As part of the initial phase of starting to use Cloud Foundry for development, your security/infrastructure team may be involved in the decisions you make. Definitely, when you are deploying to a production environment they will be heavily involved. It is a smart move to handle these situations early in the development cycle so that the impact (although not in the code) will be low. If not handled early, this choice will have an impact on your ability to deploy to a production Cloud Foundry environment:

Security on endpoints inside and outside of Cloud Foundry
Certificates and HTTPS are natively part of Cloud Foundry. There are options to run HTTPS inside of the containers, or just for the external endpoints. There are pros and cons associated with each decision.
Authentication and Authorization for the application components

OAUTH2, LDAP, and SAML integration are all possible with either a commercial Authentication / Authorization solution or open source projects such as Cloud Foundry UAA or Istio.
The storage of secrets outside of the application and Cloud Foundry is an important decision that can have an impact on the applications as code changes will need to be made to implement any of the possible solutions. Many organizations have versions of a specific operating system, agents that must run on any implemented virtual machine, and other requirements that conflict with the way that Cloud Foundry “out the box” executes.

Intended Audience

Technical management, software developers, and anybody who is evaluating Cloud Foundry or any of the distributions for use in deploying their micro-services based applications. The assumption is that you have either attended some training for Cloud Foundry or one of it’s distributions or have read the large amounts of documentation available on cloudfoundry.org.

A few words of wisdom

Don’t be afraid to start over, the whole concept of Cloud Foundry is that you can change the platform without affecting your running applications.
Don’t be afraid to fail. Trying a new service broker or BOSH release does not typically affect your running applications.
You can start using Cloud Foundry without having to make it look like your production environment from the start.
Don’t attempt to figure out all of the scenarios immediately for how you will use Cloud Foundry. From a development perspective, there is little risk in starting to use CF.
Spend time learning the underlying deployment technology that deploys Cloud Foundry onto your Infrastructure. BOSH is what is used in most distributions,

What to know before you start

Cloud Foundry is a platform where “you don’t know what you don’t know” can be a bad thing but not something that will doom your project. Using Cloud Foundry as your development environment and eventual deployment may be the thing that enables you to have tremendous success or wallow in what I like to call “The pit of despair.” There are many decisions that may not seem important at first but can doom your deployment to a production Cloud Foundry instance. It is typical, in my experience that because of the software architecture of Cloud Foundry mistakes can be remedied by re-installing the Cloud Foundry release or adding a BOSH release later. The great thing about this platform is that the act of re-installing Cloud Foundry has no effect on the running applications and the code.

Cloud Foundry Operating Environment

Is your operations or security team asking these questions? Can you change the operating system that runs the “Virtual Hosts” for Cloud Foundry and/or the operating system used to run the garden containers?

What are your expectations for the operating environment?

Introduction to the Cloud Foundry (CF) Operating Environment

If you didn’t already know, CF natively runs all of its workloads in Garden containers. For this blog post I am not going to go into detail on what containers are, but let’s think about them as an isolated system process that runs in isolation on a “Virtual Host.” Each container It has its own network, resources, and memory. For Cloud Foundry there are “Virtual Hosts” that run the containers and generally maintain them. If you want more detail the Cloud Foundry documentation has quite a bit of documentation on the Cloud Controller which manages the “Virtual Hosts” and the containers.

Each of the “Virtual Hosts” run a default Operating System that Cloud Foundry has selected through the BOSH release. To change the operating system version, or to add some specific behavior at the operating system level you will have to make changes to the release of cloud foundry for BOSH. Making changes to the BOSH release can be difficult and not something that I would advise. BOSH releases for CF are tested by the open source teams and certified to work, once changed it’s also difficult to ascertain what the resulting impact will be on the operation of Cloud Foundry.

If the operating Cloud Foundry uses a specific source container Operating System that is contained in what is called “stemcells.” By creating certified stemcells, Cloud Foundry enables a system that does quite a bit for you in the software infrastructure area but is also “opinionated” on certain items such as the Operating System used for deployment and the specific set of security software that is deployed on it.

The issue is that many larger organizations have standards for the Operating System that applications run on and the monitoring agents that must work with them as well. The solution is typically a custom stemcell. If you follow the instructions on the Cloud Foundry website a new stemcell can be created, it’s not easy but it can be done.

This decision is not one to be taken lightly as there are tradeoffs in every situation. if you have no specific software infrastructure standards for development and are willing to accept the default container (OS and environment) from the Cloud Foundry community, and default build-pack (Java environment) you can implement and use Cloud Foundry right out of the box. If you already have strict standards in place for what Operating System should be used or for example what version of Tomcat you want to support, and you just need something to enforce it then Cloud Foundry can be valuable, but difficult to implement.

Next Post - When should you create a custom CF stemcell versus creating a custom CF Buildpack.

An Example of Docker Multi-Host Networking With Hadoop Part 3 - Validating your Swarm Cluster

2016-02-18T08:55:00.003-05:00

Validating The Docker Swarm Cluster

In parts 1 and 2 of this series I have shown the process of creating a bare-metal Docker Swarm Cluster using Ambari as an example. In this post I will go through the steps that I take when I bring up the Docker Swarm Cluster in my home lab. To make it easier to understand the infrastructure referenced, I have created a post with the configuration of my home lab.

Assumptions:

All Docker hosts are booted up and running.
Docker Swarm has been installed on each host as a Linux service and is running.
There is one host that has the Swarm manager running.
There is at the minimum one host that has Consul running.
An overlay network has been created successfully.

Recommendations:

Start with just 2 hosts, run through these steps and then add the rest of your hosts.
Have a couple of images that you would like to use already pulled from the your repository.
For this cluster, use Consul as your DNS.

Pre-Validation steps:

Follow instructions from Part I and Part II to install the server components and the second host.
Pull the Ambari server and Ambari agent image from the public repository.

$ docker pull hortonworks/ambari-agent
$ docker pull hortonworks/ambari-server

On each host edit the .profile and add the DOCKER_HOST variable

Add this line to the your .profile in the home directory
export DOCKER_HOST=DockerManagerHost:DockerManagerPort

On each host edit the .profile and add the CONSUL variable

Add this line to the your .profile in the home directory
export CONSUL=ConsulManagerHost:ConsulManagerPort

Incorporate the .profile into your environment

$. .profile
or
$source .profile

Validation Steps

Check that the Docker Service, Consul, and the Docker Swarm service network components are functioning correctly.

Get the status of the Swarm cluster - The first thing to check is that all of the expected hosts are actually part of the Swarm cluster. Below is an example of the swarm list command which will provide the list of hosts. Note that you should change the ip address "192.168.1.18" to the server where you have the consul server running. The 8500 port number is the standard consul port.

$ swarm list consul://192.168.1.18:8500

192.168.1.16:2375

192.168.1.36:2375
What you should see is the list of hosts that you have setup to "advertise" as part of the Swarm Cluster. If you don't see all of your hosts, check that both the Docker daemon is advertising it's port and that you have the Docker swarm daemon running and advertising the Docker port. Try restarting all of the services on each host and then check to see if the host is showing up.

Check the Docker network components - When an overlay network is created, it creates a network that can be seen by all of the hosts. That implies that you would not be able to create the network without the Consul server running since it stores the overlay network. Also, as you can see from the example below, when you setup a multi-host network in Docker you should be able see all of the standard networks for each host, and the overlay network that is indicated by "overlay" in the Driver column.

$docker network ls

NETWORK ID NAME DRIVER

88d2173cc4a2 ubuntu-hp-jdavis/host host

8427b3e43162 ubuntu-jdavis-dell/none null

7dfe494ce0aa my-network overlay

c1569a71811a ubuntu-hp-jdavis/docker_gwbridge bridge

90531773029a ubuntu-jdavis-dell/bridge bridge

acd8851af5b2 ubuntu-hp-jdavis/bridge bridge

4614ab39398c ubuntu-hp-jdavis/none null

71fad951cfd7 my-net overlay

e3fd92dd2bf3 ubuntu-jdavis-dell/host host

57b2a293c5d8 ubuntu-jdavis-dell/docker_gwbridge bridge

Check the Docker standard networks - You should see all of the standard networks for each of your hosts in the response to displaying all of the networks. If you don't see a network entry for each host that has for "docker_gwbridge", "host", "bridge", and "none" then there could be an issue with your setup. Also, in my experience it could take a few seconds for all of the networks to display if you are doing this check immediately after creating the overlay network.

Pull a Docker image - When you pull a Docker image while using Swarm you should see that it's pulling the image into the local image repository of each server.

$ docker pull hortonworks/ambari-server

Using default tag: latest

ubuntu-hp-jdavis: Pulling hortonworks/ambari-server:latest... : downloaded

ubuntu-jdavis-dell: Pulling hortonworks/ambari-server:latest... : downloaded

Start multiple containers to see what the distribution pattern is. In this case I used the amb-start-cluster script in the docker-ambari project.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

5f85b455e50c hortonworks/ambari-agent:latest "/usr/sbin/init syste" 40 seconds ago Up 38 seconds ubuntu-jdavis-dell/amb7

210954b1138b hortonworks/ambari-agent:latest "/usr/sbin/init syste" 44 seconds ago Up 42 seconds ubuntu-jdavis-dell/amb5

4e46cbd8a36e hortonworks/ambari-agent:latest "/usr/sbin/init syste" 47 seconds ago Up 45 seconds ubuntu-jdavis-dell/amb3

e38674c6d93c hortonworks/ambari-agent:latest "/usr/sbin/init syste" 59 seconds ago Up 57 seconds ubuntu-hp-jdavis/amb6

7534f368f1c0 hortonworks/ambari-agent:latest "/usr/sbin/init syste" About a minute ago Up 58 seconds ubuntu-jdavis-dell/amb1

0550c56ac2fc hortonworks/ambari-agent:latest "/usr/sbin/init syste" About a minute ago Up About a minute ubuntu-hp-jdavis/amb4

51821a638f8e hortonworks/ambari-agent:latest "/usr/sbin/init syste" About a minute ago Up About a minute ubuntu-hp-jdavis/amb2

3a51c0cfa958 hortonworks/ambari-server:latest "/usr/sbin/init syste" About a minute ago Up About a minute 192.168.1.18:8182->8080/tcp ubuntu-jdavis-dell/amb-server

Note that you can see that the containers were distributed between the two hosts in this example.

Open a shell to a containers on one of the hosts to make sure that they have network access to the containers in the other hosts. There are a few ways that the network can work. In this case the script adds the --dns=192.168.1.18 to the docker run. This adds the ability have a FQDN (Fully Qualified Domain Name) for each of the containers.

$ docker exec -i -t amb1 bash

# ping amb2.service.consul

PING amb2.service.consul (10.0.2.4) 56(84) bytes of data.

64 bytes from amb2.node.dc1.consul (10.0.2.4): icmp_seq=1 ttl=64 time=0.707 ms

64 bytes from amb2.node.dc1.consul (10.0.2.4): icmp_seq=2 ttl=64 time=0.574 ms

The point here is that you should be able to ping a container that is on a different host. This proves that your Docker Swarm Cluster is functional.

Next Steps

Now that you have validated the cluster, you can start adding new hosts, and actually use it. In the next post, I will go through using Ambari to manager your Docker hosts, and how to enable different scenarios of Hadoop.

An example Docker Swarm Multi-Host Lab

2016-02-15T12:07:00.000-05:00

Why a Docker Multi-Host Lab?

This post provides my experience with answering some of the questions around the use of Docker Swarm Clusters.

Can Docker Swarm truly be used with a heterogenous set of hosts? Yes it can.
Is Docker Swarm by itself (without a management tool) functional enough to be used for software development? The biggest issue that I encountered was the deployment of the code to a running container, but once solved is very easy.
Does Docker Swarm out of the box truly provide a sufficient amount of resource control to run something like Docker Ambari and Hadoop without a tremendous degree of custom software and configuration? For me it was trial and error to understand the number of services that I could embed into a container. In the end, I am still working on the logging but for the most part it does not take too much configuration.
Is it possible to create a development/test cluster that could really be useful in my "home lab?" I was able to create a usable test cluster for a certain set of test data useful for a development cluster.

The goal in the beginning of this exercise was to enable a very thin datacenter environment on my hosts so that most of the actual processing power would be put into the applications running on the cluster. The resources are not very heavy on my hosts and something like Cloud Foundry, Kubernetes, OpenStack, or Mesos would add unnecessary complexity for what I needed.

Description of the Hosts:

Ubuntu-dell - 4 cores 32gb memory - Intel Xeon 2TB SATA
Ubuntu-hp - 4 cores 8gb memory - Intel I5 second generation 512gb SATA
Ubuntu-jdavis - 8 cores 16gb memory - Intel I7 second generation 300 gb SSD

Installed software

Ubuntu-dell:

Ubuntu 14.04 Trusty
Docker Daemon
Docker Swarm
Swarm Manager
Consul Server
MySQL Server (for Hive MetaStore)
Tomcat7 Server
RabbitMQ
Apache NIFI

Ubuntu-hp:

Ubuntu 14.04 Trusty
Docker Daemon
Docker Swarm
RabbitMq

Ubuntu-jdavis:

Ubuntu 14.04 Trusty
Docker Daemon
Docker Swarm
RabbitMQ
Tomcat7 Server

Creating a startup script for Docker Swarm Manager

2016-02-13T22:07:00.000-05:00

For Ubuntu-Server 14.04

Assumptions:

You have downloaded the latest Docker Swarm, compiled and have Docker Swarm running.
You have sudo access.

Create the Swarm manager configuration file:

Create a file using your favorite editor in the /etc/init directory called swarm_mgr.conf.
Below is an example of a startup script, please make sure to replace your own Docker Swarm IP and port that you have setup for Docker Swarm and the Consul Server IP and port that you are using as well.

description "Docker Swarm Manager"

start on (local-filesystems and net-device-up IFACE!=lo)

stop on runlevel [!2345]

limit nofile 524288 1048576

limit nproc 524288 1048576

respawn

kill timeout 20

pre-start script

# see also https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount

end script

script

# modify these in /etc/default/$UPSTART_JOB (/etc/default/docker)

SWARM=/usr/bin/swarm

SWARM_OPTS="-H tcp://DockerSwarmIP:2376 consul://consulServerIP:8500"

if [ -f /etc/default/$UPSTART_JOB ]; then

. /etc/default/$UPSTART_JOB

exec "$SWARM" manage $SWARM_OPTS

end script

# Don't emit "started" event until docker.sock is ready.

# See https://github.com/docker/docker/issues/6647

post-start script

DOCKER_OPTS=

echo "swarm manage is up"

end script

An Example of Docker Multi-Host Networking with Hadoop - Part2

2016-02-10T20:05:00.000-05:00

Next Steps - Add a host to your Cluster

In Part1 we showed how to install the basic components for doing mult-host networking with Docker. In this post I provide the first part of the instructions to add a host to a cluster. In Part 3 I explain the steps to validate your cluster.

If you are interested in the composition of the lab that I used for this post, it is outlined here.

Assumptions for adding a Host:

You already have a running Swarm cluster which includes have Docker Swarm installed and the Docker Swarm Manager running.
The host has a static IP on a network
You have access to the root user to install products
The hosts have direct or proxy access to the internet
Java 1.8 has been installed on the host
No DNS software has been installed on the host
IPtables has either been disabled or is not forwarding port 8081.

Adding a Host

Install Linux on your host
Install standard Linux packages
Install Docker
Install Consul (optional)
Install Go
Install GoDep
Install Swarm
Test your new host (next post).

Install your favorite Linux variant onto the host

Ubuntu 14.04 trusty is the Linux OS that has been used to build this, but you can pick any Linux OS that is compatible with the latest version of Docker. The one requirement is that it must be a 64 bit Operating System.

Install these standard Linux packages

$sudo apt get install git

Install Docker Daemon

Follow instructions here for Ubuntu Linux, it is very important that you have a version of Docker installed that is equal or greater than 1.9.x.
As part of the installation the installation will create a startup script, in Ubuntu this will be in /etc/init/Docker.
In later steps we will be changing this file so you will need to know where this file is.

Install Consul (optional)

You can install Consul on all of your hosts, or just one. So, if it's already installed on one of your hosts, this is an optional installation.

My recommendation is to use the instructions here.

Install Consul as a service by creating a script in the /etc/init directory called consul.conf. Below is an example of the file, if you follow the instructions from Consul it will provide you with explanations of all of the options in this file. Since I am using Consul as a DNS server, Consul is running as root. That is why I commented out the set user id and group id for consul.

Make sure to replace the with the host that you are running consul on.

description "Consul server process"
start on (local-filesystems and net-device-up IFACE=eth0)
stop on runlevel [!12345]
respawn
#setuid consul
#setgid consul
exec consul agent -config-dir /etc/consul.d/server

Here is an example configuration file, put this file in the --config-dir that you specified above (/etc/consul.d/server). You should follow the instructions on the consul site to install, as you will need to encrypt a pass phrase and place it in this file.
{
"bootstrap": true,
"server": true,
"data_dir": "/var/consul",
"encrypt": "",
"log_level": "INFO",
"enable_syslog": true,
"ui_dir": "/ui",
"client_addr": "0.0.0.0",
"ports": {
"dns": 53
},
"recursor": "8.8.8.8",
"disable_update_check": true
}

Install Go

Execute these commands, substituting the latest Go version instead of the 1.5.3 version that I used:

$wget https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz
$sudo tar -C /usr/local -xzf go1.5.3.linux-amd64.tar.gz

Make a directory for the code in the home directory of the user that will compile Go code.

mkdir gocode

Edit the .profile or .bash_profile depending on the Linux variant and add this variable to the user that will compile the Swarm code.

export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/gocode
export GOROOT=/usr/local/go

Install Godep

Follow the instructions here.

Install Swarm

Follow the instructions here to compile Docker Swarm. If you have an earlier version of Go installed lower than 1.9 you will have compile errors in that situation you should follow the instructions, uninstall your current version and add the new version.

Copy the swarm executable from the ${GOCODE}/bin directory to /usr/bin
sudo chmod 755 /usr/bin/swarm sudo
cp ${GOCODE}/bin/swarm /usr/bin/.

Create a startup script in /etc/init called swarm.conf, a sample is below. Please correct it with your own Docker server host Ip and Consul server IP. In addition you will want to make sure that the port you are exposing for Docker and Consul are correct as well:

description "Swarm daemon"

start on (local-filesystems and net-device-up IFACE!=lo)

stop on runlevel [!2345]
limit nofile 524288 1048576
limit nproc 524288 1048576

respawn

kill timeout 20

pre-start script
# see also https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount
end script

script
# modify these in /etc/default/$UPSTART_JOB (/etc/default/swarm)
SWARM=/usr/bin/swarm
SWARM_OPTS="--advertise=hostServerIp:2375 consul:consul-server-ip//:8500"
if [ -f /etc/default/$UPSTART_JOB ]; then
. /etc/default/$UPSTART_JOB
fi
exec "$SWARM" join $SWARM_OPTS
end script

# Don't emit "started" event until docker.sock is ready.
# See https://github.com/docker/docker/issues/6647
post-start script
SWARM_OPTS=
echo "swarm join is up"
end script

Start Swarm

$ sudo service start swarm

Configure Docker Daemon service

When you installed the Docker Daemon in the previous step it created an /etc/init/docker.conf file which has the configuration for the Docker Daemon. Just like the Swarm install you will need to make sure that it has the correct IP address and Port for the docker host and the consul server. Note that you should not have to create this file, and the only thing that should be changed is adding the entry for the Docker Options.

These options tell the Docker daemon that the cluster store is using Consul at this specific machine, and it adds this host as part of the Docker Cluster. Technically at this point you could do multi-host Docker without Swarm, since you can create an Overlay network across the hosts, but Swarm makes it much easier by handling placement of containers.

description "Docker daemon"

start on (filesystem and net-device-up IFACE!=lo)

stop on runlevel [!2345]

limit nofile 524288 1048576

limit nproc 524288 1048576

respawn

kill timeout 20

pre-start script

# see also https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount

if grep -v '^#' /etc/fstab | grep -q cgroup \

|| [ ! -e /proc/cgroups ] \

|| [ ! -d /sys/fs/cgroup ]; then

exit 0

if ! mountpoint -q /sys/fs/cgroup; then

mount -t tmpfs -o uid=0,gid=0,mode=0755 cgroup /sys/fs/cgroup

(

cd /sys/fs/cgroup

for sys in $(awk '!/^#/ { if ($4 == 1) print $1 }' /proc/cgroups); do

mkdir -p $sys

if ! mountpoint -q $sys; then

if ! mount -n -t cgroup -o $sys cgroup $sys; then

rmdir $sys || true

done

)

end script

script

# modify these in /etc/default/$UPSTART_JOB (/etc/default/docker)

DOCKER=/usr/bin/$UPSTART_JOB

DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://consulHostIp:8500 --cluster-advertise=dockerHostIp:2375"

if [ -f /etc/default/$UPSTART_JOB ]; then

. /etc/default/$UPSTART_JOB

exec "$DOCKER" daemon $DOCKER_OPTS

end script

# Don't emit "started" event until docker.sock is ready.

# See https://github.com/docker/docker/issues/6647

post-start script

DOCKER_OPTS=

if [ -f /etc/default/$UPSTART_JOB ]; then

. /etc/default/$UPSTART_JOB

if ! printf "%s" "$DOCKER_OPTS" | grep -qE -e '-H|--host'; then

while ! [ -e /var/run/docker.sock ]; do

initctl status $UPSTART_JOB | grep -qE "(stop|respawn)/" && exit 1

echo "Waiting for /var/run/docker.sock"

sleep 0.1

done

echo "/var/run/docker.sock is up"

end script

Re-Start Docker service

$ sudo service restart docker

Next Steps

At this point all of the services are running on your new host including Docker and Swarm. You should now be able to try to use these hosts as one Docker Swarm. In the next post I explain the steps to validate your cluster.

Why do MicroServices projects fail to deliver?

2016-02-10T19:59:00.002-05:00

What is the definition of a Micro Services Architecture?

A MicroService is a misused term. The best way to understand micro-services is to understand that the concept of "micro" does not describe the size of the application but the domain by which the service encompasses. It is well explained by many others, but at it's most simple it's the idea that if you can encapsulate a concept's data and functionality into a discrete set of commands you have a micro service. This explanation is a reinterpretation of Martin Fowler's as I believe it's not just about the services themselves but the domain barriers that it creates. Eric Evan's seminal discussions around Micro Services Architecture and Domain Driven Design is the closest to an understanding of what it takes to have a successful implementation.
Micro Services architecture uses some specific concepts that may be unfamiliar to you, even if you are writing Micro Services at this time:

Domain Model - A domain model in regards to Micro Services provides the business context by which the service is exposed to the outside world. The Domain model can be anything that represents the system as a singular perspective on a problem. As a design principle Micro Services applications do not directly share data in their data except by activation through a service call. For instance, the domain model that Docker exposes in it's services could be considered one that describes running systems, hosts, and applications. In this case, it's domain is bound around a specific context which is containers and the information necessary to run a container.
Service Contextual Boundary - The Contextual Boundary can be thought of as being the thin skin that separates the Domain Model of one system from another. This can be though of as an abstraction of the interaction between very similar systems that would need to cooperate/collaborate to accomplish a specific goal. The Service Contextual Boundary is used to further define when one service should have a set of functionality and therefore data, and when it should not.
Domain Based Service - A service based upon a domain is a fundamental architectural piece of the Micro Services architecture. One very important architectural decision is what domains truly belong to one service as direct data sharing is not permitted between two service based applications. It is through the understanding of Domain Based Service concepts that it is understood what data, and what context should be used when interacting with the application.
Shared Nothing Architecture - The idea that systems are built for web-scale when it does not have to share or save session state data, other than through established system level interfaces. In addition, it has a degree of independence from other technical dependencies so that in a situation of hardware or network failure it can still function with the data that it has locally.
Externally facing protocol and API(s) matter, but they don't define the architecture - A MicroService architecture should not be bound to a specific protocol such as AMQP or a public API such as JMS, or even REST. These should be just bindings that can be changed at any time.

This approach to architecture has become popular with anyone who plans to use Public Cloud Infrastructure As A Service (IAAS) such as Amazon, Google. In addition, the on-premise Platform As A Service (PAAS) such as Cloud Foundry provides considerable support for the Micro Services architecture.

Why are MicroServices projects so difficult to deliver?

The Micro Services approach to application architecture has been hyped quite a bit since 2013. The architecture has grown out of other architectures that were just as hyped such as Object Oriented Design, Functional programming, and Services Oriented Architecture. Each of these have been thought of as the "Holy Grail" for computing, and in Functional Programming's case is a resurrected approach that many had dismissed in the development community years ago. Micro Services architecture in my view can be looked at as one of the options for shared-nothing approaches to distributed application design.

With this approach though, comes a degree of difficulty that many underestimate. Organizations are attempting to put Micro Services projects in place and framing their software development around them. As the number of projects overlap that cover the same domain areas, you have truly a complex problem. This may not be a problem for a small number of people, but as the number of people grow and the number of services grow as well, everyone will have their own Micro Service to handle their own very specific problem within a very large domain.

MicroService projects have a problem delivering primarily for this one reason:

You must understand the domain and have a plan before you start developing your services.

With the advent of languages like Node.JS the size and complexity of these small almost "snippets" of code is pretty high as it is not understandable what each of these small services are actually doing. Is it a bad practice to use scripting and small sets of code to do your Micro Services, absolutely not, but you must think about what you are doing within the context of your larger application and not violate any context boundaries whatever the programming language or size of code block.

- Think of Micro Services as a "Product" not a "Project."

A problem was posed to me years ago when I was at a company that had over 175 individual financial reports that stated different numbers according to the whims of the individual requestor. The problem was, "How can I still have my individual report that has my His perspective on the numbers was while knowing that everyone else starts out with the same original facts?"

if this problem it turned on it's head you would find that the problem is not the language or the implementation of the services themselves, but the concept of how "projects" are delivered. Micro Service architecture severely depends on having an understanding of the Service Contextual Boundary and a perception of the domain that goes beyond a specific project. In order to accomplish this change, and effectively manage Micro Services the domain services need to be understood and thought of as a Product of the organization, not as an IT or software development project.

Deployment is the big advantage or disadvantage

Micro Service architectures when implemented have the greatest advantage over other architectures when the implementation is finished and it is deployed into a production environment. Due to the natural testability of MicroServices deployments can be seamless. Scaling up and down can be quite a bit more easily accomplished than other approaches.

At the same time, the disadvantage for a Micro Services deployment is that most infrastructure organizations are not used to deploying the volume of artifacts associated with this style of architecture. Without implementing tools such as a Platform As A Service deployed on-premise or in the cloud the technical and procedural hurdles may be just too high for most operations teams.

Why switch to a micro-service architecture if it's so difficult?

Everyone looks for the next big thing. At this point the MicroService architecture has been attempted because it does offer some perceived advantages over traditional Services Oriented Architecture and Object Oriented development approaches.

It works well with AGILE projects in that you can define boundaries on functionality and define specific service interactions as part of your micro services boundary.
Services Oriented Architecture and Domain Driven Design provide the technical mechanics behind deciding the style of the services that that we expose and the content of the messages we are passing. These items are important to MicroService architecture.
MicroService architecture provides specific patterns that enable horizontal scaling and failover. These are up to the individuals developing the services to implement the functionality, but it will be available if MicroService architecture principles are followed.

Conclusion

Follow the path:

Plan for failure
Implement horizontal scaling
Validate each service's use in the domain.

An example of Docker Multi Host Clustering Part 1

2016-01-20T12:33:00.001-05:00

Ambari on Docker - using multiple physical hosts for Bare-Metal Docker Swarm Clustering

A new project that I have been working on at joshuad2/docker-ambari has been created to help you to get started with multi-host Bare-Metal Docker Swarm Clustering. In this case the scripts implement an Ambari Cluster which makes it much easier to create a Hadoop Cluster. Using a heterogenous set of physical machines you create a Docker Cluster that can continue to expand over time.

The instructions for Ambari can be extended to any other service that provided clustering such as Apache Tomcat, Apache Nifi, Apache Spark, and Apache Storm. The lab that I used to do this is outlined here.

Bare-Metal Docker Swarm Clustering

Bare-Metal Docker Swarm Clusters are built using multiple physical hosts that have Docker and Swarm installed locally, not through Docker Machine.

What is needed to make this work?

More than one host with linux (Ubuntu 14.x) installed. They can be virutal machines, but if you are going to use VM's you may want to use Docker Machine which is well explained in the Docker documentation.

All hosts must have either a fixed IP or an IP that will not change

All hosts must not have a DNS installed

Why do I use Consul for DNS?

Consul is used for three reasons:

It is easy to install and use with Docker - Consul's documentation on setting it up for use is great and it has a very nice GUI. In addition it has greater ability to handle failover that may be nice to use in the future.

Consul is used as a dynamic DNS for the Ambari nodes - Hadoop and many other network based software needs to have a fully qualified domain name (FQDN) to operate. Consul provides this. The important detail is that you have to create a Consul server with Port 53 exposed, which is used for DNS in order to have the Docker nodes use it. If you have dynamic DNS already installed in your environment you can use it as well, but you will need to find a way to add the containers dynamically at run-time.

Consul is used as a "backing store" for the Docker Cluster - Docker requires that if you want to use Docker Swarm you must have a "backing store." This is explained in great detail on the Docker Site

Pre-requisites for each host

Go Language - Make sure to install the latest version of Go onto your hosts. If you do not go Dep and Swarm will not compile.

Go Dep - Follow the instructions to download and install. You will need to set a couple of environment variables (GOROOT and GOCODE) before running the instructions.

Docker - Follow the instructions for your Linux variant, you should make sure to use atleast the Docker 1.9+ version. When installing Docker you will have a startup script installed so that you can access it as a service. You will need to follow the instructions on the Docker Swarm install to modify the startup for Docker Swarm.

Docker Swarm - Follow the instructions to download and install Swarm. Once you have compiled Swarm copy the executable under ${GOCODE)/bin to /usr/bin (or the appropriate standard directory).

Consul - Follow the instructions to install Consul. If you are not deploying this to a Production environment you may not need more than one host running Consul. Make sure that you expose port 53 for DNS (instructions are on the Consul site).

Optional Installs

Docker Compose - Same as Docker Swarm, you will want to copy the executable into /usr/bin.

Prepare the hosts

Add each host to the /etc/hosts file with a short name for the machine

Pick which host(s) will start the Consul service

Pick which host(s) will start the Consul GUI (it's optional)

Create a startup script (/etc/init/consul.conf) on the consul server machine so that when it comes up it will start up consul.

Decide which host will have a Docker Swarm Manager. You will want to use a port that is easy to remember as you will use it on each Docker call once the cluster is usable.

Modify the Docker startup script on each host to advertise itself as part of the Docker Swarm Cluster.

Create a startup script to start Swarm on each host.

Create a startup script to Start the Swarm Manager on the host(s).

When you have finished setting up the Docker Swarm, setup the Docker Overlay Network.

Running Ambari and Hadoop using Docker Swarm

Use Git to pull down the repository

Export the location of your Docker Swarm Manager:

`$export DOCKER_MANAGER=tcp://192.168.1.18:2376`

Export the name of the Docker Overlay Network you are using

`$export OVERLAY_NETWORK=my-net`

Export the location of the Consul Server

`$export CONSUL=192.168.1.18`

Put the ambari functions into your environment

`$ . ambari-functions`

Start the ambari server and nodes. The command below will start an Ambari cluster with 5 nodes, one for the Ambari server and 4 nodes with Ambari agents.

`$ amb-start-cluster 5`

Find the host where the Ambari server container has been created:

`$ get-swarm-host amb-server`

`$ ubuntu-hp-jdavis`

Use the Ambari User Interface to create your cluster by going to the server to the server you received from the above query. If you have setup all of your hosts in /etc/host on each machine you will be able to access the server in your browser. Right now the port exposed for the Ambari server is 8182, you may need to change that if this port is already used on all of your hosts. Docker will not put this container onto a host where the port is already used.

In my environment the ambari server would be accessed using:

`http://ubuntu-hp-jdavis:8182/#/login`

At this point you can install your Hadoop server onto your Ambari cluster!

Part 2 is here
Part 3 is here

Using Docker as a PAAS and a comparison to Pivotal Cloud Foundry

2015-07-31T14:47:00.000-04:00

Introduction

Using Docker as a Platform as a Service (PAAS) in a similar way to Cloud Foundry is something that I did not think was possible until I actually had the challenge of putting it together. Docker is part of an ecosystem of open source projects, similar to the Cloud Foundry project. In addition to it's own projects and repositories that has been created by Docker each of the components except for the core containerization engine are available to be built by others as long as they adhere to the open standards that Docker has established through the Open Container Foundation.

When you compare the Docker container engine to Cloud Foundry it's like comparing a human arm to a complete human body, it is just not fair. But, when you add the Docker Compose, Docker Swarm, Docker Machine, and Docker Hub to the mix you are close. It is only when you add the open source projects from HashiSource (Consul, Registrator, Packer, Consul Template) and Jenkins with the Jenkins DSL do you get closer to the full set of functionality offered by Cloud Foundry.

PAAS Execution Environment

So, what is this PAAS thing?

A PAAS at a high level has these components:

Infrastructure Abstraction - A way of abstracting the hardware infrastructure and networking layer away from the developer/programmer.
Software Defined Infrastructure - A simple software abstraction that enables dynamic software defined infrastructure.
Service Enablement - A self-service ability to create releases of software that enables the three "C's".

Continuous Delivery
Continuous Integration
Continuous Deployment

Service location independence - A Services environment that enables the creation of user and system defined services where the programmer and user does not need to understand where the service actually executes, but only cares about the interface.
Operational consistency - A set of software components that regardless of Operating System or underlying infrastructure whether it be cloud or local provide consistent operational behavior.
Configuration Management - A set of API's or software components that provide object management and versioning along with recovery and backup capabilities.
Logging and Monitoring - API's should be available to enable Logging and Monitoring
Security - Once enabled, the API's for security should enable typical security profiles used in an organization.

Docker by itself is not a PAAS

Below is an example configuration for the "Execution" environment for the Docker images.

The Docker projects in the open source community (Not Orca) are projects that are setup to work with other vendor's open source products. The diagram above shows a simple high level implementation of the Docker Container engine alongside products from HashiSource, NGINX and different cloud providers. Each of these products adhere or work with the Docker container runtime through the API that is exposed, For instance, when you start looking at Docker Swarm you can substitute Google Kubernetes, Apache Mesos, or even Apache Brooklyn. In my experience though there really is nothing better than having one set of consistent interfaces to work with so Docker Swarm is actually a good place to start, especially when you are using it as a starting point for development.

What is missing?

On-premise build, release, and object storage

There are two things missing from the diagram above, the ability to store released objects (in this case images) and the ability to release. In the Docker ecosystem this is easily rectified by using a private Docker Hub repository and a Github or BitBucket account. If you happen to be using one of these cloud based repositories, that's fine but if not you will have some work on your hands. The way that I have seen this done is by using a locally hosted Git server, Apache Jenkins, and JFrog Artifactory. Although not perfect, this combination of products can be used to create dynamically implemented build environments.

Comparison to Cloud Foundry

Implementation Complexity

Where Cloud Foundry has some very specific certified implementations you find that the Docker community has an infinite number of permutations of open source and proprietary products that are integrated with it. The Cloud Foundry Foundation has reduced this complexity by providing a body by which something can be called "Cloud Foundry." This does not exist with Docker, although at it's core the Docker projects are extremely simple to implement and provide a simpler implementation "substrate" for most organizations that Cloud Foundry. That said, there are many implementation decisions that have to made to implement Docker as a PAAS and most organizations will feel that this is a burden rather than an advantage.

Ease to Deploy and use for Developers

This is definitely where Docker has an advantage. Cloud Foundry out of the box has many complexities associated with it's main concepts from a development standpoint. It is prescriptive and by doing this creates a situation where many legacy applications will just not run inside of Cloud Foundry. In contrast, Docker has a simple model that allows for any application running on Linux (and Windows very soon) to be able to run without making a change. There is always the danger that the developer may ignore the facts that drive space is typically ephemeral on Docker, but that is left to the software developer to figure out. There are GUI tools for both Cloud Foundry and Docker.

In both Cloud Foundry and Docker software developers can setup a "host" to run containerized applications locally. The difference is that Docker can run on very small environments, even a Raspberry Pi.

Ease to deploy for Operations on-premise

Cloud Foundry would seem like it has an advantage, but not for long. There are established procedures to getting a Cloud Foundry environment setup and there are vendors from the Cloud Foundry Foundation that will set it up in production. They will certify the environment and train the operations teams. There are established logging and monitoring tools available for Cloud Foundry.

The Docker ecosystem of open source projects provides many choices without one central winner. Vendors are available to implement the Docker environment such as HashiSource but there is no central operations standard for logging and performance monitoring of Docker containers while having the level of security that most enterprises require.

The Docker approach requires a much more hands on or "active" operations team to configure the exact base images, security, networking, and service registration necessary to build a functioning production system. Cloud Foundry is much more prescriptive, and while that may make things difficult at first does incur quite a bit of benefits from an operations standpoint in that things are standardized therefore implementing a more "passive" approach.

Usability in the Cloud

Different vendors have implemented Docker in the cloud such as Amazon Web Services and Google. These implementations are in Beta right now as of 7/31/2015. Cloud Foundry is available through the same vendors as well as EMC, VMware, IBM, and Hewelett Packard.

Functionality

From an overall functional standpoint for a PAAS Cloud Foundry provides in one package all that an organization would need. The consequence of having all of the functionality is that it has a larger infrastructure footprint on install and has a higher barrier to entry technically for most organizations. The Docker ecosystem on the cloud using Docker provides the same level of functionality. Without Docker Hub, it does not provide the complete set of functionality, but with other open source projects it comes close. The big difference from a functionality perspective comes down to the style of developer operations you are interested in (active vs. passive) and whether you want to roll your own PAAS that is customized for your organization. Docker can be easily customized, Cloud Foundry can be customized but the tools and capabilities necessary to do so are quite high.

Summary

Both options have some advantages, don't assume just because something is easy to download and use that it will be easy to implement in a production environment. Docker and it's ecosystem is powerful and easy for developers to use. Cloud Foundry is powerful as well, but maybe a little more difficult to use from a development perspective (almost on purpose). Make sure that you have a plan to put both into production before you start using it in development, and perhaps consider a blended approach as well.

Resources/links

Pivotal Cloud Foundry

Cloud Foundry Foundation

Docker

Stackato PAAS and contrasts to Pivotal Cloud Foundry

2015-03-09T14:41:00.001-04:00

Installing the Stackato Development Environment

It is easy to install the Stackato packaged virtual machine for development. It loaded right into VMware Fusion. The resources that it needs are minimal which is excellent for developers and for users trying out PAAS. There is no BOSH release to create and deploy and initially getting it up and running is easy.

You have to add memory resources to the Virtual Machine to enable applications to run, especially if you are using VMware Fusion. An apparent issue is that I experienced was a resource issue when deploying; I just did not see the problem until the staging failed on an install of Jenkins. When an application failed to stage because of memory or resource issues it does not show the error (Pivotal Cloud Foundry has the same issue). Normally, I would bosh ssh into the Bosh process. I don’t see a way of doing that with Kato although it may be possible using stackato ssh because it’s using the same container for staging the application as it is for running it.

Deploying an Application to the Stackato PAAS

Once the memory was increased to 4 gigabytes of ram storage on the virtual machine I was able to deploy the smallest application possible, a GO application. The experience is a different from Pivotal Cloud Foundry (PCF) because PCF does not have an application marketplace. PCF only has a services marketplace, which I could not find on the Stackato GUI. You load the application from the developer console via the Application Marketplace. That is interesting since in PCF you can’t load an application in any way from the developer console. When started the logs show Stackato uses the Docker container, which implies that Stackato is in one way more advanced than “out of the box” PCF.

Services

Stackato hosted data services are not cluster enabled. This is not an issue because from a marketing perspective Stackato follows a very pragmatic approach establishing that production data services will most likely be hosted outside of the PAAS. Most customers have external services that they want to use, but the advantage of having consistent deployment environments can’t be undervalued. This is important for a good continuous delivery process.

The Stackato documentation encourages the use of Service Brokers to enable automated provisioning to the external data services. This is an excellent approach when working with customers who have legacy applications. The potential issue is the services are not managed directly by the Stackato health management and logging services. From a service-offering standpoint, other than RabbitMQ, MySQL, Redis, and others that are available as roles, the services are minimal and not production ready out of the box when compared to PCF which is described on the Stackato site.

Management

User and System Management for Stackato is accomplished in one central website. At first glance it’s a very simple but well designed site. It is responsive and has the capability to be customized to a customer’s needs, even re-branding. There are a few things that you can’t do on the developer site that you can in the PCF Developer console. One is that you cannot scale, create, or bind to a service directly from the application page. You either need to use the auto-scalar or do the scaling from the command line.

Where Stackato differs greatly from Pivotal Cloud Foundry

Stackato roles seem to relate to a BOSH release but not exactly because each VM can be assembled using kato node attach which is similar to the concepts of BOSH. Kato seems to be an abstraction of BOSH. Roles are a different way of abstracting product installs into CF in that they are a more concrete implementation. Also, In Pivotal Cloud Foundry you can map ports, but the Harbor service in Stackato seems to imply that you could possibly use it to enable RMI or any other binary protocol.

Summary

Stackato PAAS is an interesting take on the Cloud Foundry Open Source Project. There are many differences between the commercial offerings including the fact that Stackato has a smaller version for developers and PCF does not. The fundamental difference between the Stackato and PCF distributions is the way the virtual machines are managed. One of the most difficult things to explain about PCF is BOSH. It truly is a PAAS platform component that has many generic management capabilities. The Kato client and the concept of roles are simpler and more manageable concepts to understand, and truly enable a lower point of entry for most DevOps resources.

Solving the problem of poorly performing Web Applications

2015-03-09T14:38:00.001-04:00

Introduction

There have been many times since the “Web” era started that I have been asked to provide a solution for a non-performing website or web application. The first thing that I create is a set of questions to understand the application or environment that is having problems. There are some basic questions that I would ask, especially if I was coming into the situation fresh without any background or context.

Sample Use Case:

A customer was having trouble accessing their authorization policies in a timely manner from a custom built authorization system that they had created. It used a complex policy RDBMS that could return the data under normal load. When faced with load that was higher than normal it did not handle the number of queries and changes to the policies that had to happen at the same time.

GemFire was used (embedded) to create a local application NoSQL layer on top of the main relational database. The performance increased 200% or more based upon the complexity of the policy. The embedded mode for GemFire was used to enable the data to be as close as possible to the application. It also enabled a NoSQL layer that backed a set of REST services for managing policies that were much more responsive than hitting the RDBMS directly.

Questions that should be asked

Is the problem a user’s perception of the performance of the website not being responsive?

Perception of a website’s performance can be affected by many factors including network, server performance, and choice of front end tool (JavaScript vs. others). Most of these issues can be overcome with good application architecture focused on a responsive web experience that has an asynchronous backend. Typically, I have found that this question can uncover many of the typical user experience issues that are incorrectly implemented by the application. One example would be a shopping cart that saves in a synchronous manner to a back-end database at the conclusion of each step.

Does the performance problem center around search or retrieval of data from the back-end of the website?

It has been proven in many surveys over the years that most problems with application performance can be driven down to the queries that are run in the database. Usually, there are two sources of data retrieval that cause problems. The first is a heavily indexed relational database that is used by every application and reporting environment in the organization. The second is the retrieval of data through REST or SOAP messages across many different sources. Either of these or both can cause issues with performance in the long-term and constitute a form of technical debt.

Is the performance issue only when saving in-process data in a long running business transaction?

Many websites and web based applications in general have issues holding and preserving in-process state. This data is the voluminous information that is garnered by the application before an actual physical transaction is recorded. In many applications this data is either saved directly to the back end database, or held in the session state of the web application. Either one of these solutions is likely to create problems when the application needs to scale and deal with failover.

Does the back-end database have issues when there is a peak usage of the database?

Most databases are not used just for the web application(s) that create the transactions. They have other responsibilities such as reporting and batch processing. Peak usage of this kind of database come at unusual times and creates a poor user experience.

Does your application have “pausing” issues waiting for the data sources to be available?

In certain cases if the application relies on databases, application services or an Enterprise Service Bus that is externally hosted network reliability can be an issue that affects application performance. Having the data contained within the application can create a serious performance improvement as well as improving reliability. Since the data is published to the backend asynchronously the data can be queued and wait for network connectivity to “reappear.”

Solution Architecture

Why use the GemFire Embedded Topology as the in-memory NoSQL database for your application?

Using an embedded in-memory database such as GemFire is a great way of mitigating issues around Application performance.

It is a simple solution that can be placed into your application “in-between” your existing front end user interface and your back end database.
It is a NoSQL solution that can represent the domain of the web-site instead of forcing your web application to have the same domain model as your back end database.
Creates a high performance layer that can be asynchronously synchronized with your back end database.
Introduces transactional layer that will only communicate with your database when a transaction needs to placed into the back end database as the source of record.
Enables the future use of a distributed data cache when you need it instead of dealing with the up-front costs of implementation.

Why use the embedded GemFire Topology over the Client/Server Topology?

GemFire is a product that contains many options for deployment, only one of them is embedding the server within your application. The Client/Server approach is an In Memory Data Grid (IMDG) that employs a horizontal scaling and failover approach. The additional GemFire component when compared to the embedded topology is the GemFire Locator. Locators keep track of the nodes within the IMDG and coordinate queries. This enables GemFire clients in applications to access an abstraction for the “location” of where the GemFire nodes are on the network and their exact topology implementation.

In the embedded mode you have the ability to share and exchange data between the nodes of embedded GemFire, but you must either know of the other nodes, or use the Multicast protocol to discover the other embedded nodes. There are other differences when it comes to security options, WAN replication, load balancing and complexity that should be considered when choosing a topology.

The basic rules of thumb for using the GemFire embedded topology are:

Is the GemFire database used to support specific application(s) and the data would not need to be shared with applications that do not have a GemFire embedded database?
Since this topology does not support replication of data across a WAN (Wan replication) is their some other type of replication implemented in the application stack or is it not needed?
Is a simple solution needed for degrading application performance?

How to use GemFire in your application

There are a few resources where there are some great examples of using GemFire within your Java application.

Specifically, I recommend the use of the writethrough example which provides a template for enabling back-end database synchronous and asynchronous replication from the front-end GemFire cache. Another recommended example that provides an end-to-end example for a real world application is a version of the Spring Pet Clinic application adapted for GemFire.

How to Estimate RabbitMQ part 1

2015-03-03T07:37:00.000-05:00

How do you estimate the infrastructure needed for a RabbitMQ installation?

It's a serious problem, you want to use or are already using RabbitMQ but you have no idea what the infrastructure should be for each node, and the cluster as a whole. Over the past 3 years I have been building a method for answering these questions for RabbitMQ, but it could be useful for any distributed cluster product.

How many messages per day do you have going through your current system?

How much message growth in % are you expecting year over year?

Do you have peak hours where messaging is on the average sustainted higher than normal (peak hours)? If so what are the hours?

What are the non-peak hours where there a fewer messages on average?

How many messages/second do you have/plan on having for peak and non-peak?

What is the planned life-span of a message (on average, or give detailed percentages) if it is not ingested by a consumer?

What is the size of your messages? Can you break down into small/medium/large and give percentages?

What percentage of your messages need to be recoverable if the broker fails?

What percentage of your messages and queues need to be available in real-time if a broker fails?

Are you planning on sharing data across two data Centers separated by a WAN?

Are you planning on having an active-active configration for Disaster Recovery?

If you are planning on using RabbitMQ in multiple Data Centers, what percentage of the messages do you estimate will be shared?

What is the expected amount of time (in seconds) that you need to have messages saved in the queues for the worst case scenario?

How many messages are transformed by some intermediate system?

Do you need to have confirmation that a message has been placed onto a queue?

Do you need confirmation that the message has been processed by a consumer?

Do you have any clients that are expecting JTA transactions?

Do you have any clients that are expecting XA transactions?

What is your existing infrastructure plan? How did you reach this plan?

What is the expected uptime of the hardware infrastructure? i.e. 99.9% or 99.99%

What is the expected RabbitMQ server uptime? i.e. 99.9% or 99.99%

Is there a hardware load balancer? If not, are you using a software load balancer?

What is the monitoring plan, if any for RabbitMQ? If there is none, what is the preferred tool in the organization?

Explain the scenario of working with RabbitMQ outages, is it a shared services team or a dedicated team?

Are the machines that RabbitMQ deployed on VM's or physical hardware? If VM, are you using SAN for disk?

Is the link between the two WAN's encrypted or does RabbitMQ need to encrypt traffic between the nodes?

What is the corporate standard for authentication of applications access to resources such as a Messaging system?

Is LDAP authentication required?

Is LDAP authorization required?

If LDAP is not required: Do you plan on having users have only the ability to access their own queues/exchanges?

If LDAP is not requireed: Does each application accessing RabbitMQ need a username/password, is does every User who uses the application need a username/password?

What does the wildcard DNS requirement in Pivotal Cloud Foundry give you?

2015-03-02T13:44:00.000-05:00

What wildcard DNS enables in Pivotal Cloud Foundry is the ability to deploy apps and create the route for the app on-demand. If we didn't have a wildcard DNS entry you could deploy an app but it would not be accessible until that route was added via the operations or networking team to the organization's DNS server. So you'd completely lose some of the great things like zero-downtime deploys, blue/green testing, or A/B testing. At the minimum you'd lose the ability to do those things easily without having to file a ticket with operations for each incident.

Introducing RabbitMQEstimator

2015-03-02T12:55:00.000-05:00

RabbitMQEstimator - RabbitMQ configuration management database, provisioner, and heuristic analysis of traffic.

It is here on Github. (updated 4/1/2015 after leaving Pivotal)

This project is a result of many discussions around the use and provisioning of RabbitMQ which has become pervasive in the past year. The issue is that when you initially setup RabbitMQ and expose it to the different needs of the business it typically will fail. As the use increases, the setup out of the box does not suffice. This is where the RabbitMQEstimator comes into play. The plan is for it to be a small clustered application that will work in a Pivotal Cloud Foundry, Docker compose or bare metal environment. Eventually the work can be done to enable Puppet, Chef, and Ansible.

Note that this code base uses ideas and code snippets from the RabbidManagement project https://github.com/Berico-Technologies/RabbidManagement

This project is a REST service that provides a set of RESTful endpoints protected by OAuth 2. The REST service is based on the Building a RESTful Web Service getting started guide. This project incorporates the new Java-based configuration support, now available in Spring Security OAuth 2.0.

This project builds multiple virtual RabbitMQ clusters in an internal database through a provided resource based REST interface. This set of clusters is a "Project" which has clusters and then nodes. The purpose of this is to have one place where a RabbitMQ Federation or just one RabbitMQ cluster can be stored either through provisioning the changes to the RabbitMQ cluster or provisioning the cluster through:

BOSH release
Docker Compose
Bare Metal (create the RabbitMQ config files)
Puppet (later)
Chef (later)
Ansible (later)

The purpose of the RabbitMQEstimator service is that once the project is provisioned, it will continually collect heuristic information about the cluster. This would be things like

Drive space and memory used by queues
Memory used by the system overall
Number of Exchanges created
Use of Federation

Using the original template by which the project (and it's clusters) are created the RabbitMQEstimator will have the capability to automatically create the new cluster in the environments and then transition the load to the new cluster(s). An HAProxy instance will be used to to create the no-downtime upgrade/deployment. In the future it should support hardware based load balancers.

RabbitMQ and OpenStack

2015-02-02T20:22:00.001-05:00

Introduction

After spending time at a few customers that are using OpenStack with RabbitMQ it is very apparent that the documentation that is available on the OpenStack website is not enough to provide implementors with the basic understanding of how to really implement and administer RabbitMQ. This blog posting will bring some best practices that already exist on the RabbitMQ website . I would always say that everything that I talk about originates from within those pages.

Implementing OpenStack with RabbitMQ

The option to implement OpenStack with RabbitMQ is a good one! RabbitMQ is extremely dependable, fast, and can be very easy to use.

Default OpenStack implementation

This implementation has one RabbitMQ server on the Control Node, which is the simplest but most risky setup. It is risky because if this server goes down your OpenStack installation will completely stop working.

Notes

One Server does not make a cluster
OpenStack uses RPC so you need to understand how RPC works

RabbitMQ clustering basics implies that you need:

1. A low prime number of servers

2. A load balancer, either software or hardware

3. Keep everything in memory as much as possible

4. A procedure to bring down RabbitMQ and one to bring it back up.

5. Monitoring of the server for unsynchronized queues

6. A TTL on some queues that could hold data long term but should not

7. A DLQ that can hold messages that are not picked up

Persistence and Mirroring involves:

1. Understanding that you could lose messages under specific circumstances

2. Auto synchronization of queues

3. All will halt if one halts.

4. The concept of a "master node for the queue."

RabbitMQ Policies:

1. The default OpenStack RabbitMQ policy, why and how does it work?

Frequently asked questions:

1. What is the difference between Durability and Persistence?

2. Why do I have a bunch of queues that are Auto Delete but are not removed from the Broker.

3. Why can't I just re-boot the server that RabbitMQ is on without having problems?

4. Why does it always work with just one RabbiMQ Server?

Open Source Cloud Foundry on Bosh-Lite Installation Notes

2015-02-02T20:19:00.000-05:00

Important undocumented items:

If you run the automated install, you can re-run the yml consolidation manually it will work.

If for some reason the Bosh does not startup well, use bosh cck.

If you get 500 error when logging in:

add this to your cf-manifest.yml

enabled: false

Joshua-Davis-MacBook-Pro:postgres-release jdavis$ bosh deploy --recreate

Processing deployment manifest

------------------------------

Getting deployment properties from director...

Compiling deployment manifest...

Please review all changes carefully

Deploying

---------

Deployment name: `postgres.yml'

Director name: `Bosh Lite Director'

Are you sure you want to deploy? (type 'yes' to continue): yes

Director task 11

Started preparing deployment

Started preparing deployment > Binding deployment. Done (00:00:00)

Started preparing deployment > Binding releases. Done (00:00:00)

Started preparing deployment > Binding existing deployment. Done (00:00:00)

Started preparing deployment > Binding resource pools. Done (00:00:00)

Started preparing deployment > Binding stemcells. Done (00:00:00)

Started preparing deployment > Binding templates. Done (00:00:00)

Started preparing deployment > Binding properties. Done (00:00:00)

Started preparing deployment > Binding unallocated VMs. Done (00:00:00)

Started preparing deployment > Binding instance networks. Done (00:00:00)

Done preparing deployment (00:00:00)

Started preparing package compilation > Finding packages to compile. Done (00:00:00)

Started preparing dns > Binding DNS. Done (00:00:00)

Started preparing configuration > Binding configuration. Done (00:00:00)

Started updating job postgres-server > postgres-server/0 (canary). Done (00:00:15)

Started refilling resource pools > infrastructure1/0. Done (00:00:02)

Task 11 done

Started 2014-12-11 14:53:14 UTC

Finished 2014-12-11 14:53:31 UTC

Duration 00:00:17

Deployed `postgres.yml' to `Bosh Lite Director'

Technical Interview Interviewing Advice

2015-02-02T19:54:00.005-05:00

Interviewing Advice

A few days ago a friend's son who is a college student asked me for a list of simple principles for getting through a Technical Interview. After having done interviews, and been the interviewee many times these are some of the items that I believe in.

1. Make sure that you have a quiet place to answer the phone during the interview.

2. Don't be late to the interview

3. Be yourself and take a deep breath and talk slowly over the phone.

4. Remember that it's ok to say that you don't know something. At the same time, if you have something to share about a question or topic make sure that it relates to the subject.

5. Don't talk negatively about a former employer or school situation. This is a common trap.

6. When being asked about money you should say that you would prefer a paid internship but would take a look at all offers.

7. Always ask 4 questions (like these) at the end of the interview:

A. Am I a fit for the position that you are trying to fill?

B. What are the responsibilities of the position?

C. When are you planning on having someone start?

D. I like this position, what are the next steps?

Pivotal Cloud Foundry and Disaster Recovery

2015-02-01T10:53:00.000-05:00

What is the best approach to architect for disaster recovery with Pivotal Cloud Foundry?

1. Warm Standby - You can have a daily backup script run that will backup PCF according to our documentation

This backup information can be moved to a DR site and re-created when needed.

2. Active-Active - Two live sites running PCF, and all applications and bosh deployments are deployed to both sites.

There are also degrees of usage for both of these solutions based upon factors such as maybe only a set of the applications have to be available in Disaster Recovery, or perhaps the PCF environment is dependent on a database that has to be recovered first. In my experience, the deciding factor is really how fast you would like to be able to cutover to a Disaster Recovery site. You need to understand the Service Level Agreements that are understood for DR in order to select the correct approach.

Using Spring-XD a real life example - Part 2

2014-11-19T10:18:00.001-05:00

Data Ingestion

Historical Data Load

The technical use case is that by off-loading the data from the Enterprise Data Warehouse (EDW) and placing the data onto HDFS the most commonly queried data can stay in the EDW and the extended historical source data can be moved off for querying and reporting.

Technical Implementation

Data is loaded through a comma separated values file into a Spring-XD stream that runs a utility that loads data efficiently into HAWQ. The GemFire data load is not performed in this case because GemFire only holds (in this case) 4 weeks of current data.

Dimensional Data Load

Dimensional Data is reference data that is used to help perform any business rules. This would be used to reduce the size of the historical data load. At times in any set of data there is repeated text such as flags or headings that can be substituted with a reference to a row in a dimension table.

Technical Implementation

Data is loaded through a Spring-XD stream that takes a comma separated value file and updates both GemFire and HAWQ.

Streaming Data

Data that is continually fed from devices can easily overwhelm a system not prepared to handle data in an asynchronous manner.

Technical Implementation

A custom source is used to pull files from HDFS and fed through the RabbitMQ message broker, and then process the data into a format that can be used by a GemFire and HAWQ import Spring-XD stream.

Using Spring-XD a real life example - Part 1

Market Based approach to Micro-Service delivery in Pivotal Cloud Foundry Part II

2014-11-05T09:22:00.002-05:00

Foundational Concepts behind a Market Based Approach

Measure, measure, measure

Effective markets are measured by their ability to provide products that are needed, competition to encourage innovation, and the right amount of regulation. The way to know if a Market is working is to measure and perform analysis of these factors. Feedback on how effective the systems are is essential for success.

Create an effective marketplace

Markets that work well are effective in providing the environment by which products can be bought and sold with a varying degree of "friction." Friction in markets could be defined as the regulatory or governance framework that is put into place to manage the market. Removing this layer would create a scenario where fake or inherently low quality products could make it into the market and create havoc. Too much friction in markets have been shown to increase the barriers to entry and eventually create "Black Markets" where products are sold without regard to the established market because it is ineffective.

Enable, not discourage innovation

Markets are built for innovation in that new practices and products will be adopted based upon their merit, but only if the market is "fair." A market based approach implements this by attaching a "cost" to the consumption of a Product/Micro-Service. This would be similar to the concepts of a Service Level Agreement that many organizations create between groups, but much simpler. Each micro-service establishes a cost structure that would "fund" their development and support efforts thereby providing a very easy way for organizations to understand where their costs will be in this new way of operating.

To enable innovation Markets must have a way to implement a "Low Barrier To Entry." This is where a product can be implemented quickly on a small scale, and then over time after demand picks up the product can be scaled and priced in a way that is appropriate. Conversely, if the product fails to garner attention it can be removed quickly and easily from the environment without creating a large impact to the organization. An effective Market enables innovation by allowing products to fail as well as succeed.

Cost Drives Innovation

What can be seen in markets is the basic supply and demand scenarios that drives cost lower (and higher). This is very similar to the Legacy services that are offered by many organizations. It is difficult to realize the cost to maintain these services beyond their true lifetimes because the users of the service do not understand the costs associated with the use of it. Also, by hiding all of complexity behind the service in an abstraction there is not an understanding of the underlying impact of the maintenance of this service.

Market Based approach to Micro-Service delivery in Pivotal Cloud Foundry Part I

2014-11-04T11:45:00.003-05:00

As part of my job as a Field Engineer for Pivotal the past few months has been spent doing implementations of Pivotal Cloud Foundry our Platform As a Service platform. As part of the discussion around this new "Third Platform" we talk quite a bit about the concept of MicroServices with our customers. This is because Micro-Services provide the avenue to innovation that most companies are looking for as we move into a world where the software produced at an organization defines it's value in the marketplace. To understand Micro-Services the best place to start is Martin Fowlers discussion around Micro Services.

Martin Fowler has some very interesting points, and provides a definitive concise delivery of what Mirco-Service concepts define and how it is being used by different companies. The fun part of the article from my perspective is that he does not define this as a "new" concept, and describes it mostly as an "evolutionary" concept, not a a revolutionary one. What is revolutionary in my opinion is that the realities of software development in large organizations is now recognized as an actual best practice.

Modularization of code becomes a Product and Micro-Service - I have seen this trend growing over time as quality demands have grown over time. Projects are driven at many organizations not by what it can be used for to solve the present problem, but by the value that it can give to other software that can integrate with it. Modularization into libraries can only take a team so far when the cost to change the code underlying the library is very high. A product can have different versions, can have a support structure.
Code responsibility in production given to the Software Development teams - What can I say? Martin Fowler's point hits home as the biggest incentive for improvement is to be the one who gets called at 2:00 AM when there is a problem. Many organizations have not recognized this fact, or have not come up with appropriate ways of measuring quality and Total Cost of Ownership to take this into account.
Removal of the importance of Canonical Domain Models - This points to the endless discussions organizations have around data, and views of data. This is a recognition that the "nature" of data is changing so quickly that trying to capture exactly where it is and what it means may just be impossible. It reminds me of the Heisenberg uncertainty principle which states that you can never simultaneously know the exact position and the exact speed of an object.

What all of these concepts have in common is that they are loosely based on social science, specifically the use of Market Economics.

What is a Market Based Approach to Micro-Services?

This approach is based around the economics of Products in a market economy. One of the main concepts of Micro-Services is that each service is a Product. The development teams are divided into Product Groups where they are responsible for the Micro-Services from "cradle to grave."

The users of the services will:

Invest in the Micro-Service by using the Product
Punish a Product by either not using it or "recalling" the product due to poor performance/functionality.

The producers of the services will:

Charge a specific cost for the use and support of the Product/Micro-Service
Continually produce new and upgraded services based upon the economics of developing the service.

Part II - Foundational Concepts in a Market Based Approach

Using Spring-XD a real life example - Part 1

2014-11-04T10:29:00.000-05:00

Introduction

I have been spending quite a bit of time working on Spring-XD and Hadoop over the past few months doing a proof of concept around high speed ingestion and transformation of files from a Linux file system to the Hadoop File System (HDFS).

The Business Use Case

In many situations, time series data is fed from a source on a continuous basis, whether that data is sensor data from a device, customer data from a feed, Tweets from Twitter, or Facebook Posts, or health claims data that comes from an external source. In the past this data would be staged, ingested, and then transformed using a tool like Informatica or Microsoft BizTalk. The data would then be placed into a relational database and then used for analytics.

There are two types of data for this use case, data that "batched" up into very large files (20 gigabytes+) that are received on a periodic basis such as 15 minutes or an hour. The second type of data is received as a stream of data and will need to be processed continually to show the results of the analysis.

The Technical Use Case

A Spring-XD cluster was setup to process very large files coming from devices at the 15 minute interval. These files are of consistent format and range from 500 megabytes to 20 gigabytes in size. The process was setup to use Spring-Batch to the files from XML to .csv and then put the data into different databases such as HBase, Hive, GemFire-XD, and Hawq.

The interesting part of this was that processing the file using Spring-Batch alone is just not possible under normal circumstances. Spring-XD enables the processing of each file to be easily be shared across a larger number of machines.

The Solution

We decided to use a combination of Spring-XD, RabbitMQ, Redis, ZooKeeper, GemFire-XD, HAWQ, and Pivotal Hadoop (Pivotal HD). In addition, a relational database (HSQLDB), Spring Framework, Map Reduce, HBASE, and Spring-HATEOS are used in this solution. This combination was used to implement both a real-time and batch system to integrate the ingestion process into Hadoop and HAWQ.

Products/Frameworks Used

Spring XD - A server that is used to abstract the mechanisms behind ingestion, processing, and depositing data. The ingestion step is called a "Source", a processor is called a "Transform", and a deposit is called a "Sink." There are many components included in Spring-XD that can be used for processing data.
RabbitMQ - An AMQP message broker that is used in this case to assist in Spring-XD job management and as a source, transform, and sink.
Redis - An object caching server that is used to hold statistics for Spring-XD.
ZooKeeper - A node based database that holds what has been deployed and what could be deployed for Spring-XD. It helps Spring-XD to keep a constant view of what should be running and what should be deployed to each instance of Spring-XD node.
GemFire-XD - An SQL 2003 compliant in memory data grid (IMDG) that is used to hold a window for fast querying of data. This is used as a sink in this case, but could be used as a source as well. It is used a the source for a REST services layer. This database has the ability to overflow and write to HDFS using OLTP type semantics.
HAWQ - HAdoop With Query is an analytical database query engine built on top of HDFS. The query engine originated with GreenPlum but has been built to enable a fully SQL compliant engine for Hadoop.
Pivotal Hadoop (HD) - A version of open source Hadoop that has been hardened and certified for use in an enterprise. This version of Hadoop has the ability to take advantage of HAWQ and GemFire-XD.
HSQLDB - A simple relational database used with Spring-XD. It would be wise to use your preferred relational database.
Spring-Framework - A comprehensive Java framework that provides a comprehensive programming and configuration model for modern Java-based enterprise applications - on any kind of deployment platform.
Map Reduce - A fundamental part of the Hadoop ecosystem, Map Reduce provides a Java Framework for parallel processing of data. For certain use cases including processing large files Map Reduce is a great solution.
HBASE - A no-sql database which runs on Hadoop, HBASE is a very powerful tool to use for time-series data and querying.
Spring HATEOS - A framework that can be used with Spring-REST to expose a database in a resource based manner.

Using Spring-XD a real life example - Part 2

Presenting "Moving from JEE to Cloud Foundry a Fruitful Journey"

2014-05-19T12:02:00.000-04:00

Just in case anyone is interested I will be presenting "Presenting "Moving from JEE to Cloud Foundry a Fruitful Journey" at the Tampa Java Users Group May 20th, and at the Orlando Java User Group May 22nd.

Moving from JEE to Pivotal Cloud Foundry PAAS a fruitful journey

2014-05-18T15:47:00.000-04:00

Introduction

This set of blog posts will go through the important concepts behind Platform As A Service (PAAS) and how Cloud Foundry works to make Java Development and software development in general faster and more productive. We will first go through some of the motivations behind moving to PAAS from the traditional application server approach. We will walk through developing an application and show the differences and advantages to using Cloud Foundry over other approaches. In addition the blog entries will show the pre-requisites for using Cloud Foundry, and the steps to convert a JEE application built with JMS into one that can be enabled on the cloud using open source frameworks and RabbitMQ.

This set of blog posts will not be providing a tutorial on Cloud Foundry, or PAAS in general. That has been done very well in other blogs and on the Cloud Foundry web site. The foremost goal here is to enable developers to quickly understand the important high-level problems that PAAS and Cloud Foundry solves, and provide the conceptual background for taking the next step and either installing the product, or evaluating the product on our public Cloud Foundry site. First though, we need to understand why using a PAAS is important to software development and most software developers.

A little bit of history

If we look back a little over 10 years to see what were the software development trends software development struggled to deploy web applications to a server in a standard and managed way. The paradigm was that you published web applications that typically were low bandwidth and a small part of everyday business for most companies large or small. Still, these application servers gave rudimentary ability to scale applications and provide an ecosystem for developers to publish their applications. Application servers were dedicated to the style and ecosystem of a specific vendor's tools and recommended architecture. This spanned a large area, from IBM's WebSphere which concentrated on working with their mainframes and MQSeries, to JBOSS which concentrated on strict J2EE adherence to Apache Tomcat. In this time open source projects like Apache Tomcat were in their infancy.

Targeting Software Development for Cloud and PAAS

The "Cloud" environment that has been created over the past couple of years has been placed into the public where we thing of Amazon and Google as the cloud whereas the reality is that these environments are basically no more convenient than bare metal machines.

They are nice to be able to have a cloud environment that you can call on new machines and you can duplicate those virtual machines anytime that you want,.
The real advantage to cloud is not just that but is what is called Infrastructure as a Service (IAAS).

PAAS environment == Platform as a Service

PAAS implies many things to different vendors and groups. What I understand it to be is that you have a set of server components that you can deploy applications to. The focus is not on the provisioning of the servers or the virtualized environment, but on the applications. Why is that important? Well, when we start thinking about PAAS or increasing the amount of time we can have doing the development rather than provisioning, and decreasing the time it takes to provision the environment, we really see that decreasing the time that we are spending on infrastructure we can spend more on actually developing applications and solving business problems.

De-Coupling the Application from the Application Server

PAAS also implies some separation from the Application Server that executes the application. This is really about creating Application Server agnostic applications so that our applications can work beyond WebSphere or Oracle BEA. Over the past 10 years developers for many reasons good and bad wrote code that tightly coupled their applications to the Java Application Server and to the database that was being used for their application. This was not necessarily a bad thing for numerous reasons:

Performance is always important, and some Application Servers were faster than others on certain hardware platforms.
Why re-create the wheel if it was already done in the Application Server?
Application Servers were sold as a "platform" by which you could depend on for years to come.

As in any other situation where code stays around for a long time and the intention of the framework changes, this proposition can be difficult, if not impossible to keep. The advantage of using non-standard features within a JEE application server was very popular a few years ago when Open Sources was not as big of a force as it is today.

Also, when we look at the options today for Application Servers, they are basically "Commodity Services." The "technical" advantages of going with a commercial Application Server should only extend to the support that a vendor will supply if there is a problem. Each situation is different, and there are always situations where it makes sense to use such extensions. What is a smart way to use those Application Server extensions if they are not covered by an established standard?

Write your own adapter to "abstract" the service away from your code, and separate the abstraction into a separate deliverable. An example would be if the commercial cache vendor does not supply a adapter class to implement JSR 107.
Use the SpringFramework (or any other injection framework) to inject an interface that implements the functionality.

Developer Notes I

JEE components that use RMI such as "Session Beans" do not work at all in Cloud Foundry due to the restrictions that have been placed upon the virtual machines. If you have to use session beans, re-develop them to be REST based services or use a messaging environment like RabbitMQ.
JMS message drive beans (MDB) that are tied to a message service that is embedded locally in the application server will not work in Cloud Foundry due to the restriction that it is tied to a specific instance of the messaging environment. If asynchronous messaging is absolutely needed take a look at other solutions that work in Cloud Foundry such as RabbitMQ and Redis.

Why is it important?

To be able to take advantage of the best open-source technologies available today and into the future development groups must be flexible enough to be polyglot in environments not just in languages. Also, to enable the application to scale horizontally and vertically at run-time the application should not be bound by a specific implementation of server interface such as messaging, database, or services.

It is really a challenge when we as developers, write applications tied to a specific Application Server and/or a specific implementation of IAAS. This "tight coupling" of our code to a specific Application Server creates issues when one of these dependencies change or becomes un-supported.

Generally, these items are important to any development process, but especially when we are developing a system that will be deployed into a PAAS:

Where we store our data - Instances of the application can come an go, data can't be stored locally to be used for any kind of permanent storage. Databases, NAS, or service caches are the preferred method of storing data when you use a PAAS.
Where we are retrieving our data - Databases and Cache's must be able to handle initial creation of tables or maintenance of the cache of data through the application itself. Retrieving data from the local filesystem is not recommended.

Where we to run our applications from a file-system perspective - As filesystems are transient all file system references should be relative paths.
What does the application depend on - What system dependencies are part of the application environment?
What enables the application to scale horizontally, vertically or both - Does the application enable scaling, or is it accomplished using the PAAS?

Developer Notes ||

All of the above issues are part of the reason why Cloud Foundry was created to begin with.
Cloud Foundry assumes that you application has some standard property file that you use for configuring your application for each environment (prod, dev, qa) which can be placed into the configuration of the PAAS.
Cloud Foundry does not have the facility for storing files on the local drive and enabling it to be used in the next execution of the application. It is based around the concept that you can have many instances of the application running, and it may not return to the same instance if you leave some kind of "state" behind in the local filesystem. In these cases you really should be using a relational database or something like Redis or RabbitMQ.
In most Platform As a Service Offerings including Cloud Foundry scaling horizontally and/or vertically should always be done within the PAAS itself. Threading within the application is not as important as making sure that the application is completely stable.

Why does it matter? (or why should I care?)

System Dependencies are Ubiquitous

System dependencies are resources that you either have chosen to use, or have been given to you to use for your application to perform basic functionality that enables your application to work within a deployed environment. Examples of these kinds of dependencies are:

Relational database that is not embedded in your application.
Distributed cache to handle failover of your application
Load balancer that sits in front of other dependencies
Centralized authentication mechanism such as LDAP or Active Directory.
Messaging environment such as JMS or RabbitMQ that is not embedded in your application.
Other external applications or services that you do not control.

With the trends progressing today, even if your application does not interact with these kinds of dependencies, if it is deployed into a company with enough size and complexity your application will need to adapt to handle these dependencies.
This really isn’t that much to think about, but who has requirements for these kinds of applications today?

It seems from my experience lately that the days are in the past where customers have just a website with some information to share with their users.
Typically, even if we start talking about doing applications without data management dependencies the users still have content management systems that work to integrate other enterprise systems.

In this manner, what we presume as developers to be the simplest of web applications can very quickly start to look like a full-fledged complex web application by having extended complexity that was not assumed at the start. Therefore, in reality there is no such thing as a web application these days without any dependencies or future complexity, so either we as software development teams need to write these mechanisms into our applications, or we use a tool such as PAAS (and injection frameworks like Spring Framework) to help us along the way.

Developer Notes III

Dependencies are handled very elegantly in Cloud Foundry by the use of CF Services.

Services are enabled across all users in Cloud Foundry and are typically implement a multi-tennant security model.
Services are bound specifically to applications and are configured at deploy-time or at development time using configuration classes. This works with JVM based languages such as Scala and Java.
Environment variables are used to get and set the values within the application so the application can be easily deployed to a local application server and to Cloud Foundry.

Complexity in applications are assumed in Cloud Foundry, but not applied until it is absolutely necessary as you can move from a local JDBC database to an enterprise database with just a change in configuration if you adhere to some basic precepts:

The database must have the ability to create itself using scripts executed within the application.
Default data for the database must have the ability to populate itself using scripts executed within the application.
Non-standard SQL should be avoided if possible.

Application development teams are always striving to put together web applications together that can handle interacting with the resources that we did not expect at the beginning of the project. To accomplish this, we need to have ways to include those resources without having drastic impact to our applications should those services have problems such as:

Resources/dependencies become overloaded
Resources/dependencies are inaccessible due to network failure
Resources/dependencies are non-responsive

The concepts behind PAAS from a developer perspective is truly to make your life easier when it comes to these "System Level" dependencies that are totally out a programmer's hands. My personal experience has always been that by virtue of being in a "development" environment I don't have the ability to truly test the system that I'm working on at scale with all of the services that I would typically have in either my QA or Production environment. A well implemented PAAS offers interfaces to services that implement those interfaces, and a way to easily scale and test your development system.

Deployment is hard

One of the biggest challenges for most software development organizations is to actually deploy their code into their different environments. There are many tools that I would recommend for this including Bamboo and Jenkins. These tools can help to put your process in order, but can also just add another set of complexity to the mix.

Cloud Foundry simplifies the journey of the application from initial development inception to putting it into production by enabling developers and developer operations to have a fully scriptable deployment environment to use as a basis for development, quality assurance, and production deployment.
Deployment and promotion from one environment to the next becomes seamless, automated and a trusted process instead of a manual, tricky one.

Solution Framework

To solve these kinds of issues in the past (prior to PAAS) you would you go ahead and put together an application architecture where you have a few choices:

Deployment is hard

Create scriptable environments using Puppet and Chef to automate the creation of different environments.
Use non-standard complex configuration management software and processes that are restrictive to the component development process.
Employ a group of people that all they do is deploy software.

System Dependencies

Hardcode the dependencies into your application
Put in place someplace in your environment where those things are and then when your connection to those things these resources really (using the Service Locator Pattern).
Use a service registry such as UDDI to find and locate your System Dependencies and use JNDI to return a reference to that service.
Put in place a set of abstractions in your application that would allow you to "inject" using an injection framework such as Spring Framework to create abstractions around your dependencies.

Horizontal and Vertical Scaling

Enable multi-threading within your application, thereby increasing complexity and the likely case that it will not be maintainable. This will only enable vertical scaling.
Enable application to be stateless and multi tenant. Deploy to an established number of servers, and number of instances of the application. This architecture allows for the the maximum number of users, and is not efficient with infrastructure resources as it uses a steady amount of resources all of the time.
Enable distributed processing - Deploy a distributed cache environment that can scale vertically and horizontally (similar to Gemfire).

As Cloud Foundry becomes part of the development process:

Deployments are easy
Dependencies are abstracted away by Cloud Foundry, not by your code/application.
Horizontal and Vertical Scaling are done through Cloud Foundry, not your code/application.

Why Spring Boot, why now?

2014-05-16T09:28:00.000-04:00

I have been taking a very long look at the concept of a new architecture for developing Java applications. As many of you who have read my posts in the past have seen, I'm a big advocate of convention based development.

What is "Convention Based Software Development?" This is a development strategy that is based upon the concept that if you are going to typically use a specific way of working, it should be something that you can repeat without having to spend time re-doing it again.

To really take a look at Spring Boot you should understand that it's not going to do everything for you, but if you are going to use Cloud Foundry with Java it certainly makes it easier. To really simplify here is what you get:

No XML configuration files for Spring (unless you absolutely want one)
No web.xml, it's all done within Spring Boot!
Using annotations, Gradle and Spring Boot you can use any of the Pivotal Spring Framework projects without having to create very much configuration.
A plain and simple way to get started with any Java Spring Framework application, no fuss, no worry.

Pivotal Cloud Foundry Setup Notes

2014-04-26T11:46:00.001-04:00

I have been doing some Pivotal Cloud Foundry installs lately, here are some interesting things that I have run into:

Always use Chrome or Firefox, not Internet Explorer when you get to the point where you have to use the Operations manager.
If you are going to install MySQL you need to allocate another 200GB or more.
Make sure that you have at least a minimum of 32 IP addresses on the segment you will be deploying Cloud Foundry to.
If you are not going to deploy many applications you can reduce the amount of NFS storage that you need. The NFS storage is to hold the applications, not your actual data for your applications.
Make sure that you use the vCenter Portgroup for the network name when setting up the vSphere Network. Bosh uses this information to setup the Network Interface in Ubuntu. If you don't do this correctly the install will fail even though it validates correctly.
The NTP servers must be in exact synchronization with vSphere. One solution is to use the Cloud Controller as the NTP server by starting it within the installation VM before you install Pivotal Cloud Foundry.
If you run into problems installing Pivotal Cloud Foundry, especially if you have network issues it's much easier to just re-install from scratch then to try to fix it manually.
Make sure that you have a wildcard domain already created in your DNS for your Cloud Foundry installation. It does not say this exactly in the instructions but it won't work without it.
The wildcard you put into the DNS must match the domains that you put into your self signed cert or public cert. This is very important. It will get to the end of the install and fail if you don't do this.
The first user that is available to setup users is the UAA user. The credentials for this user is in the "Credentials" table in Operations Manager for the Elastic Runtime.
If you "push" an application to Cloud Foundry and it's taking a long time, or it seems to just not work when it is pushed delete the application and increase the memory for the application.

Important cf command line (cli) commands:

cf target https://.... This command has to come first as it points the CLI to the correct PCF instance
cf push - Don't be surprised if there is a .yml file in the directory it will use that file for pushing it.
cf tail --app tails the log file for all of the instances running the application
cf crashlogs --app this gives the crash log for when it is up and crashes.
If you are having trouble with pushing applications use cf push -t . The -t option gives you a nice trace.
cf delete --app deletes the app from cloud foundry

What do you do once you have PCF installed

First, you need to install the "CLI" -

http://docs.cloudfoundry.org/devguide/installcf/install-go-cli.html

Then, you need to target the cloud foundry instance

->cf target api.southeast1.fe.gopivotal.com

Setting target to https://api.southeast1.fe.gopivotal.com... OK

Then you need to login, using the administrator for UAA that you retrieve off of the "Credentials" tab in the Elastic Runtime from Pivotal Operations Manager.

->cf login

target: https://api.southeast1.fe.gopivotal.com
Email> jdavis
Password> *******
Authenticating... OK
1: org1
2: org2
3: jdavis-org
Organization> 3
Switching to organization jdavis-org... OK
1: development
2: production
3: staging
Space> 1
Switching to space development... OK

http://docs.cloudfoundry.org/adminguide/cli-user-management.html

In general, I would start taking a look at this documentation to learn more about operating PCF:

http://docs.cloudfoundry.org/

Java and Cloud Architecture Rambings

How to Start Developing with Cloud Foundry for JVM based Languages - Part 1

Intended Audience

A few words of wisdom

What to know before you start

Cloud Foundry Operating Environment

What are your expectations for the operating environment?

Introduction to the Cloud Foundry (CF) Operating Environment

An Example of Docker Multi-Host Networking With Hadoop Part 3 - Validating your Swarm Cluster

Validating The Docker Swarm Cluster

Assumptions:

Recommendations:

Pre-Validation steps:

Incorporate the .profile into your environment $. .profile or $source .profile

Validation Steps

Next Steps

An example Docker Swarm Multi-Host Lab

Why a Docker Multi-Host Lab?

Description of the Hosts:

Installed software

Ubuntu-dell:

Ubuntu-hp:

Ubuntu-jdavis:

Creating a startup script for Docker Swarm Manager

For Ubuntu-Server 14.04

Assumptions:

Create the Swarm manager configuration file:

An Example of Docker Multi-Host Networking with Hadoop - Part2

Next Steps - Add a host to your Cluster

Assumptions for adding a Host:

Adding a Host

Install your favorite Linux variant onto the host

Install these standard Linux packages

Install Docker Daemon

Install Consul (optional)

Install Go

Install Godep

Follow the instructions here.

Install Swarm

Start Swarm

Configure Docker Daemon service

Re-Start Docker service

Next Steps

Why do MicroServices projects fail to deliver?

What is the definition of a Micro Services Architecture?

Why are MicroServices projects so difficult to deliver?

- Think of Micro Services as a "Product" not a "Project."

Deployment is the big advantage or disadvantage

Why switch to a micro-service architecture if it's so difficult?

Conclusion

An example of Docker Multi Host Clustering Part 1

Ambari on Docker - using multiple physical hosts for Bare-Metal Docker Swarm Clustering

Bare-Metal Docker Swarm Clustering

Bare-Metal Docker Swarm Clusters are built using multiple physical hosts that have Docker and Swarm installed locally, not through Docker Machine.

What is needed to make this work?

Why do I use Consul for DNS?

Pre-requisites for each host

Optional Installs

Docker Compose - Same as Docker Swarm, you will want to copy the executable into /usr/bin.

Prepare the hosts

Running Ambari and Hadoop using Docker Swarm

Using Docker as a PAAS and a comparison to Pivotal Cloud Foundry

Introduction

So, what is this PAAS thing?

Docker by itself is not a PAAS

What is missing?

On-premise build, release, and object storage

Comparison to Cloud Foundry

Implementation Complexity

Ease to Deploy and use for Developers

Ease to deploy for Operations on-premise

Usability in the Cloud

Functionality

Summary

Resources/links

Stackato PAAS and contrasts to Pivotal Cloud Foundry

Installing the Stackato Development Environment

Deploying an Application to the Stackato PAAS

Services

Management

Incorporate the .profile into your environment

$. .profile
or
$source .profile