At The Server Labs, we often run into a need that comes naturally with the on-demand nature of Cloud Computing. Namely, we’d like to keep our team’s AWS bill in check by ensuring that we can safely turn off our Amazon EC2 instances when not in use. In fact, we’d like to take this practice one step further and automate the times when an instance should be operational (e.g. only during business hours). Stopping an EC2 instance is easy enough, but how do we ensure that our data and configuration persist across server restarts? Fortunately, there are a number of possible approaches to solve the persistence problem, but each one brings its own pitfalls and tradeoffs. In this article, we analyze some of the major persistence strategies and discuss their strengths and weaknesses.

A Review of EC2

Since Amazon introduced their EBS-backed AMIs in late 2009 [1], there has been a great deal of confusion around how this AMI type impacts the EC2 lifecycle operations, particularly in the area of persistence. In this section, we’ll review the often-misunderstood differences between S3 and EBS-backed EC2 instances, which will be crucial as we prepare for our discussion of persistence strategies.

Not All EC2 Instances are Created Equal

An Amazon EC2 instance can be launched from one of two types of AMIs: the traditional S3-backed AMI and the new EBS-backed AMI [2]. These two AMIs exhibit a number of differences, for example in their lifecycle [3] and data persistence characteristics [4]:

As explained in the table above, EBS-backed EC2 instances introduce a new stopped state, unavailable for S3-backed instances. It is important to note that while an instance is in a stopped state, it will not incur any EC2 running costs. You will, however, continue to be billed for the EBS storage associated with your instance. The other benefit over S3-backed instances is that a stopped instance can be started again while maintaining its internal state. The following diagrams summarize the lifecycles of both S3 and EBS-backed EC2 instances:

Lifecycle of an S3-backed EC2 Instance.

Lifecycle of an EBS-backed EC2 Instance.

Note that while the instance ID of a restarted EBS-backed instance will remain the same, it will be dynamically assigned a new set of public and private IP and DNS addresses. If you would like assign a static IP address to your instance, you can still do so by using Amazon’s Elastic IP service [5].

Persistence Strategies

With an understanding of the differences between S3 and EBS-backed instances, we are well equipped to discuss persistence strategies for each type of instance.

Persistence Strategy 1: EBS-backed Instances

First, we’ll start with the obvious choice: EBS-backed instances. When this type of instance is launched, Amazon automatically creates an Amazon EBS volume from the associated AMI snapshot, which then becomes the root device. Any changes to the local storage are then persisted in this EBS volume, and will survive instance failures and restarts. Note that by default, terminating an EBS-backed instance will also delete the EBS volume associated with it (and all its data), unless explicitly configured not to do so [6].

Persistence Strategy 2: S3-backed Instances

In spite of their ease of use, EBS-backed instances present a couple of drawbacks. First, not all software and architectures are supported out-of-the-box as EBS-backed AMIs, so the version of your favorite OS might not be available. Perhaps more importantly, the EBS volume is mounted as the root device, meaning that you will also be billed for storage of all static data such as operating systems files, etc., external to your application or configuration.

To circumvent these disadvantages, it is possible to use an S3-backed EC2 instance that gives you direct control over what files to persist. However, this flexibility comes at a price. Since S3-backed instances use local storage as their root device, you’ll have to manually attach and mount an EBS volume for persisting your data. Any data you write directly to your EBS mount will be automatically persisted. Other times, configuration files exist at standard locations outside of your EBS mount where you will still want to persist your changes. In such situations, you would typically create a symlink on the root device to point to your EBS mount.

For example, assuming you have mounted your EBS volume under /ebs, you would run the following shell commands to persist your apache2 configuration:

# first backup original configuration
mv /etc/apache2/apache2.conf{,.orig}
# use your persisted configuration from EBS by creating a symlink
ln –s /ebs/etc/apache2/apache2.conf /etc/apache2/apache2.conf

Once your S3-backed instance is terminated, any local instance storage (including symlinks) will be lost, but your original data and configuration will persist in your EBS volume. If you would then like to recover the state persisted in EBS upon launching a new instance, you will have to go through the process of recreating any symlinks and/or copying any pertinent configuration and data from your EBS mount to your local instance storage.

Synchronizing between the data persisted in EBS and that in the local instance storage can become complex and difficult to automate when launching new instances. In order to help with these tasks, there are a number of third-party management platforms that provide different levels of automation. These are covered in more detail in the next section.

Persistence Strategy 3: Third-party Management Platforms

In the early days of AWS, there were few and limited third-party platforms available for managing and monitoring your AWS infrastucture. Moreover, in order to manage and monitor your instances for you, these types of platforms necessarily need access to your EC2 instance keys and AWS credentials. Although a reasonable compromise for some, this requirement could pose an unacceptable security risk for others, who must guarantee the security and confidentiality of their data and internal AWS infrastructure.

Given these limitations, The Server Labs developed its own Cloud Management Framework in Ruby for managing EC2 instances, EBS volumes and internal SSH keys in a secure manner. Our framework automates routine tasks such as attaching and mounting EBS volumes when launching instances, as well providing hooks for the installation and configuration of software and services at startup based on the data persisted in EBS. It even goes one step further by mounting our EBS volumes using an encrypted file system to guarantee the confidentiality of our internal company data.

Today, companies need not necessarily develop their homegrown frameworks, and can increasingly rely on third-party platforms. An example of a powerful commercial platform for cloud management is Rightscale. For several of our projects, we rely on Rightscale to automatically attach EBS volumes when launching new EC2 instances. We also make extensive use of scripting to install and configure software onto our instances automatically at boot time using Rightscale’s Righscript technology [7]. These features make it easy to persist your application data and configuration in EBS, while automating the setup and configuration of new EC2 instances associated with one or more EBS volumes.

Automating Your Instance Uptimes

Now that we have discussed the major persistence strategies for Amazon EC2, we are in a good position to tackle our original use case. How can we schedule an instance in Amazon so that it is only operational during business hours? After all, we’d really like to avoid getting billed for instance uptime during times when it is not really needed.

To solve this problem, we’ll have to address two independent considerations. First, we’ll have to ensure that all of our instance state (including data and configuration) is stored persistently. Second, we’ll have to automate the starting and stopping of our instance, as well as restoring its state from persistent storage at boot time.

Automation Strategy 1: EBS-backed Instances

By using an EBS-backed instance, we ensure that all of its state is automatically persisted even if the instance is restarted (provided it is not terminated). Since the EBS volume is mounted as the root device, no further action is required to restore any data or configuration. Last, we’ll have to automate starting and stopping of the instance based on our operational times. For scheduling our instance uptimes, we can take advantage of the Linux cron service. For example, in order to schedule an instance to be operational during business hours (9am to 5pm, Monday-Friday), we could create the following two cron jobs:

0 9 * * 1-5 /opt/aws/bin/ec2-start.sh i-10a64379
0 17 * * 1-/opt/aws/bin/ec2-stop.sh i-10a64379

The first cron job will schedule the EBS-backed instance identified by instance ID i-10a64379 to be started daily from Monday to Friday at 9am. Similarly, the second job schedules the same instance to be stopped at 5pm Monday through Friday. The cron service invokes the helper scripts ec2-start.sh and ec2-stop.sh to facilitate the configuration of the AWS command-line tools according to your particular environment. You could run this cron job from another instance in the cloud, or you could have a machine in your office launch it.

The following snippet provides sample contents for ec2-start.sh, which does the setup necessary to invoke the AWS ec2-start-instances command. Note that this script assumes that your EBS-backed instance was previously launched manually and you know its instance ID.

#!/bin/bash
# Name: ec2-start.sh
# Description: this script starts the EBS-backed instance with the specified Instance ID
# by invoking the AWS ec2-start-instances command
# Arguments: the Instance ID for the EBS-backed instance that will be started.

export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20
export EC2_HOME=/opt/aws/ec2-api-tools-1.3
export EC2_PRIVATE_KEY=/opt/aws/keys/private-key.pem
export EC2_CERT=/opt/aws/keys/cert.pem
# uncomment the following line to use Europe as the default Zone
#export EC2_URL=https://ec2.eu-west-1.amazonaws.com
PATH=$PATH:${EC2_HOME}/bin
INSTANCE_ID=$1

echo "Starting EBS-backed instance with ID ${INSTANCE_ID}"
ec2-start-instances ${INSTANCE_ID}

Similarly, ec2-stop.sh would stop your EBS-backed instance by invoking ec2-stop-instances followed by your instance ID. Note that the instance ID of EBS-backed instances will remain the same across restarts.

Automation Strategy 2: S3-backed Instances

Amazon instances backed by S3 present the additional complexity that the local storage is not persistent and will be lost upon terminating the instance. In order to persist application data and configuration changes independently of the lifecycle of our instance, we’ll have to rely on EBS. Additionally, we’ll have to carefully restore any persisted state upon launching a new EC2 instance.

The Server Labs Cloud Manager allows us to automate these tasks. Among other features, it automatically attaches and mounts a specified EBS volume when launching a new EC2 instance. It also provides hooks to invoke one or more startup scripts directly from EBS. These scripts are specific to the application, and can be used to restore instance state from EBS, including any appropriate application data and configuration.

If you must use S3-backed instances for your solution, you’ll either have to develop your own framework along the lines of The Server Labs Cloud Manager, or rely on third-party management platforms like Rightscale. Otherwise, EBS-backed instances provide the path of least resistance to persisting your instance data and configuration.

Automation Strategy 3: Rightscale

Rightscale provides a commercial platform with support for boot time scripts (via Righscripts) and automatic attachment of EBS volumes. In addition, Rightscale allows applications to define arrays of servers that grow and shrink based on a number of parameters. By using the server array schedule feature, you can define how an alert-based array resizes over the course of a week [8], and thus ensure a single running instance of your server during business hours. In addition, leveraging boot time scripts and the EBS volume management feature enables you to automate setup and configuration of new instances in the array, while persisting changes to your application data and configuration. Using these features, it is possible to build an automated solution for a server that operates during business hours, and that can be shutdown safely when not in use.

Conclusion

This article describes the major approaches to persisting state in Amazon EC2. Persisting state is crucial to building robust and highly-available architectures with the capacity to scale. Not only does it promote operational efficiency by only consuming resources when a need exists; it also protects your application state so that it if your instant fails or is accidentally terminated you can automatically launch a new one and continue where you left off. In fact, these same ideas can also enable your application to scale seamlessly by automatically provisioning new EC2 instances in response to a growth in demand.

References

[1] New Amazon EC2 Feature: Boot from Elastic Block Store. Original announcement from Amazon explaining the new EC2 boot from EBS feature.

[2] Amazon Elastic Compute Cloud User Guide: AMI Basics. Covers basic AMI concepts for S3 and EBS AMI types.

[3] The EC2 Instance Life Cycle: excellent blog post describing major lifecycle differences between S3 and EBS-backed EC2 instances.

[4] Amazon Elastic Compute Cloud User Guide: AMIs Backed by Amazon EBS. Learn about EBS-backed AMIs and how they work.

[5] AWS Feature Guide: Amazon EC2 Elastic IP Addresses. An introduction to Elastic IP Addresses for Amazon EC2.

[6] Amazon Elastic Compute Cloud User Guide: Changing the Root Volume to Persist. Learn how to configure your EBS-backed EC2 instance so that the associated EBS volume is not deleted upon termination.

[7] RightScale User Guide: RightScripts. Learn how to write your own RightScripts.

[8] RightScale User Guide: Server Array Schedule. Learn how to create an alert-based array to resize over the course of the week.

Recently, we have been working quite extensively with Apache Servicemix, a JBI-compliant ESB.
One of areas we have been looking into is securing services in general and how to perform ldap-based authentication and authorization for those services in particular.

A good starting point to understand Servicemix (SMX) security features can be found here. I will give a brief overview of the security features of JBI and SMX .

JBI does not very concisely define authentication and authorization mechanisms, but it does define the following:

• JBI relies on JAAS definitions for security features (i.e. security Subject).
• The Binding Components (e.g. servicemix-cxfbc) are expected to transform protocol-specific (e.g. WS-Security) credentials into a normalized format, a security Subject, and inject them into the normalized message.
• The ESB will then apply the authentication and authorization mechanism based on Subject information.

In particular for SMX we have:

• For authentication it relies on JAAS, being able to use custom Login modules to integrate different technologies (e.g. LDAP) and implement specific authentication logic. The SMX configuration for JAAS is located under $SERVICEMIX_HOME/conf/login.properties.
• SMX comes by default with a file based authentication mechanism, which is not suitable for use in many environments since it requires an SMX restart after any change.
• For authorization, SMX does not use JAAS, due to the different nature of the items to be authorized (i.e. service endpoints instead of Java code). It defines an specific ACL AuthorizationMap, used by the NormalizedRouter (SecureBroker) to allow or deny the invocation. The configuration file for this is located in $SERVICEMIX_HOME/conf/security.xml
• Also in this case, SMX comes by default with a file based ACL definition, included in the security.xml file. Any change in the ACLs requires an SMX restart.

So the idea behind this post is to provide a dynamic authentication/authorization mechanism in SMX that does not require an ESB restart after changes whilst also making use of LDAP.

In this post I will go through a complete example that requires the following:

1. Apache ServiceMix 3.3.2.
2. OpenDS 2.2 as a test LDAP Server. However you can use any other LDAP server you prefer.
3. Maven 2.0.9 or higher to build the projects.

The complete source code of this post can be found here. It contains:

1. A LDAP ldif file with the sample directory structure used.
2. As a test service, a secured version of the SMX’s cxf-wsdl-first, now called cxf-wsdl-first-secure.
3. A smx-ldap maven project with the classes we need to deploy in SMX in order to enable LDAP.
4. A smxConfig directory with the required SMX configuration.

With that in mind, let’s go through the details:

Install Servicemix and OpenDS

You can skip the OpenDS section if you have your own LDAP server, just make sure the relevant LDAP configuration parameters are used in the coming sections.

1. Download and install Servicemix. No additional modifications are required at this stage.
2. Download and install OpenDS. If you want to use the sample LDAP ldif file, setup the directory server with the following configuration:
• Select as “Directory Base DN”: dc=theserverlabs,dc=com.
• Select import data from LDIF and choose the file included in the source package.
• Complete installation.

The users and passwords defined in the sample ldif are:

• reader:reader
• smx:smx
• testuser:testuser

Create a test service that requires authentication

To show the authentication and authorization of a service deployed in SMX I took the “cxf-wsdl-first” example that is included in the SMX distribution and enabled WS-Security authentication headers. The result project is included in the source files as “cxf-wsdl-first-secure”.

For that I just added the following interceptors configuration into the xbean.xml file of the CXFBC service unit “wsdl-first-cxfbc-secure-su”:

 
<cxfbc:consumer wsdl="classpath:person.wsdl"
                        targetService="person:PersonService"
                        targetInterface="person:Person">
     <cxfbc:inInterceptors>
             <bean class="org.apache.cxf.interceptor.LoggingInInterceptor" />
             <ref bean="UserName_RequestIn" />
      </cxfbc:inInterceptors>
</cxfbc:consumer>
 
 
<bean class="org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor"  id="UserName_RequestIn">
     <constructor-arg>
         <map>
		<entry key="action" value="UsernameToken" />
		<!--
			No real implementation of PasswordCallbackClass is required, thus the
	    	        dummy class. However it must be defined. This is due to the fact
			that the SMX will take care of the authentication, not WSS4J.
		-->
		<entry key="passwordCallbackClass" value="com.tsl.jbi.smx.security.DummyCallbackHandler" />
		<entry key="passwordType" value="PasswordText"/>
	</map>
    </constructor-arg>
</bean>

In this case as you see the required passwordCallbackClass just defines a DummyCallbackHandler (also included in the cxfbc su) that does nothing at all, delegating the whole authentication/authorization to SMX.
This is the simplest case and obviously you could have a class that performs additional checks or needs to deal with keystore passwords in case encryption is used. I will not get into the details here about WSS4J as this can be a subject worth a whole entry by itself.

Also, for a very good background reading of WS-Security and SMX check this blog post by Torsten Mielke.

To install the example project into SMX, just build the maven project and copy the service assembly to the hotdeploy directory of SMX:

cd Source/cxf-wsdl-first-secure/
mvn clean install
cp  wsdl-first-cxf-secure-sa/target/wsdl-first-cxf-secure-sa-3.3.2.zip $SERVICEMIX_HOME/hotdeploy

An example soap request with the WS-Security headers would be:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:typ="http://servicemix.apache.org/samples/wsdl-first/types">
   <soapenv:Header>
       <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
            <wsse:UsernameToken wsu:Id="UsernameToken-9" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
                   <wsse:Username>testuser</wsse:Username>
                   <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">testuser</wsse:Password>
            </wsse:UsernameToken>
       </wsse:Security>
   </soapenv:Header>
   <soapenv:Body>
      <typ:GetPerson>
         <typ:personId>world</typ:personId>
      </typ:GetPerson>
   </soapenv:Body>
</soapenv:Envelope>

Setup the required LDAP classes for JAAS and SMX AuthorizationMap

The required JAAS LDAP Login Module and the SMX AuthorizationMap contains many LDAP connectivity logic that are common and can be found in several open-source implementations.
I took as a reference the ActiveMQ classes and modify them to fit the SMX needs.

You can find the two classes in the smx-ldap maven project:

• LdapLoginModule: This class implements the JAAS LoginModule interface and it is responsible to perform the authentication (login method) based on the passed credentials. It is also responsible to extract the user’s roles once it is authenticated and inject them into the Subject as additional principals. The provided class is basically the same implementation used for ActiveMQ, no changes are required.
• LdapAuthorizationMap: This class implements the AuthorizationMap interface offered by SMX. The interface has just one method:

  Set<Principals> getAcls(endpoint, operation)

  The class simply provides a set of Principals allowed to invoke the passed endpoint and operation. Also in this case, I took as a baseline the LdapAuthorizationMap include in ActiveMQ. We need to adapt it as ActiveMQ maps work on queues and topics, not endpoints.

Having reached this point I needed to define the level of granularity and also the format of the entries in LDAP defining the ACLs. I provide one implementation possibility here and it is definitely not perfect. Depending on your needs you can take this as a baseline an implement a more complex logic, add caching of endpoints ACLs, etc…

So this is the format I defined:

• Endpoint ACLs are located under “ou=endpoints,ou=smx” in the LDAP tree.
• The entries defining the ACLs will be “GroupOfUniqueNames”, having an entry per allowed group.
• The “cn” of the ACLs has the form:

  cn=<Namespace>:<ServiceName>:<Operation>

  In the defined schema you can also use wildcards on Service and Operation. Examples of ACLs definition for the cxf-wsdl-first-secure project would be:

  cn=http://servicemix.apache.org/samples/wsdl-first:*:*   
         Grant permissions to all services on that ns.    
   
  cn=http://servicemix.apache.org/samples/wsdl-first:PersonService:*   
         Grant permissions to all operations of PersonService.    
   
  cn=http://servicemix.apache.org/samples/wsdl-first:*:GetPerson   
         Grant permissions to GetPerson operations of all services in that ns.

  Another example in case you use dynamic SMX ftp endpoints would be:

  cn=urn:servicemix:ftp:* 
         Grant permissions to all dynamic endpoints of the servicemix-ftp BC.

This logic is implemented in the LdapAuthorizationMap class.

Both ActiveMQ classes are highly configurable using properties as we will see how to setup their usage in SMX.

To install the classes, build the maven project and copy the jar file to the lib directory of SMX:

cd Source/smx-ldap/
mvn clean install
cp  cp target/smx-ldap-1.0-SNAPSHOT.jar $SERVICEMIX_HOME/lib
 
<Restart ServiceMix>

Configure SMX

To make use of the LDAP authentication and authorization two SMX files need to be modified:

• $SERVICEMIX_HOME/conf/login.properties: This is the JAAS authentication configuration and must define the LdapLoginModule. The parameters are for locally configured OpenDS LDAP, modify them with your configuration if required:

  servicemix-domain {
      com.tsl.jbi.smx.security.login.LDAPLoginModule REQUIRED
          debug=true
          initialContextFactory=com.sun.jndi.ldap.LdapCtxFactory
          connectionURL="ldap://localhost:389"
          connectionUsername="cn=reader,ou=people,dc=theserverlabs,dc=com"
          connectionPassword=reader
          connectionProtocol=s
          authentication=simple
          userBase="ou=people,dc=theserverlabs,dc=com"
          userRoleName=dummyUserRoleName
          userSearchMatching="(uid={0})"
          userSearchSubtree=false
          roleBase="ou=groups,dc=theserverlabs,dc=com"
          roleName=cn
          roleSearchMatching="(member={0})"
          roleSearchSubtree=true
          ;
  };

  You can see that there are two parameters, userSearchMatching and roleSearchMatching that allow us to define the user/roles search filter that fit our environment.

• $SERVICEMIX_HOME/conf/security.xml. This file contains general SMX security configuration and here we will define the bean for the LdapAuthorizationMap:

    <!-- Custom, LDAP based authorisation map -->
    <!-- MessageFormat used as LDAP filter to search for Roles of an Specific Endpoint -->
    <!-- {0} will be replaced by Endpoint Namespace, {1} by Endpoint Name and -->
    <!-- {2} by Endpoint operation -->
    <bean id="roleFilterMessageFormat" class="java.text.MessageFormat">
   <constructor-arg value="(|(|(|(cn={0}:{1}:{2})(cn={0}:{1}:\*))(cn={0}:\*:{2}))(cn={0}:\*:\*))" />
    </bean>
    <bean id="LdapAuthorizationMap" class="com.tsl.jbi.smx.security.LDAPAuthorizationMap">
      <property name="initialContextFactory" value="com.sun.jndi.ldap.LdapCtxFactory"/>
      <property name="connectionURL" value="ldap://localhost:389"/>
      <property name="connectionUsername" value="cn=reader,ou=people,dc=theserverlabs,dc=com"/>
      <property name="connectionPassword" value="reader"/>
      <property name="connectionProtocol" value="s"/>
      <property name="endpointSearchBase" value="ou=endpoints,ou=smx,dc=theserverlabs,dc=com"/>
      <property name="endpointRoleFilterFormat" ref="roleFilterMessageFormat"/>
      <property name="roleAttribute" value="uniqueMember"/>
    </bean>

  Also in this case, we have an endpointRoleFilterFormat that defines the ACL’s search filter. This can be modified to match whatever other LDAP directory setup you want to define for ACLs.

• $SERVICEMIX_HOME/conf/servicemix.xml. We need to modify the authorizationMap used by the secured broker and point to the new ldap-based map.

      <sm:broker>
        <sm:securedBroker authorizationMap="#LdapAuthorizationMap">
   
         .....
      </sm:broker>

Restart ServiceMix to apply the changes and you are ready to go!

Test the service and play around

You can test the configuration with SoapUI (example project included in sources) or make use of the client.html (thanks SMX guys for this!!!) included in the cxf-wsdl-first-secure project.

The simplest one is the client.html, just load it in your web browser and fire away.

Test Web Service Client

The test should run successfully and return the Person information.

Now, it’s time to start playing around with the LDAP and see dynamic authentication/authorization:

• For instance, you can change the ACL group in the endpoint LDAP to SMX-GROUP2 and see how you get a “Endpoint is not authorized for this user” exception.
• Create a more fine grained endpoint ACL that only assigns SMX-GROUP1 to:

  cn="http://servicemix.apache.org/samples/wsdl-first:PersonService:GetPerson"

• Change the password used for authentication of testuser in the SOAP request to get an LDAP login error.

Conclusions

Dynamic authentication/authorization configuration in an ESB with mechanisms like LDAP is mandatory in any real deployment scenario.

You have seen that adding LDAP-based authentication/authorization to SMX is not difficult but to have a robust model some implementation work is required. The implementation shown in this post can be taken as a good starting point but there are areas of improvement both in terms of performance (e.g. cache, etc…) and in the authorization model itself (e.g. precedence of ACLs, etc…).

Also, the SMX authentication/authorization mechanisms described are for version 3.x. They should be applicable also for SMX 4 with its new architecture based on OSGi, as the same type of NMR is sitting on top of the OSGi runtime.

Nearly 18 months ago, I wrote an article on this blog which showed how to develop a simple flex web application using ActiveMQ as a JMS server. It has proved a popular article so there must be a fair amount of interest in this topic.

With all the hype surrounding HTML5 at the moment, I figured I’d try to update the application for the HTML5 world, at the same time seeing how easy it would be to replace Adobe Flex. I’m sure you’re all familiar with the fact that Steve Jobs won’t let Flash (and therefore Flex) onto either the iPhone or iPad, so it’s worth investigating if the alternatives (in this case HTML5 websocket) are really up to scratch.

The example shown in the article uses HTML 5 websocket, Apache ActiveMQ, the Stomp messaging protocol and (indirectly) Jetty 7. It currently only runs in Google Chrome 5 since this is the only browser with support for websocket available as of May 2010.

What is Websocket?

In short, it’s a way of opening up a firewall-friendly bi-directional communications channel with a server. The channel stays open for a long period of time, allowing the server and client web browser to interchange messages without having to do polling which consumes bandwidth and a lot of server resources.

This article describes websocket in much more depth.

Websocket forms part of the (still unapproved) HTML5 specification which all browsers will eventually have to implement.

Updating the Trader app

In order to get the best out of this section, it is probably best to have read the original Flex-JMS article.

Here is the code for the updated trader app. Instead of having a server webapp component with BlazeDS acting as a message proxy and the flex clients, we have a bunch of HTML/CSS/JS files (in the /html folder) and some code to put messages in the JMS queues published by Apache ActiveMQ (in the /server folder).

Setting up Apache ActiveMQ

Apache ActiveMQ now comes with support for Websocket (implemented behind the scenes with Jetty 7). We will use ActiveMQ both as our messaging server and web server in this example, though that is not perhaps the best production configuration.

This article explains how to configure ActiveMQ and Websocket. I will repeat the key instructions here for the sake of simplicity:

1. Download the latest snapshot of Apache ActiveMQ 5.4 from here and unzip it somewhere on your filesystem that we will call $ACTIVEMQ_HOME
2. Edit $ACTIVEMQ_HOME/conf/activemq.xml and change the transportConnectors section so that it looks like the example below:

           <transportConnectors>
               <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
               <transportConnector name="websocket" uri="ws://0.0.0.0:61614"/>
               <transportConnector name="stomp" uri="stomp://0.0.0.0:61612?transport.closeAsync=false"/> 
               <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613?transport.closeAsync=false"/> 
           </transportConnectors>

3. Start ActiveMQ by running bin/activemq from the $ACTIVEMQ_HOME directory. Go to http://localhost:8161/admin/ and log in with the username/password admin/admin to check that everything is working ok.

Configure an install the aplication

Download the trader app code and copy the /html folder to $ACTIVEMQ_HOME/webapps/demo, renaming it to /trader-app-websocket (i.e. the full path should be $ACTIVEMQ_HOME/webapps/demo/trader-app-websocket).

Edit stock-quote.js and note the following section:

    var url = 'ws://localhost:61614/stomp';
    var login = 'guest';
    var passcode = 'guest';
    destination = '/topic/stockQuoteTopic';

The url attribute referes to the URL of the ActiveMQ server. Due to a bug in Chrome running on Ubuntu 9.10, I had to put the IP address of my machine here but if you’re running on another linux flavour, OSX or windows, I would imagine that leaving it as localhost should be OK. The username and password are guest/guest which is standard for ActiveMQ.

Download the latest version of Google Chrome (I have 5.0.375.55 which was released a few days ago) and open the URL http://localhost:8161/demo/trader-app-websocket/. You should see a UI that is similar to the Flex app developed in the original article:

Open up a terminal window and go to the location to which you extracted the /server part of the code download. Run the following (assumes Maven installed);

mvn clean compile exec:java -Dexec.mainClass="com.theserverlabs.flex.trader.JSONFeed"

You should see a whole bunch of stock quote information like that shown below scrolling in the terminal:

{"symbol": "XOM",
"name": "Exxon Mobile Corp",
"low": 61.317410451219146,
"high": 61.56,
"open": 61.56,
"last": 61.317410451219146,
"change": 61.317410451219146}

This Java program is the same as that used in the original post. It generates random stock price information for a variety of stocks and publishes it in the stockQuote topic in ActiveMQ. In this case, it generates JMS text messages which contain data formatted in the JSON format.

If you go back to the Chrome browser window, you should see the stock quotes update. If they don’t update, click on refresh:

This is pretty much exactly the same as how the original Flex application worked. The UI colours etc. are slightly different and I’ve not implemented the functionality for subscribing/unsubscribing from a stock price - but that was just on time grounds, not because it is difficult.

How does it work?

When the browser opens the page, it executes the following code in stock-quote.js, which subscribes to the stock quote service:

$(document).ready(function(){
 
    var client, destination;
 
    ... 
 
    var url = 'ws://localhost:61614/stomp';
    var login = 'guest';
    var passcode = 'guest';
    destination = '/topic/stockQuoteTopic';
 
    client = Stomp.client(url);
    client.connect(login, passcode, onconnect);
 
});

Here we use the library provided by Jeff Mesnil which enables us to access ActiveMQ using the Stomp protocol instead of JMS. We use it here because it is simple and cross-platform. There is no way of directly subscribing from JavaScript to a JMS server via JMS because there is no client available.

In the same block of code, you can see the code that we execute when we receive a message:

 
    var onconnect = function(frame) {
 
      client.subscribe(destination, function(message) {
        var quote = JSON.parse(message.body);
        $('.' + quote.symbol).replaceWith("<tr class=\"" + quote.symbol + "\">" + 
            "<td>" + quote.symbol + "</td>" +  
            "<td>" + quote.open.toFixed(2) + "</td>" + 
            "<td>" + quote.last.toFixed(2) + "</td>" + 
            "<td>" + quote.change.toFixed(2) + "</td>" + 
            "<td>" + quote.high.toFixed(2) + "</td>" + 
            "<td>" + quote.low.toFixed(2) + "</td>" +  
            "</tr>");
      });
    };

When we receive a message, we parse it using the JSON parser that is part of the JQuery library and then we find the HTML element with class attribute equal to the symbol (e.g. if the symbol is IBM, we look for the HTML element with class=”IBM”) and replace it’s contents with the HTML table row code generated in the method. Simple really.

The rest of the code is just HTML and CSS and is not really that interesting for this article.

Conclusions

It is pretty easy to develop an application that uses Websocket - you only have to look at how little real code there is in this example. I’d say it is as easy as developing the original Flex app so from a development point of view, there’s little to chose between these technologies.

Unfortunately the only browser that currently supports Websocket is Google Chrome (and the implementation is a bit buggy). Other browsers (especially Firefox and Safari) should have this functionality soon though. One of the arguments used in support of Flash/Flex has always been the large installed base. Given that the iPhone and iPad are not part of this installed base, it is questionable as to whether this can still be used as a justification for using Flash/Flex. Sure there aren’t that many browsers that support Websocket but in 6 months time they probably all will and you won’t need any proprietary plugin to access the apps build using them. I’d definitely recommend that people developing internal corporate apps who can force their end users to use a browser with Websocket support take a look at this technology. People developing publically accessible webapps are probably going to have to wait till it is more widely implemented in browsers and provide graceful fallback in the case in which it isn’t.

I should point out that I have an iPhone and use Ubuntu 9.10 64-bit at work and I hate not being able to see content on my iPhone because it has been implemented in Flash and my entire firefox browser often crashes completely due to the 64-bit linux Flash plugin.

I’ve gone from being a fan of Flash/Flex to not being so sure about it. It’s hard to escape the feeling that HTML5 will offer much of the functionality currently offered by Flash/Flex in a short period of time. I think the future of these technologies is going to depend on Adobe innovating and offering stuff that is not possible in HTML5, otherwise there is little reason to use it.

When we are diagnosing problems in a Java (EE or otherwise) application, is often a good idea to check how garbage collection is performing. One of the basic and most unobtrusive actions is to enable garbage collection logging.

As you may know, if we add the following arguments to the java start command…

-Xloggc:<file_name> –XX:+PrintGCDetails -XX:+PrintGCDateStamps

… the JVM will start writing garbage collection messages to the file we set with the parameter -Xlogcc. The messages should be something like:


2010-04-22T18:12:27.796+0200: 22.317: [GC 59030K->52906K(97244K), 0.0019061 secs]
2010-04-22T18:12:27.828+0200: 22.348: [GC 59114K->52749K(97244K), 0.0021595 secs]
2010-04-22T18:12:27.859+0200: 22.380: [GC 58957K->53335K(97244K), 0.0022615 secs]
2010-04-22T18:12:27.890+0200: 22.409: [GC 59543K->53385K(97244K), 0.0024157 secs]


The bold part is simply the date and time when reported garbage collection event starts.

Unfortunately -XX:+PrintGCDateStamps is available only for Java 6 Update 4 and later JVMs. So, if we are unlucky and our application is running on older JVMs we are forced to use…

-Xloggc:<file> –XX:+PrintGCDetails

… and the messages will be like:


22.317: [GC 59030K->52906K(97244K), 0.0019061 secs]
22.348: [GC 59114K->52749K(97244K), 0.0021595 secs]
22.380: [GC 58957K->53335K(97244K), 0.0022615 secs]
22.409: [GC 59543K->53385K(97244K), 0.0024157 secs]


Now, the bold numbers (also present in previous format) are the seconds elapsed from JVM start time.

Mmm… way harder to correlate GC events with information from other log files in this case :/

Wouldn’t it be easier to process the gc log file and calculate date and time from seconds elapsed? It seems so, but seconds elapsed from… when? Or, putting it in other words, where do we extract the JVM startup date and time from?

In order to be as unobtrusive as possible, we should try to calculate the start date and time from the same GC log file. That brings us to the file attributes. We have different options:

We discard access time (for obvious reasons) and change time and creation time as they are not available in both platforms, so we are left with modification time, which represents the time when the file was last modified.

In Windows, modification time is maintained when the file is copied elsewhere, but in Unix we should use the -p flag to preserve timestamp attributes if we want to copy the GC log file prior to our processing.

The last modification time of the GC log file should match the last timestamp recorded for a GC event in the log file. Well… for the purists, it should match exactly the last elapsed time plus the execution time (both in bold) as each log line is written piece by piece as it executes.


22.409: [GC 59543K->53385K(97244K), 0.0024157 secs]


In our approach, we discard the execution time as we don’t need accurate precision to have a rough idea of what time each garbage collection event occurred. Nevertheless, keep in mind that GC execution time could sometimes be as long as several seconds in large heaps.

When we experienced this situation in a client recently, we needed to quickly develop a simple and portable script, so we used Python for the task. You already knew we don’t do just Java, didn’t you?

#!/usr/bin/env python
 
import sys, os, datetime
 
# true if string is a positive float
def validSeconds(str_sec):
    try:
        return 0 < float(str_sec)
    except ValueError:
        return False
 
# show usage                
if len(sys.argv) < 2:
    print "Usage: %s <gc.log>" % (sys.argv[0])
    sys.exit(1)
 
file_str = sys.argv[1]
lastmod_date = datetime.datetime.fromtimestamp(os.path.getmtime(file_str))
 
file = open(file_str, 'r')
lines = file.readlines()
file.close()
 
# get last elapsed time
for line in reversed(lines):
    parts = line.split(':')
    if validSeconds(parts[0]):
        break
 
# calculate start time
start_date = lastmod_date - datetime.timedelta(seconds=float(parts[0]))
 
# print file prepending human readable time where appropiate  
for line in lines:
    parts = line.split(':')
    if not validSeconds(parts[0]):
        print line.rstrip()
        continue
    line_date = start_date + datetime.timedelta(seconds=float(parts[0]))
    print "%s: %s" % (line_date.isoformat(), line.rstrip())

The script output can be redirected to another file, where we’ll have

2010-04-22T18:12:27.796375: 22.317: [GC 59030K->52906K(97244K), 0.0019061 secs]
2010-04-22T18:12:27.828375: 22.348: [GC 59114K->52749K(97244K), 0.0021595 secs]
2010-04-22T18:12:27.859375: 22.380: [GC 58957K->53335K(97244K), 0.0022615 secs]
2010-04-22T18:12:27.890375: 22.409: [GC 59543K->53385K(97244K), 0.0024157 secs]


You may note the date format is not 100% the same as the one with -XX:+PrintGCDateStamps argument, but it should be enough to get an idea of when each GC event happened (Timezone management in Python is way out of the scope of this blog entry).

This has been my first blog entry for The Server Labs and I hope some of you find it useful. Of course, all comments, suggestions and feedback are very welcome.

This week The Server Labs launched a second blog - BizBlog - which, as my colleague Greg says, aims to “address IT Architecture and Cloud topics taking a less technical but more business relevant angle” than that found in this more technical blog.

I encourage any regular readers of this blog to take a look at Greg’s initial post and the posts that he will be publishing in the next few weeks for a slightly different take on IT.

It seems that every day projects have more and more dependencies on libraries (internal or external) and, of course, many of these depend on other libraries, resulting in a large dependency tree for any given project. How do you know if any of those libraries contain some code which is licensed in a way that is incompatible with your company’s policies e.g. no GPL?

BT (the former British Telecom) apparently didn’t and ended up having to publish all the code used in one of the routers it distributes due to a GPL violation.

To give you an idea of the scale of this problem, doing a quick search of my local Maven repository reveals that it has 1760 JAR files in it. Admittedly not all of these belong to one single project but maybe they are spread out over 20 different projects. It is pretty infeasible to try to manage such a task manually.

Tools like Maven are a great help for managing dependency trees in your project but doesn’t help much with checking the licenses that each dependency uses. The pom.xml file permits the use of a <license> element but it is optional, many libraries either don’t use Maven or don’t specify the license and you have to check compliance manually in any case.

This is where IPR monitoring tools come in. Such tools allow the definition of licensing policies at an organizational level and provide mechanisms to monitor compliance with these policies in software projects, raising alerts on detected violations.

We recently had to take a look at such tools for one of our clients. After studying the market, we discovered that are currently no open-source solutions covering this problem domain, but several commercial tools address the problem of continuous IPR monitoring.

For reference purposes, here is a list of the providers that we discovered:

All of these commercial products offer common features:

• Automated binary and source code analysis with multi-language support (Java, C/C++, C#,
  Visual Basic, Perl, Python, PHP). The analysis is performed against an external proprietary
  database that contains the code of most open-source products.
• Provide workflows in order to control the IPR of the software projects through the whole
  lifecycle, based on defined licensing policies.
• Approval/disapproval licensing mechanisms as well as billing of materials for
  software releases summarizing components, licenses, approval status and license/policy
  violations.
• Different levels of code fragment recognition to detect reuse of code.
• User interfaces offering policy management, reporting and dashboard features.
• Support for integration of code scan in Continuous Integration platforms via command line
  interface execution.

We think that these products are going to become increasingly important as the total number of libraries used in projects shows no sign of decreasing and there will always be a need to protect intellectual property.

Logging is an uninteresting but nevertheless fundamental part of application development. It is useful not just when you are coding an application but in diagnosing problems of any nature once an application passes into production.

Unfortunately, it is not that clear how to do logging well when developing JEE applications that are to be deployed on a Weblogic server. With this article, I hope to clear up some of the confusion or, at the very least, explain a method that works for me!

The article is based on the use of Apache log4j since that is the most commonly used logging library. The article explains:

• how to configure log4j properly in a JEE app to be run on Weblogic
• how to dynamically configure the log level using JMX and Weblogic Scripting Tool

The article assumes some familiarity with Apache Maven, Eclipse and Weblogic.

The code is available here, although you may just prefer to download the EAR file which saves you having to compile any code with Maven. The code.zip file contains the code for the JEE application, the code for the JMX library developed by The Server Labs and used in this article and the scripts for WLST.

Configuring log4j in Weblogic

Weblogic 10 comes with a modified version of log4j by default so you can use log4j in your application without having to deploy the log4j library in your application. Unfortunately, my experience (in WLS 10.0) is that if you do this, you will not be able to configure log4j for your application because the classloader in which log4j is loaded is one that is separate to that of your application. Any log4j.properties that you include will be ignored.

Instead, what you have to do is the following:

1) Include the log4j JAR in your application. If your app is an EAR (like mine), then the log4j JAR should go in the .ear file (in the root directory or in /APP-INF/lib). In my project, I have used a Maven multi-project setup with 3 projects - 1 for the EAR, 1 for the EJBs and 1 webapp. In this case, I have to add the log4j JAR to the EAR project’s pom.xml:

	<dependencies>
		<dependency>
			<groupId>log4j</groupId>
			<artifactId>log4j</artifactId>
			<version>1.2.12</version>
		</dependency>
	</dependencies>

2) Tell Weblogic to use your version of the log4j library and not it’s own. To do this, you specify the following in the weblogic-application.xml file which is in the EAR (/META-INF/weblogic-application.xml):

	<wls:prefer-application-packages>
		<wls:package-name>org.apache.log4j.*</wls:package-name>
	</wls:prefer-application-packages>

This basically says that when the app asks for a class in the package org.apache.log4j or it’s sub-packages, that it should look for it on the app’s classloader and not in the Weblogic server classloader.

3) Configure log4j using log4j.properties

Log4j requires that you configure it in order to receive log messages. I include the following contents in the /APP-INF/classes/log4j.properties file which is in the EAR file:

log4j.rootLogger=INFO, stdout, file

log4j.debug=true

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %-5p %c %x - %m%n

log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.File=/bea/logs/aplicaciones/blog-logging/logging_${weblogic.Name}.log
log4j.appender.file.Append = false
log4j.appender.file.layout = org.apache.log4j.PatternLayout
log4j.appender.file.MaxFileSize = 20000Kb
log4j.appender.file.MaxBackupIndex = 5
log4j.appender.file.layout.ConversionPattern=[%d] %-5p %c %x - %m%n

log4j.category.com.theserverlabs.blog.wls=DEBUG

This configures log4j to send output both to the console in which Weblogic was started and to a file (/bea/logs/aplicaciones/blog-logging/logging_${weblogic.Name}.log). You will probably want to change the file path to something that works on your machine.

Testing that log4j is working OK

In order to test these changes, download the EAR file and deploy it to a Weblogic server. I tested on Weblogic 10.3 but it should work on any Weblogic 10.x server or later I believe.

Go to the URL http://localhost:7001/logging/test and you should see a very basic HTML page that tells you to check the logs:

Take a look at the console in which weblogic is running (or the Server.out file if you have the output redirected) and you should see output like that shown below:

[2010-04-21 15:29:47,888] WARN  com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - warn
[2010-04-21 15:29:47,888] INFO  com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - info
[2010-04-21 15:29:47,888] DEBUG com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - debug
[2010-04-21 15:29:47,889] ERROR com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - error
[2010-04-21 15:29:47,889] FATAL com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - fatal
[2010-04-21 15:29:48,056] WARN  com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - warn
[2010-04-21 15:29:48,056] INFO  com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - info
[2010-04-21 15:29:48,056] DEBUG com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - debug
[2010-04-21 15:29:48,056] ERROR com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - error
[2010-04-21 15:29:48,056] FATAL com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - fatal

If you examine the application, you can see that accessing the URL causes a Servlet to get executed. The servlet writes it’s messages to the log (those recorded with the class “com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet”) and invokes an EJB3 session bean which also writes messages to the log (those tagged with “com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService”).

This shows that this log solution works equally for web as for EJB and that there is no interference with the weblogic logging system.

Making it dynamic

This is a great solution when you’re still developing the app because you can change the debug level just by modifying the log4j.properties file and redeploying the app. However, this is not something you can feasibly do in a production environment - not at the companies I’ve worked at anyway - due to the interruption in service this causes to the end users.

Why would you want to change the log level? Well, if you have to diagnose a problem in the application, it often makes sense to increase the log level of either the whole application or a subset of it such that you receive more information which helps you to diagnose the problem. Once you have diagnosed the problem, you return the logging to it’s original level so as not to affect the performance of the application unnecesarily.

What we really want is to be able to change the log level dynamically (i.e. without redeploying the application), ideally using administrator tools (since that is what the people who manage production systems tend to use) - not developer tools.

First I’ll show you how to change the log level and then I’ll explain how it works.

Using WLST and JMX to change the log level

If you have deployed the example application, go to your domain home in a terminal window and execute the setDomainEnv script. The idea is to update your shell with the domain environment variables such as JAVA_HOME, CLASSPATH etc. On my linux box, I execute the following:

$ cd $DOMAIN_HOME/bin
$ source setDomainEnv.sh

To check it has worked, run “echo $CLASSPATH” on unix or “echo %CLASSPATH%” on windows and you should see a long classpath.

Once you have set up the environment, execute the changeAppLogLevel.py script that is included in the /scripts directory of the code ZIP:

java -classpath $CLASSPATH  $JAVA_OPTIONS  weblogic.WLST changeAppLogLevel.py

You should see the following output:

$ java -classpath $CLASSPATH  $JAVA_OPTIONS  weblogic.WLST changeAppLogLevel.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

==>Insufficient arguments
==>Syntax: changeAppLogLevel.py [APP NAME] [LOGGER] [LEVEL]
==> where: APP NAME=context root of the app e.g. xxx for /xxx
==>        LOGGER=log4j logger name
==>        LEVEL=TRACE/DEBUG/WARN/INFO/ERROR/FATAL
==>   e.g: changeAppLogLevel.py xxx com.theserverlabs.blog.logging DEBUG

As the instructions indicate, you have to call the script with the name of the app (which is the context root of the application web), the Log4j logger that you wish to modify (normally a Java package name) and the level that you want to set (TRACE/DEBUG/WARN/INFO/ERROR/FATAL).

In the example below, we set the log level to INFO for the package com.theserverlabs.blog.wls in the application “logging”, which is what the example application is called. The output is the following:

java -classpath $CLASSPATH  $JAVA_OPTIONS  weblogic.WLST changeAppLogLevel.py logging com.theserverlabs.blog.wls ERROR 

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'log4j'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Location changed to custom tree. This is a writable tree with No root.
For more help, use help(custom)

changing log level to ERROR for app logging on server t3://localhost:7001
  ... done. New log levels are:
{com.theserverlabs.blog.wls=ERROR}

Exiting WebLogic Scripting Tool.

As you can see from the line “New log levels are:{com.theserverlabs.blog.wls=ERROR}”, the log level was changed from the default in the log4j.properties file (DEBUG) to ERROR.

If you go to the logging test page again (http://localhost:7001/logging/test), you should see that there are no longer entries for the DEBUG, INFOR or WARN levels:

[2010-04-21 17:47:09,728] ERROR com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - error
[2010-04-21 17:47:09,728] FATAL com.theserverlabs.blog.wls.logging.view.servlet.LoggingTestServlet  - fatal
[2010-04-21 17:47:09,728] ERROR com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - error
[2010-04-21 17:47:09,728] FATAL com.theserverlabs.blog.wls.logging.services.ejb.SessionLoggingTestService  - fatal

So, we can now dynamically change the log level of a logger in our application using an administrator tool (WLST). Pretty cool, huh?

Here I have specified a single logger “com.theserverlabs.blog.wls” which is a high-level logger in my application. You can only pass 1 logger as arguments to the script but there is nothing to stop you from running the same script different times with different loggers e.g. once to set com.theserverlabs.blog.wls.logging.services.ejb to DEBUG to get detailed logging of EJB session beans and another time to set org.hibernate to INFO to get more info on what hibernate is doing.

How do I get this to work in my application?

Simple. Include our JMX-logging JAR in your WEB application and add the following lines to your web.xml file:

	<listener>
		<listener-class>com.theserverlabs.jmx.web.JmxServletContextListener</listener-class>
	</listener>

At startup time, a JMX Mbean will be registered with the Weblogic JMX subsystem. The Mbean will be registered under a name which includes your application name, meaning there will be one MBean per application deployed on the server.

Once deployed, you can use the WLST script discussed in the previous section to interact with the MBean for your application and thereby change the log level of your application.

OK, how does all this really work?

If you look at the source code for the jmx-logging JAR (included in the source code bundle, you will find a class called com.theserverlabs.jmx.ApplicationLogManager. This class implements the interface ApplicationLogManagerMBean, whose methods are JMX MBean-compatible. It provides methods that allow a caller to set the log level for a given log4j logger:

	public void setLogLevel(String logger, String level) {
		Logger l = Logger.getLogger(logger);
		l.setLevel(Level.toLevel(level));
	}

and to interrogate log4j for the currently-configured loggers and their levels, returning a map of Strings:

	@SuppressWarnings("unchecked")
	public Map<String, String> getLogLevels() {
		HashMap<String, String> result = new HashMap<String, String>();
		Enumeration<Logger> e = Logger.getRootLogger().getLoggerRepository().getCurrentLoggers();
		while (e.hasMoreElements()) {
			Logger l = e.nextElement();
			if (l.getLevel() != null) {
				result.put(l.getName(), l.getLevel().toString());
			}
		}
		return result;
	}

This MBean can be registered with any JMX system, allowing it’s methods to be invoked via JMX calls. In this case, we really want to register it with the Weblogic JMX system when the application starts up. To accomplish this, we use the class com.theserverlabs.jmx.web.JmxServletContextListener, which is registered in the web.xml of the web application:

	<listener>
		<listener-class>com.theserverlabs.jmx.web.JmxServletContextListener</listener-class>
	</listener>

A modified version of the source code for JmxServletContextListener is shown below which error-handling and logging removed for clarity. It is pretty clear that when Weblogic starts our application and calls the contextInitialized() method, we register a new instance of ApplicationLogManager with the object name “com.theserverlabs.jmx:type=ApplicationLogManager,name=[NAME OF APPLICATION]LogManager”. This name is generated in the getObjectName() method. When Weblogic stops the web application, it calls the contextDestroyed() method which un-registers the instance of ApplicationLogManager from the JMX MBean server.

public class JmxServletContextListener implements ServletContextListener {
 
	private InitialContext ctx;
	private MBeanServer server;
 
	{
	    ctx = new InitialContext();
	    server = MBeanServer.class.cast(ctx.lookup("java:comp/env/jmx/runtime"));
	}
 
	public void contextInitialized(ServletContextEvent sce) {
		server.registerMBean(new ApplicationLogManager(), getObjectName(sce));
	}
 
	public void contextDestroyed(ServletContextEvent sce) {
		if (server.isRegistered(getObjectName(sce))) {
		    server.unregisterMBean(getObjectName(sce));
		} 
	}
 
	private ObjectName getObjectName(ServletContextEvent sce) throws MalformedObjectNameException, NullPointerException { 
		String appName = sce.getServletContext().getContextPath().substring(1);
		return new ObjectName("com.theserverlabs.jmx:type=ApplicationLogManager,name=" + appName + "LogManager");
	}
 
}

The WLST script

That’s pretty much all there is to it in terms of application code. The only thing left to discuss is the Weblogic Scripting Tool (WLST) script that modifies the log level. Below is the source code (it is Python code):

def changeServerAppLogLevel(app, logger, level, serverUrl):
 
    ## connect to the server
    connect("weblogic", "weblogic", url=serverUrl)
 
    ## go to the custom MBean tree 
    custom()
 
    ## go to the place where our app log level mbeans are stored. 
    cd('com.theserverlabs.jmx/com.theserverlabs.jmx:type=ApplicationLogManager,name=' + app + 'LogManager')
 
    ## horrible code necessary to invoke a Java method that changes the log level
    args = jarray.array([java.lang.String(logger), java.lang.String(level)],java.lang.Object)
    sig = jarray.array(['java.lang.String', 'java.lang.String'],java.lang.String)
    print 'changing log level to ' + level + ' for app ' + app + ' on server ' + serverUrl
    invoke('setLogLevel',args,sig)
 
    print '  ... done. New log levels are:'
    print(get('LogLevels'))
 
# "main method" of the python script
argslength = len(sys.argv)
 
if argslength < 3 :
 
    print '==>Insufficient arguments'
    print '==>Syntax: changeAppLogLevel.py [APP NAME] [LOGGER] [LEVEL]'
    print '==> where: APP NAME=context root of the app e.g. xxx for /xxx'
    print '==>        LOGGER=log4j logger name'
    print '==>        LEVEL=TRACE/DEBUG/WARN/INFO/ERROR/FATAL'
    print '==>   e.g: changeAppLogLevel.py xxx com.theserverlabs.blog.logging DEBUG'
    exit()
 
else:
 
    app = sys.argv[1]
    logger = sys.argv[2]
    level = sys.argv[3]
 
    # change the log level for the server. if there was more than one server
    # in the cluster, you could add multiple calls here, each with a different 
    # server URL
    changeServerAppLogLevel(app, logger, level, 't3://localhost:7001')
 
    exit()

Hopefully the majority of this code will be clear, even if you don’t understand python. The method/function changeServerAppLogLevel() declared at the start modifies the log level for a defined logger in an application running on a specific Weblogic server. To do this it:

• Connects to the Weblogic server at the specified URL (note that this example uses the hardcoded username and password weblogic/weblogic
• Moves to the custom JMX MBean tree which is where Weblogic puts all non-Weblogic MBeans
• Moves to the place where our MBean is stored in the tree
• Executes the setLogLevel() method of the MBean with the required arguments to change the log level. This translates to an invocation of the ApplicationLogManager.setLogLevel() method.
• Lists the new log levels

In the “main” method of the script (i.e. everything that is not in a method/function), there is some basic code to make sure the script is properly invoked and, if it is, a call to changeServerAppLogLevel() with the supplied arguments and the hard-coded URL of ‘t3://localhost:7001′. Obviously you should modify this value to reflect your Weblogic environment.

One interesting aspect of using a script to do this is that you can include a call to changeServerAppLogLevel() for each server in your cluster, if you have a multi-node setup. MBeans are published at server level so there is no concept of publishing a cluster-wide MBean (so far as I know anyway).

An interesting extension of this script would be to interrogate a Weblogic Admin server to find all the nodes in the cluster that were up and then execute the code to change the log level against each available node.

Conclusion

Hopefully this article has cleared up any confusion as to how to perform logging with log4j in a JEE app deployed on Weblogic. I think that the ability to modify dynamically the log level for a given log4j logger without having to redeploy the application is extremely useful and hopefully will be of use to people looking to do a similar thing either with Weblogic or any other JEE server that supports JMX.

Come and see us at CeBIT 2010 where we will have a booth on Amazon’s Stand in the Main Exhibition Hall: Hall 2, Stand B26. We will be there on March the 4th and March the 5th.

We will be happy to chat to you about our added value in bringing IT Architecture solutions to the Cloud.

We will also be presenting the work on HPC we have been doing for the European Space Agency, at 13:30 on March the 4th, and at 11:00 on March the 5th, both in the Theatre at the back of the Amazon stand.


Complex low-cost HPC Data Processing in the Cloud

With the maturing of cloud computing, it is now feasible to run even the most complex HPC applications in the cloud. Data storage and high performance computing resources - fundamental for these applications – can be outsourced thus leveraging scalability, flexibility and high availability at a fraction of the cost of traditional in-house data processing. This presentation evaluates Amazon’s EC2/S3 suitability for such a scenario, by running a distributed astrometric process The Server Labs developed for the European Space Agency’s Gaia mission in Amazon EC2.


I’ve recently been investigating the use of the JBI4Corba JBI Binding Component with the Apache ServiceMix Enterprise Service Bus. JBI4Corba permits a JBI compliant ESB to call corba code running outside of the bus (the consumer scenario) and to external code to call endpoints published on the bus via corba (the provider scenario).

Why is this useful? Well, there is quite a lot of corba code out there but new applications are not generally written to use corba technologies. JBI4Corba permits you to make services offered by legacy corba code available to new applications and enables existing corba code to call services published on the ESB. This means you don’t have to rewrite code that uses corba in order to integrate it into an SOA architecture.

This post will show you how to get started with JBI4Corba and Apache ServiceMix, using an example taken from the JBI4Corba integration tests. The idea is to show from “first principles” how to arrive at a simple JBI4Corba solution, which is something that I have failed to find myself on the web.

In this example, we have a simple ‘legacy’ corba interface which just has one method that echos an input parameter String back to the caller. This interface is implemented in a Java corba Servant. We wish to make this corba interface available as a web service and call it using a standard HTTP client - which knows nothing about corba. The diagram below (taken from the JBI4Corba website) indicates how the various components will interact in this example:

To get started, download this code from here and unzip it to a folder on your local machine. This example requires you to have Maven and Apache Servicemix installed on your machine. I have versions 2.09 and 3.3.1 of these two products, respectively.

The Corba Code

Once you have downloaded the code, run

mvn install

in the root directory in which you unzipped the code. The provider-simple-servant project contains the existing Corba code. In src/main/idl, you will find the Corba IDL file which expresses the Corba interfaces. In this case, it is a very simple interface:

module it{
	module imolinfo{
		module jbi4corba{
			module test{
				module testprovidersimple{
						interface Echo {
							string echo(in string msg);
						};
				};
			};
		};
	};
};

In this project, we use the Maven idlj plugin to generate the Java classes from this Corba IDL file. Below is the pom.xml configuration:

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>idlj-maven-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>generate</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <debug>true</debug>
          <sources>
            <source>
              <includes>
                <include>Echo.idl</include>
              </includes>
			  <emitStubs>true</emitStubs>
              <emitSkeletons>true</emitSkeletons>
              <compatible>false</compatible>
            </source>
		</sources>
	    </configuration>        
      </plugin>

When you run the maven generate-sources goal or a later goal (such as compile or install), idlj generates the Corba Java classes and puts them in the /target/generated-sources/idl directory, which is on the classpath.

The /src/main/java/it/imolinfo/jbi4corba/test/servant/testprovidersimple/EchoSimpleImpl.java class is a Corba Servant that provides the behaviour for the Echo Corba interface. To run it, we must first be running the Sun Corba ORB daemon, which depends on your OS. Unix:

  orbd -ORBInitialPort 1050 -ORBInitialHost localhost&

Windows:

  start orbd -ORBInitialPort 1050 -ORBInitialHost localhost

Once this is running ok, we run the Echo server by executing the following Maven command in the /provider-simple-servant directory:

mvn exec:java -Dexec.mainClass="it.imolinfo.jbi4corba.test.servant.testprovidersimple.EchoSimpleImpl" -Dexec.args="sunorb.properties"

In another command window, execute the following in the same /provider-simple-servant directory to run the EchoClient that tests that the server is working ok:

mvn exec:java -Dexec.mainClass="it.imolinfo.jbi4corba.test.servant.testprovidersimple.EchoClient" -Dexec.args="-ORBInitialPort 1050 -ORBInitialHost localhost"

When you run the client, you should see output like that below:

.....
[INFO] [exec:java]
Obtained a handle on server object: IOR:000000000000003b49444c3a69742f696d6f6c696e666f2f6a626934636f7262612f746573742f7465737470726f766964657273696d706c652f4563686f3a312e300000000000010000000000000082000102000000000a3132372e302e312e3100881a00000031afabcb0000000020cd9636e200000001000000000000000100000008526f6f74504f410000000008000000010000000014000000000000020000000100000020000000000001000100000002050100010001002000010109000000010001010000000026000000020002
  result: test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
....

If not, your Echo server is not correctly working and you need to fix this before continuing with the rest of the post.

IMPORTANT: Leave the Echo server running - we will need it later on in the post.

Introducing JBI4Corba

Now that we know that the Corba code is working ok, we can create a JBI service unit that uses JBI4Corba to publish this Corba application on the ServiceMix ESB. The configuration to do this is in the provider-simple-jbi4corba-provider project. Unfortunately, there do not seem to be any Maven archetypes for JBI4Corba projects so we won’t look at how to create this project completely from scratch, but we will see the important steps.

You must generate a WSDL that describes your Corba interface so that it can be used with JBI4Corba. This can be done with the IDL to WSDL tool. Download this tool from here, and modify the shell/batch script that is in the /bin directory (if necessary) to point to your JDK installation. Once you have it installed, run the following command in the /provider-simple-jbi4corba-provider/src/main/resources directory, substituting [path-to] for the path where you installed IDL2WSDLTool:

 /[path-to]/bin/IDL2WSDLTool Echo.idl jbi4corba.properties

This generates the Echo.wsdl file which describes the Corba IDL in a WSDL format that JBI4Corba understands. The generation of this file is a bit of a pain in the neck to be honest. You have to generate a properties file which describes the IDL file, such as provider-simple-jbi4corba-provider/src/main/resources/jbi4corba.properties:

InterfaceCount=1
Interface0.FileName=Echo.wsdl
Interface0.IDLInterfaceName=Echo
Interface0.Address=Echo
Interface0.LocalizationType=NameService
Interface0.ORBPropertiesCount=3
Interface0.ORBPropertyName0=org.omg.CORBA.ORBInitialPort
Interface0.ORBPropertyValue0=1050
Interface0.ORBPropertyName1=org.omg.CORBA.ORBClass
Interface0.ORBPropertyValue1=com.sun.corba.ee.impl.orb.ORBImpl
Interface0.ORBPropertyName2=org.omg.CORBA.ORBInitialHost
Interface0.ORBPropertyValue2=localhost

This file states that there is 1 interface in the IDL file, that it is called “Echo” and that it should be published at address “Echo” in the NameService. The WSDL file generated is Echo.wsdl.

Note that the jbi-services.xml file in /provider-simple-jbi4corba-provider/src/main/resources makes reference to the target namespace (targetNamespace=”http://it.imolinfo.jbi4corba.test.testprovidersimple.Echo”) declared at the top of the Echo.wsdl file, as well as other attrributes contained within that file.

  <services binding-component="true" xmlns="http://java.sun.com/xml/ns/jbi" 
  xmlns:ns0="http://it.imolinfo.jbi4corba.test.testprovidersimple.Echo">
 
    <provides interface-name="ns0:Echo" 
              service-name="ns0:Echo" 
              endpoint-name="EchoCorbaPort"/>
  </services>

Once you have generated the WSDL file, run

mvn install

in the root of the provider-simple-jbi4corba-provider project to install the final version.

Accessing the Corba code via HTTP Web services

Now we will create a JBI Service Unit that publishes the Endpoint created in the JBI4Corba Service Unit via HTTP web services. To do this, we will use the Apache Servicemix maven archetypes which make it much easier to create JBI service unit projects since they auto-generate the relevant structure and file and create the JBI XML descriptor automatically at runtime. In the directory to which you downloaded the source code, run the following:

mvn archetype:create \
    -DarchetypeGroupId=org.apache.servicemix.tooling \
    -DarchetypeArtifactId=servicemix-http-consumer-service-unit \
    -DgroupId=it.imolinfo.jbi4corba.test-provider-simple \
    -DartifactId=provider-simple-http-consumer \
    -DremoteRepositories=http://people.apache.org/repo/m2-incubating-repository

You should now have a new project called “provider-simple-http-consumer”. Edit the provider-simple-http-consumer/src/main/resources/xbean.xml file and replace the contents with the following code:

<?xml version="1.0"?>
<beans
  xmlns:http="http://servicemix.apache.org/http/1.0"
  xmlns:jbi4corba-test="http://it.imolinfo.jbi4corba.test.testprovidersimple.Echo">      
  <http:endpoint
    service="jbi4corba-test:Echo"
    endpoint="EchoCorbaPort"
    interfaceName="jbi4corba-test:Echo"
    role="consumer"
    locationURI="http://localhost:8192/Service/test-provider-simple/"
    defaultMep="http://www.w3.org/2004/08/wsdl/in-out"
    soap="true" />
</beans>

This publishes a new HTTP web service at http://localhost:8192/Service/test-provider-simple/ which calls out to the jbi4corba-test:Echo Endpoint established by the provider-simple-jbi4corba-provider JBI service unit.

In the provider-simple-http-consumer/src/main/resources directory, create a new file called EchoSimple.wsdl and paste in the following content:

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions name="EchoSimple" targetNamespace="urn:jbi4corba/test-provider-simple" xmlns:tns="http://it.imolinfo.jbi4corba.test.testprovidersimple.Echo" xmlns:imolacorba="uri://schemas.imola.it/jbi/wsdl-extensions/corba/" xmlns:plnk="http://docs.oasis-open.org/wsbpel/2.0/plnktype" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
  <wsdl:types>
    <xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://it.imolinfo.jbi4corba.test.testprovidersimple.Echo" xmlns="http://it.imolinfo.jbi4corba.test.testprovidersimple.Echo" xmlns:xs="http://www.w3.org/2001/XMLSchema">
 
   <xs:element name="echo">
        <xs:complexType>
            <xs:sequence>
                <xs:element minOccurs="0" name="msg" type="xs:string"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:element name="echoResponse">
        <xs:complexType>
            <xs:sequence>
                <xs:element minOccurs="0" name="return" type="xs:string"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>
  </wsdl:types>
  <wsdl:message name="echo">
    <wsdl:part name="parameters" element="tns:echo">
    </wsdl:part>
  </wsdl:message>
  <wsdl:message name="echoResponse">
    <wsdl:part name="parameters" element="tns:echoResponse">
    </wsdl:part>
  </wsdl:message>
  <wsdl:portType name="EchoSimple">
    <wsdl:operation name="echo">
      <wsdl:input name="echo" message="tns:echo">
    </wsdl:input>
      <wsdl:output name="echoResponse" message="tns:echoResponse">
    </wsdl:output>
    </wsdl:operation>
  </wsdl:portType>
 <wsdl:binding name="EchoSimpleCorbaBinding" type="tns:EchoSimple">
 
    <soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"/>
    <wsdl:operation name="echo">
      <soap:operation/>
      <wsdl:input name="echo">
        <soap:body use="literal"/>
      </wsdl:input>
      <wsdl:output name="echoResponse">
        <soap:body use="literal"/>
      </wsdl:output>
    </wsdl:operation>
  </wsdl:binding>
  <wsdl:service name="EchoSimple">
    <wsdl:port name="EchoSimpleCorbaPort" binding="tns:EchoSimpleCorbaBinding">
      <soap:address location="http://localhost:8192/Service/test-provider-simple/"/>
    </wsdl:port>
  </wsdl:service>
<plnk:partnerLinkType name="EchoSimple">
<plnk:role name="EchoSimplePortTypeRole" portType="tns:EchoSimple"/>
</plnk:partnerLinkType>
</wsdl:definitions>

Run

mvn install

in the root of the project to create and install the JBI Service unit.

Creating the Service Assembly

A JBI Service assembly is a way of packaging up JBI Service units. We will create the Service Assembly maven project using another Servicemix archetype. Run the following in the directory to which you downloaded the source code:

mvn archetype:create \
    -DarchetypeGroupId=org.apache.servicemix.tooling \
    -DarchetypeArtifactId=servicemix-service-assembly \
    -DgroupId=it.imolinfo.jbi4corba.test-provider-simple \
    -DartifactId=provider-simple-service-assembly \
    -DremoteRepositories=http://people.apache.org/repo/m2-incubating-repository

This should create a new maven sub-project called provider-simple-service-assembly which just contains a pom.xml file. Edit this file and remove the Junit dependency. Add the following instead:

    <dependency>
      <groupId>${pom.groupId}</groupId>
      <artifactId>provider-simple-jbi4corba-provider</artifactId>
      <version>${pom.version}</version>
    </dependency>
    <dependency>
      <groupId>${pom.groupId}</groupId>
      <artifactId>provider-simple-http-consumer</artifactId>
      <version>${pom.version}</version>
    </dependency>

Run mvn install in the service assembly project directory and a new file should be created in the provider-simple-service-assembly/target directory called provider-simple-service-assembly-1.0-SNAPSHOT.jar. This is the file we will deploy to ServiceMix.

Deploying to ServiceMix

Download and install Apache ServiceMix 3.3.1 if you have not already done so. I will refer to the installation directory as $SERVICEMIX_HOME. Download Jbi4Corba and copy it into $SERVICEMIX_HOME/hotdeploy directory. Ensure you have a Java JDK installed. Copy the $JDK_HOME/lib/tools.jar file to $SERVICEMIX_HOME/lib/optional.

Start up ServiceMix by running the servicemix or servicemix.bat (depending on your OS) file in the $SERVICEMIX_HOME/bin folder.

Ensure that you still have the EchoServer running from earlier in this post. Now, copy the provider-simple-service-assembly/target/provider-simple-service-assembly-1.0-SNAPSHOT.jar file to the $SERVICEMIX_HOME/hotdeploy directory. You should see a whole bunch of messages on the ServiceMix console but hopefully no (serious) errors.

To test if the service assembly deployed ok, go to http://localhost:8192/Service/test-provider-simple/main.wsdl and you should see the WSDL descriptor for the HTTP web service. Save this WSDL file somewhere on your local machine.

Testing using SoapUI

SoapUI is a very effective tool for testing web services. If you have not already installed it, download and install the standard (free) edition and run it.

Select File > New soapUI project. Enter “test jbi4corba” as the project name and for the Initial WSDL/WADL field, browse for the WSDL file you saved from http://localhost:8192/Service/test-provider-simple/main.wsdl. Ensure the “create requests” option is ticked and click ok. Keep clicking OK till soapUI creates a TestSuite for you. Save the project.

Right-click on the “EchoCorbaPortBinding Test Suite” element. Choose “Launch Test Runner” and select “Launch” from the dialog. The test should complete successfully and you should see the message “message received: ?” in the debug output from the Corba Echo Server, proving that the message is making its way to the Corba code.

Conclusion

You can download the finished code from this article here.

With the aid of the ServiceMix Maven archetypes and a bit of knowledge of JBI4Corba, you can make services offered by legacy Corba applications available to modern Service Oriented Architecture components via an ESB, with all of the advantages that gives.

Check out the rest of the JBI4Corba integration tests, available here for more complex examples.

After all these years dealing with servers, switches, routers and virtualisation technologies we think it´s time to move our lab into the next phase, the Cloud, specifically the Amazon EC2 Cloud.

We are actively working in the Cloud now for different projects, as you´ve seen in previous blog posts. We believe and feel this step is not only a natural one but also takes us in the right direction towards a more effective management of resources and higher business agility. This fits with the needs of a company like ours and we believe it will also fit for many others of different sizes and requirements.
Cloud computing is not only a niche for special projects with very specific needs. It can be used by normal companies to have a more cost effective It infrastructure, at least in certain areas.

In our lab we had a mixture of server configurations, comprising Sun and Dell servers running all kinds of OSs, using VMWare and Sun virtualisation technology. The purpose of our Lab is to provide an infrastructure for our staff, partners and customers to perform specific tests, prototypes, PoC´s, etc… Also, the Lab is our R & D resource to create new architecture solutions.

Moving our Lab to the cloud will provide an infrastructure that will be more flexible, manageable, powerful, simple and definitely more elastic to setup, use and maintain, without removing any of the features we currently have. It will also allow us to concentrate more in this new paradigm, creating advanced cloud architectures and increasing the overall know-how, that can be injected back to customers and the community.

In order to commence this small project the first thing to do was to perform a small feasibility study to identify the different technologies to use inside the cloud to maintain confidentiality and secure access primarily, but also to properly manage and monitor that infrastructure. Additionally, one of the main drivers of this activity was to reduce our monthly hosting cost, so we needed to calculate, based on the current usage, the savings of moving to the cloud.

Cost Study

Looking at the cost for moving to the cloud we performed an inventory of the required CPU power, server instances, storage (for both Amazon S3 and EBS) and the estimated data IO. Additionally, we did an estimation of the volume of data between nodes and between Amazon and the external world.

We initially thought to automatically shutdown and bring up those servers that are only needed during working hours to save more money. In the end, we will be using Amazon reserved instances, that give a even lower per-hour price similar to the one that we would get using on-demand servers.

Based on this inventory and estimations, and with the help of the Amazon Cost Calculator, we reached a final monthly cost that was aprox. 1/3 of our hosting bill!.

This cost is purely considering the physical infrastructure. We need to add on top of this the savings we have on hardware renewal, pure system administration and system installation. Even if we use virtualization technologies, sometimes we´ve had to rearrange things as our physical infrastructure was limited. All these extra costs mean savings on the cloud.

Feasibility Study

Moving to the cloud gives a feeling to most IT managers that they lose control and most importantly, they lose control of the data. While the usage of hybrid clouds can permit the control of the data, in our case we wanted to move everything to the cloud. In this case, we are certainly not different and we are quite paranoid with our data and how would be stored in Amazon EC2. Also, we still require secure network communication between or nodes in the Amazon network and the ability to give secure external access to our staff and customers.

There are a set of open-source technologies that have helped us to materialize these requirements into a solution that we feel comfortable with:

• Filesystem encryption for securing data storage in Amazon EBS.
• Private network and IP range for all nodes
• Cloud-wide encrypted communication between nodes within a private IP network range via OpenVPN solution
• IPSec VPN solution for external permanent access to the Cloud Lab, as for instance connection of private cloud/network to public EC2 Cloud
• Use of RightScale to manage and automate the infrastructure

Overview of TSL Secure Cloud deployment

Implementation and Migration

The implementation of our Cloud Lab solution has gone very smoothly and it is working perfectly.
One of the beneficial side effects you get when migrating different systems into the cloud is that it forces you to be much more organised as the infrastructure is very focused on reutilisation and automatic recreation of the different servers.

We have all our Lab images standardized, taking prebuilt images available in Amazon and customising them to include the security hardening, standard services and conventions we have defined. We can in a matter of seconds deploy new images and include them into our Secure VPN-Based CloudLab network ready to be used.

Our new Cloud Lab is giving us a very stable, cost-effective, elastic and secure infrastructure, which can be rebuilt in minutes using EBS snapshots.