Mike's Site

Adding microbenchmarking to your build process

2016-12-29T11:27:00.001-08:00

Introduction

As an industry, we are adopting higher transparent and more predictable build processes in order to reduce the risks in building software. One of the core principles of Continuous Delivery is to gather feedback via Feedback Loops. At Dev9, we have adopted a "first to know" principle that aligns with the CD principle which means that we (the dev team) wants to be the first to know when there is a failure, degradation of performance or any result not consistent with the business objectives.

Maven and other build tools have provided developers a standardized tool and ecosystem in which to establish and communicate feedback. While unit tests, functional, build acceptance, database migration, performance testing and code analysis tools have become a mainstay in a development pipeline, benchmarking has largely remained outside of the process. This could be due to the lack of open sourced, low cost tooling or lightweight libraries that add minimal complexity.

The existing tools often compound complexity by requiring an outside tool to be integrated with the runtime artifact and the tests are not saved in the same source repository or even stored in a source repository. Local developers are unable to run the benchmarks without effort and therefore the tests lose their value quickly. Adding to the mainstream solution problems, benchmarking is not typically taught in classes and is often implemented without the necessary isolation required to gather credible results. This makes all blogs or posts about benchmark results a ripe target for trolls.

With all that said, it is still very important to put some sort of benchmark coverage around critical areas of your codebase. Building up historical knowledge about critical sections of code can help influence optimization efforts, inform the team about technical debt, alert when a performance threshold change has been committed and compare previous or new versions of algorithms. The question should now be, how do find and easily add benchmarking to my new or existing project. In this blog, we will focus on Java projects (1.7+). The sample code will utilize Maven, though Gradle works very similarly. I make a few recommendations throughout the blog and they are based on experience from past projects.

Introducing JHM

There are many strong choices when looking to benchmark Java based code, but most of them have drawbacks that include license fees, additional tooling, byte code manipulation and/or java agents, tests outlined using non-Java based code and highly complex configuration settings. I like to have tests as close to the code under test as possible to reduce brittleness, lower cohesion and reduce coupling. I consider most of the benchmarking solutions I have previously used to be too cumbersome to work with or the code to run the tests are either not isolated enough (literally integrated in the code) or contained in a secondary solution far from the source.

The purpose of this blog is to demonstrate how to add a lightweight benchmarking tool to your build pipeline so I will not go into detail about how to use JMH, the following blogs are excellent sources to learn:

Benchmarking Modes

There are a small number of items I want to point out with respect to the modes and scoring as they play an important role in how the base configuration is setup. At a basic level, JMH has two main types of measure: throughput and time-based.

Throughput Measuring

Throughput is the amount of operations that can be completed per the unit of time. JMH maintains a collection of successful and failed operations as the framework increases the amount of load on the test. Note: ensure the method or test is well isolated and dependencies like test object creation is done outside of the method or pre-test in a setup method. With Throughput, the higher the value, the better as it indicates that more operations can be run per unit-time.

Time-Based Measuring

Time-based measuring is the counter-partner to throughput. The goal of time-based measuring is to identify how long a particular operation takes to run per unit-time.

AverageTime

The most common time-based measurement is the "AverageTime" which calculates the average time of the operation. JMH will also produce a "Score Error" to help determine confidence in the produced score. The "Score Error" is typically 1/2 of the confidence interval and indicates how close the results deviated from the average time. The lower the result, the better as it indicates a lower average time to run per operation.

SampleTime

SampleTime is similar to AverageTime, but JMH attempts to push more load and look for failures which produces a matrix of failed percentages. With AverageTime, lower numbers are better and the percentages are useful to determine where you are comfortable with failures due to throughput and length of time.

SingleShotTime

The last and least commonly used mode is SingleShotTime. This mode is literally a single run and can be useful for cold testing a method or testing your tests. SingleShotTime could be useful if passed in as a parameter when running benchmarking tests, but reducing the time required to run tests (though, this diminishes the value of the tests and may make them deadweight). As with the rest of the time-based measurements, the lower the value the better.

Adding JMH to a Java Project

Goal: This section will show how to create a repeatable harness that allows new tests to be added with minimal overhead or duplication of code. Note, the dependencies are in the "test" scope to avoid JMH being added to the final artifact. I have created a github repository that uses JMH while working on Protobuf alternative to REST for Microservices. The code can be found here: https://github.com/mike-ensor/protobuf-serialization

1) Start by adding the dependencies to the project:

2) JMH recommends that benchmark tests and the artifact be packaged in the same uber jar. There are several ways to implement an uber jar, explicitly using the "shade" plugin for maven or implicitly using Spring Boot, Dropwizard or some framework with similar results. For the purposes of this blog post, I have used a Spring Boot application.

3) Add a test harness with a main entry class and global configuration. In this step, create an entry point in the test area of your project (indicated with #1). The intention is to avoid having benchmarking code being packaged with the main artifact.

3.1) Add the BenchmarkBase file (indicated above #2). This file will serve as the entry point for the benchmark tests and contain all of the global configuration for the tests. The class I have written looks for a "benchmark.properties" file containing configuration properties (indicated above in #3). JMH has an option to output file results and this configuration is setup for JSON. The results are used in conjunction with your continuous integration tool and can (should) be stored for historical usage.

This code segment is the base harness and entry point into the Benchmark process run by Maven (setup in step #5 below) At this point, the project should be able to run a benchmark test, so let's add a test case.

4) Create a Class to benchmark an operation. Keep in mind, benchmark tests will run against the entirety of the method body, this includes logging, file reading, external resources, etc. Be aware of what you want to benchmark and reduce or remove dependencies in order to isolate your subject code to ensure higher confidence in results. In this example, the configuration setup during

Caption: This gist is a sample benchmark test case extracted from Protobuf Serialization

All of your *Benchmark*.java test classes will now run when you execute the test jar, but this is often not ideal as the process is not segregated and having some control over when and how the benchmarks are run is important to keeping build times down. Let's build a Maven profile to control when the benchmarks are run and potentially start the application. Note, for the purposes of showing that maven integration tests start/stop the server, I have included this in the blog post. I would caution the need to start or stop the application server as you might be incurring the costs of resource fetching (REST calls) which would not be very isolated.

5) The concept is to create a maven profile to run all of the benchmark tests in isolation (ie. no unit or functional tests). This will allow the benchmark tests to be run in parallel with the rest of the build pipeline. Note that the code uses the "exec" plugin and runs the uber jar looking for the full classpath path to the main class. Additionally, the executable scope is only limited to the "test" sources to avoid putting benchmark code into final artifacts.

This code segment shows an example maven profile to run just the Benchmark tests

6) Last, optional item is to create a runnable build step in your Continuous Integration build pipeline. In order to run your benchmark tests in isolation, you or your CI can run:

Conclusion

If you are using a Java based project, JMH is relativly easy to add to your project and pipeline. The benefits of a historical ledger relating to critical areas of your project can be very useful in keeping the quality bar high. Adding JMH to your pipeline also adheres to the Continuous Delivery principles including feedback loops, automation, repeatable, and improving continuously. Consider adding a JMH harness and a few tests to the critical areas of your solution.

Protobuf alternative to REST for Microservices

2016-12-26T21:11:00.000-08:00

Introduction

A few months ago a colleague and long-time friend of mine published an intriguing blog on a few of the less discussed costs associated with implementing microservices. The blog post made several important points on performance when designing and consuming microservices. There is an overhead to using a remote service beyond the obvious network latency due to routing and distance. The blog describes how there is a cost attributed to serialization of JSON and therefore a microservice should do meaningful work to overcome the costs of serialization. While this is a generally accepted guideline for microservices, it is often overlooked and thus a concrete reminder helps to illustrate the point. The second point of interest is the costs associated to the bandwidth size of JSON based RESTful API responses. One potential pitfall of having a more substantive endpoint is that the payload of a response can degrade performance and quickly consume thread pools and overload the network.

These two main points made me think about alternatives and I decided to create an experiment to see if there were benefits from using Google Protocol Buffers (aka, "Protobuf" for short) over JSON in RESTful API calls. I set out to show this by first highlighting performance differences between converting JSON using Jackson into POJOs versus Protobuf messages into and out of the a data model. I decided to create a sufficiently complex data model that utilized nested objects, lists and primitives while trying to keep the model simple to understand; Therefore I ended up with a Recipe domain model that I would probably not use in a serious cooking application, but serves the purpose for this experiment.

Test #1: Measure Costs of Serialization and Deserialization

The first challenge I encountered was how to work effectively with Protobuf messages. After spending some time reading through sparse documentation that focused on an elementary demonstration of Protobuf messages, I finally decided on a method for converting Messages in and out of my domain model. The preceding statements about using Protobufs is opinionated and someone who uses them often may disagree, but my experience was not smooth and I found messages to be rigid and more difficult than I expected.

The second challenge I encountered came when I wanted to measure the "performance" of both marshaling JSON and Serializing Protobufs. I spent some time learning JMH and designed my plan on how to test both methods. Using JMH, I designed a series of tests that allowed me to populate my POJO model, then construct a method that converted into and out of each of the technologies. I isolated the conversion of the objects in order to capture just the costs associated with conversion.

Test #1: Results

My results were not surprising as I expected Protobuf to be more efficient. I measured the average time to marshal an object into JSON at 876.754 ns/operation (±43.222ns) versus 148.160 ns/operation (±6.922ns) showing that equivalent objects converted into Protobuf was nearly 6 times faster than into JSON.

Reversing a JSON and Protobuf message into a POJO yielded slower results and were closer together, but Protobuf still out performed JSON un-marshaling. Converting a JSON string into the domain object took on average 2037.075 ns/operation (±121.997) and Protobuf message to object took on average 844.382 ns/operation (±41.852), nearly 2.4 times faster than JSON.

Serialize/Deserialize times in μSeconds

Run the samples yourself using the github project created for this project: https://github.com/mike-ensor/protobuf-serialization

Test #2: Bandwidth differences

I did not find a straight forward way to capture bandwidth using traditional Java-based tools, so I decided to setup a service on AWS and communicate to the API using JSON and Protobuf requests. I then captured the traffic using Wireshark and calculated the total amount of bytes sent for these requests. I included the headers and payload in the calculation since both JSON and Protobufs require Accepts and Content-Type mime-type headers.

Test #2: Results

The total size of the request for the JSON request was 789 bytes versus the Protobuf at 518 bytes. While the JSON request was 45% greater in size than the Protobuf, there was no optimization applied to either request. The JSON was minified but not compressed. Using compression can be detrimental to the overall performance of the solution based on the payload size. If the payload is too small, the cost of compressing and decompressing will overcome the benefits of a smaller payload. This is a very similar problem to the costs associated with marshaling JSON with small payloads as found by Jeremy's blog.

Conclusion

After completing a project to help determine the overall benefits of using Protobuf over JSON I have come to a conclusion that unless performance is absolutely critical and the developing team's maturity level is high enough to understand the high costs of using Protobufs, then it is a legitimate option to increase the performance associated with message passing. That being said, the costs of working with Protobufs is very high. Developers lose access to human readable messages often useful during debugging. Additionally, Protobufs are messages, not objects and therefore come with more structure and rigger which I found to be complicated due to the inflexibility using only primitives and enums, and updating messages requires the developer to mark new fields as "optional" for backwards compatibility. Lastly, there is limited documentation on Protocol Buffers beyond the basic "hello world" applications.

AES-256 Encryption with Java and JCEKS

2014-02-02T08:47:00.001-08:00

Overview

Security has become a great topic of discussion in the last few years due to the recent releasing of documents from Edward Snowden and the explosion of hacking against online commerce stores like JC Penny, Sony and Target. While this post will not give you all of the tools to help prevent the use of illegally sourced data, this post will provide a starting point for building a set of tools and tactics that will help prevent the use of data by other parties.

This post will show how to adopt AES encryption for strings in a Java environment. It will talk about creating AES keys and storing AES keys in a JCEKS keystore format. A working example of the code in this blog is located at https://github.com/mike-ensor/aes-256-encryption-utility

It is recommended to read each section in order because each section builds off of the previous section, however, this you might want to just jump quickly jump to a particular section.

Setup - Setup and create keys with keytool
Encrypt - Encrypt messages using byte[] keys
Decrypt - Decrypt messages using same IV and key from encryption
Obtain Keys from Keystore - Obtain keys from keystore via an alias

What is JCEKS?

JCEKS stands for Java Cryptography Extension KeyStore and it is an alternative keystore format for the Java platform. Storing keys in a KeyStore can be a measure to prevent your encryption keys from being exposed. Java KeyStores securely contain individual certificates and keys that can be referenced by an alias for use in a Java program. Java KeyStores are often created using the "keytool" provided with the Java JDK.

NOTE: It is strongly recommended to create a complex passcode for KeyStores to keep the contents secure. The KeyStore is a file that is considered to be public, but it is advisable to not give easy access to the file.

Setup

All encryption is governed by laws of each country and often have restrictions on the strength of the encryption. One example is that in the United States, all encryption over 128-bit is restricted if the data is traveling outside of the boarder. By default, the Java JCE implements a strength policy to comply with these rules. If a stronger encryption is preferred, and adheres to the laws of the country, then the JCE needs to have access to the stronger encryption policy. Very plainly put, if you are planning on using AES 256-bit encryption, you must install the Unlimited Strength Jurisdiction Policy Files. Without the policies in place, 256-bit encryption is not possible.

Installation of JCE Unlimited Strength Policy

This post is focusing on the keys rather than the installation and setup of the JCE. The installation is rather simple with explicit instructions found here (NOTE: this is for JDK7, if using a different JDK, search for the appropriate JCE policy files).

Keystore Setup

When using the KeyTool manipulating a keystore is simple. Keystores must be created with a link to a new key or during an import of an existing keystore. In order to create a new key and keystore simply type:

keytool -genseckey -keystore aes-keystore.jck -storetype jceks -storepass mystorepass -keyalg AES -keysize 256 -alias jceksaes -keypass mykeypass

Important Flags

In the example above here are the explanations for the keytool's parameters:

Keystore Parameters

genseckey: Generate SecretKey. This is the flag indicating the creation of a synchronous key which will become our AES key
keystore: Location of the keystore. If the keystore does not exist, the tool will create a new store. Paths can be relative or absolute but must be local
storetype: this is the type of store (JCE, PK12, JCEKS, etc). JCEKS is used to store symmetric keys (AES) not contained within a certificate.
storepass: password related to the keystore. Highly recommended to create a strong passphrase for the keystore

Key Parameters

keyalg: algorithm used to create the key (AES/DES/etc)
keysize: size of the key (128, 192, 256, etc)
alias: alias given to the newly created key in which to reference when using the key
keypass: password protecting the use of the key

Encrypt

As it pertains to data in Java and at the most basic level, encryption is an algorithmic process used to programmatically obfuscate data through a reversible process where both parties have information pertaining to the data and how the algorithm is used. In Java encryption, this involves the use of a Cipher. A Cipher object in the JCE is a generic entry point into the encryption provider typically selected by the algorithm. This example uses the default Java provider but would also work with Bouncy Castle.

Generating a Cipher object

Obtaining an instance of Cipher is rather easy and the same process is required for both encryption and decryption. (NOTE: Encryption and Decryption require the same algorithm but do not require the same object instance)

Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

Once we have an instance of the Cipher, we can encrypt and decrypt data according to the algorithm. Often the algorithm will require additional pieces of information in order to encrypt/decrypt data. In this example, we will need to pass the algorithm the bytes containing the key and an initial vector (explained below).

Initialization

In order to use the Cipher, we must first initialize the cipher. This step is necessary so we can provide additional information to the algorithm like the AES key and the Initial Vector (aka IV).

cipher.init(Cipher.ENCRYPT_MODE, secretKeySpecification, initialVector);

Parameters

The SecretKeySpecification is an object containing a reference to the bytes forming the AES key. The AES key is nothing more than a specific sized byte array (256-bit for AES 256 or 32 bytes) that is generated by the keytool (see above).

Alternative Parameteters

There are multiple methods to create keys such as a hash including a salt, username and password (or similar). This method would utilize a SHA1 hash of the concatenated strings, convert to bytes and then truncate result to the desired size. This post will not show the generation of a key using this method or the use of a PBE key method using a password and salt. The password and/or salt usage for the keys is handled by the keytool using the inputs during the creation of new keys.

Initialization Vector

The AES algorithm also requires a second parameter called the Initialiation Vector. The IV is used in the process to randomize the encrypted message and prevent the key from easy guessing. The IV is considered a publicly shared piece of information, but again, it is not recommended to openly share the information (for example, it wouldn't be wise to post it on your company's website). When encrypting a message, it is not uncommon to prepend the message with the IV since the IV will be a set/known size based on the algorithm. NOTE: the AES algorithm will output the same result if using the same IV, key and message. It is recommended that the IV be randomly created each time an encryption takes place.

With the newly initialized Cipher, encrypting a message is simple. Simply call:

byte[] encryptedMessageInBytes = Cipher.doFinal((message.getBytes("UTF-8"));
String base64EncodedEncryptedMsg = BaseEncoding.base64().encode(encryptedMessageInBytes);
String base32EncodedEncryptedMsg = BaseEncoding.base32().encode(encryptedMessageInBytes);

Encoding Results

Byte arrays are difficult to visualize since they often do not form characters in any charset. The best recommendation to solve this is to represent the bytes in HEX (base-16), Double HEX (base-32) or Base64 format. If the message will be passed via a URL or POST parameter, be sure to use a web-safe Base64 encoding. Google Guava library provides a excellent BaseEncoding utility. NOTE: Remember to decode the encoded message before decrypting.

Decrypt

Decrypting a message is almost a reverse of the encryption process with a few exceptions. Decryption requires a known initialization vector as a parameter unlike the encryption process generating a random IV.

Decryption

When decrypting, obtain a cipher object with the same process as the encryption method. The Cipher object will need to utilize the exact same algorithm including the method and padding selections. Once the code has obtained a reference to a Cipher object, the next step is to initialize the cipher for decryption and pass in a reference to a key and the initialization vector.

// key is the same byte[] key used in encryption
SecretKeySpec secretKeySpecification = new SecretKeySpec(key, "AES");
cipher.init(Cipher.DECRYPT_MODE, secretKeySpecification, initialVector);

NOTE: The key is stored in the keystore and obtained by the use of an alias. See below for details on obtaining keys from a keystore

Once the cipher has been provided the key, IV and initialized for decryption, the cipher is ready to perform the decryption.

byte[] encryptedTextBytes = BaseEncoding.base64().decode(message);
byte[] decryptedTextBytes = cipher.doFinal(encryptedTextBytes);
String origMessage = new String(decryptedTextBytes);

Strategies to keep IV

The IV used to encrypt the message is important to decrypting the message therefore the question is raised, how do they stay together. One solution is to Base Encode (see above) the IV and prepend it to the encrypted and encoded message: Base64UrlSafe(myIv) + delimiter + Base64UrlSafe(encryptedMessage). Other possible solutions might be contextual such as including an attribute in an XML file with the IV and one for the alias to the key used.

Obtain Key from Keystore

The beginning of this post has shown how easy it is to create new AES-256 keys that reference an alias inside of a keystore database. The post then continues on how to encrypt and decrypt a message given a key, but has yet shown how to obtain a reference to the key in a keystore.

Solution

// for clarity, ignoring exceptions and failures
InputStream keystoreStream = new FileInputStream(keystoreLocation);

KeyStore keystore = KeyStore.getInstance("JCEKS");
keystore.load(keystoreStream, keystorePass.toCharArray());

if (!keystore.containsAlias(alias)) {
    throw new RuntimeException("Alias for key not found");
}

Key key = keystore.getKey(alias, keyPass.toCharArray());

Parameters

keystoreLocation: String - Location to local keystore file location
keypass: String - Password used when creating or modifying the keystore file with keytool (see above)
alias: String - Alias used when creating new key with keytool (see above)

Conclusion

This post has shown how to encrypt and decrypt string based messages using the AES-256 encryption algorithm. The keys to encrypt and decrypt these messages are held inside of a JCEKS formatted KeyStore database created using the JDK provided "keytool" utility. The examples in this post should be considered a solid start to encrypting/decrypting symmetric keys such as AES. This should not be considered the only line of defense when encrypting messages, for example key rotation. Key rotation is a method to mitigate risks in the event of a data breach. If an intruder obtains data and manages to hack a single key, the data contained in multiple files should have used several keys to encrypt the data thus bringing down risk of a total exposure loss.

All of the examples in this blog post have been condensed into a simple tool allowing for the viewing of keys inside of a keystore, an operation that is not supported out of the box by the JDK keytool. Each aspect of the steps and topics outlined in this post are available at: https://github.com/mike-ensor/aes-256-encryption-utility. NOTE: The examples, sample code and any reference is to be used at the sole implementers risk and there is no implied warranty or liability, you assume all risks.

How to Publish Maven Site Docs to BitBucket or GitHub Pages

2013-01-22T23:14:00.000-08:00

Introduction

In this post we will Utilize GitHub and/or BitBucket's static web page hosting capabilities to publish our project's Maven 3 Site Documentation. Each of the two SCM providers offer a slightly different solution to host static pages. The approach spelled out in this post would also be a viable solution to "backup" your site documentation in a supported SCM like Git or SVN. This solution does not directly cover site documentation deployment covered by the maven-site-plugin and the Wagon library (scp, WebDAV or FTP).

There is one main project hosted on GitHub that I have posted with the full solution. The project URL is https://github.com/mike-ensor/clickconcepts-master-pom/. The POM has been pushed to Maven Central and will continue to be updated and maintained.

<parent>
    <groupId>com.clickconcepts.project</groupId>
    <artifactId>master-site-pom</artifactId>
    <version>0.16</version>
</parent>

GitHub Pages

GitHub hosts static pages by using a special branch "gh-pages" available to each GitHub project. This special branch can host any HTML and local resources like JavaScript, images and CSS. There is no server side development.

To navigate to your static pages, the URL structure is as follows:

http://<some-username>.github.com/<project-name>

An example of the project I am using in this blog post: http://mike-ensor.github.com/clickconcepts-master-pom/ where the first bold URL segment is a username and the second bold URL segment is the project.

GitHub does allow you to create a base static hosted static site for your username by creating a repository with your username.github.com. The contents would be all of your HTML and associated static resources. This is not required to post documentation for your project, unlike the BitBucket solution.

There is a GitHub Site plugin that publishes site documentation via GitHub's object API but this is outside the scope of this blog post because it does not provide a single solution for GitHub and BitBucket projects using Maven 3.

BitBucket

BitBucket provides a similar service to GitHub in that it hosts static HTML pages and their associated static resources. However, there is one large difference in how those pages are stored. Unlike GitHub, BitBucket requires you to create a new repository with a name fitting the convention. The files will be located on the master branch and each project will need to be a directory off of the root.

mikeensor.bitbucket.org/
     /some-project
      +index.html
      +...
          /css
          /img
     /some-other-project
      +index.html
      +...
          /css
          /img
index.html
.git
.gitignore

The naming convention is as follows:

<username>.bitbucket.org

An example of a BitBucket static pages repository for me would be: http://mikeensor.bitbucket.org/. The structure does not require that you create an index.html page at the root of the project, but it would be advisable to avoid 404s.

Generating Site Documentation

Maven provides the ability to post documentation for your project by using the maven-site-plugin. This plugin is difficult to use due to the many configuration options that oftentimes are not well documented. There are many blog posts that can help you write your documentation including my post on maven site documentation. I did not mention how to use "xdoc", "apt" or other templating technologies to create documentation pages, but not to fear, I have provided this in my GitHub project.

Putting it all Together

The Maven SCM Publish plugin (http://maven.apache.org/plugins/maven-scm-publish-plugin/ publishes site documentation to a supported SCM. In our case, we are going to use Git through BitBucket or GitHub. Maven SCM Plugin does allow you to publish multi-module site documentation through the various properties, but the scope of this blog post is to cover single/mono module projects and the process is a bit painful.

Take a moment to look at the POM file located in the clickconcepts-master-pom project. This master POM is rather comprehensive and the site documentation is only one portion of the project, but we will focus on the site documentation. There are a few things to point out here, first, the scm-publish plugin and the idiosyncronies when implementing the plugin.

In order to create the site documentation, the "site" plugin must first be run. This is accomplished by running site:site. The plugin will generate the documentation into the "target/site" folder by default.

The SCM Publish Plugin, by default, looks for the site documents to be in "target/staging" and is controlled by the content parameter. As you can see, there is a mismatch between folders. NOTE: My first approach was to run the site:stage command which is supposed to put the site documents into the "target/staging" folder. This is not entirely correct, the site plugin combines with the distributionManagement.site.url property to stage the documents, but there is very strange behavior and it is not documented well.

In order to get the site plugin's site documents and the SCM Publish's location to match up, use the content property and set that to the location of the Site Plugin output (<siteOutputDirectory>).

If you are using GitHub, there is no modification to the siteOutputDirectory needed, however, if you are using BitBucket, you will need to modify the property to add in a directory layer into the site documentation generation (see above for differences between GitHub and BitBucket pages). The second property will tell the SCM Publish Plugin to look at the root "site" folder so that when the files are copied into the repository, the project folder will be the containing folder. The property will look like:

<siteOutputDirectory>${project.build.directory}/site/${project.artifactId}</siteOutputDirectory>
<scm-publish.siteDocOuputDirectory>${project.build.directory}/site</scm-publish.siteDocOuputDirectory>

Next we will take a look at the custom properties defined in the master POM and used by the SCM Publish Plugin above. Each project will need to define several properties to use the Master POM that are used within the plugins during the site publishing. Fill in the variables with your own settings.

BitBucket

<!-- Override Site Documentation SCM publishing parameters -->
<properties>
...
...
<scm-publish.scmBranch>master</scm-publish.scmBranch>
<scm-publish.pubScmUrl>scm:git:git@bitbucket.org:mikeensor/mikeensor.bitbucket.org.git</scm-publish.pubScmUrl>

<!-- Location of where "site" documentation is output; This is for BitBucket only!!! -->
<siteOutputDirectory>${project.build.directory}/site/${project.artifactId}</siteOutputDirectory>
<scm-publish.siteDocOuputDirectory>${project.build.directory}/site</scm-publish.siteDocOuputDirectory>

<!-- Overwrite from Parent Pom  -->
<changelog.fileUri>${changelog.bitbucket.fileUri}</changelog.fileUri>
<changelog.revision.fileUri>${changelog.revision.bitbucket.fileUri}</changelog.revision.fileUri>
...
...
</properties>

GitHub

<!-- Override Site Documentation SCM publishing parameters -->
<properties>
...
...
<scm-publish.scmBranch>gh-pages</scm-publish.scmBranch>
<scm-publish.pubScmUrl>scm:git:git@github.com:mikeensor/clickconcepts-master-pom.git</scm-publish.pubScmUrl>

<!-- Location of where "site" documentation is output; This is for 
<scm-publish.siteDocOuputDirectory>${project.build.directory}/site</scm-publish.siteDocOuputDirectory>

<!-- Overwrite from Parent Pom  -->
<changelog.fileUri>${changelog.github.fileUri}</changelog.fileUri>
<changelog.revision.fileUri>${changelog.revision.github.fileUri}</changelog.revision.fileUri>
...
...
</properties>

NOTE: changelog parameters are required to use the Master POM and are not directly related to publishing site docs to GitHub or BitBucket

How to Generate

If you are using the Master POM (or have abstracted out the Site Plugin and the SCM Plugin) then to generate and publish the documentation is simple.

mvn clean site:site scm-publish:publish-scm

mvn clean site:site scm-publish:publish-scm -Dscmpublish.dryRun=true

Gotchas

In the SCM Publish Plugin documentation's "tips" they recommend creating a location to place the repository so that the repo is not cloned each time. There is a risk here in that if there is a git repository already in the folder, the plugin will overwrite the repository with the new site documentation. This was discovered by publishing two different projects and having my root repository wiped out by documentation from the second project. There are ways to mitigate this by adding in another folder layer, but make sure you test often!

Another gotcha is to use the -Dscmpublish.dryRun=true to test out the site documentation process without making the SCM commit and push

Project and Documentation URLs

Here is a list of the fully working projects used to create this blog post:

Master POM with Site and SCM Publish plugins &ndash https://github.com/mike-ensor/clickconcepts-master-pom. Documentation URL: http://mike-ensor.github.com/clickconcepts-master-pom/
Child Project using Master Pom &ndash http://mikeensor.bitbucket.org/fest-expected-exception. Documentation URL: http://mikeensor.bitbucket.org/fest-expected-exception/

How to test a Custom Exception using custom FEST assertions

2012-11-19T18:37:00.001-08:00

Introduction

This is part three of my posts on assertions testing using Fest, JUnit and custom Exceptions. The first post was covering the basics of assertions, then followed up with testing custom Exceptions using JUnit and JUnit's ExpectedException class. At this point you should know that you have a custom Runtime Exception class and you would like to test it using the Fluent API provided by FEST.

Custom Assertions with Fest

This blog post will not go into the details on creating a custom assertion, but the solution posted in the Github project does contain a custom assertion. In addition, please refer to the official site docs.

Building off of the last post, you will see that ExpectedException is a great improvement over the @Test(expected) and Try/Catch techniques, however the ExpectedException object can still be improved by adding in a fluent-style API backed by Fest Assertion project. So, how do you do it? Let's get right to the solution!

Expected Exceptions with FEST and JUnit @Rule

Now that we have an understanding of FEST assertions, JUnit's @Rule functionality and ClickConcept's @ExpectedFailure we can combine the first two to provide fluent-style expected exception behavior while testing the assertion class using @ExpectedFailure annotations.

Testing your custom exception with FEST

Let's begin by creating a new @Rule object "ExpectedException" which extends TestRule. When creating the class, we will expose the construction through a simple factory method to return a new ExpectedException. The default factory will return a base implementation where the functionality is muted in all other cases where exceptions are not desired.

We can start out with the code first, but I will explain that in order to build your own custom Fluent API for FEST, you must re-create the API for the base exception assertion. The fluent API you create will be in addition to the FEST exception assertion class. Fluent API help was derived off of several blogs, but the most informative has been http://www.unquietcode.com/blog/2011/programming/using-generics-to-build-fluent-apis-in-java/.

NOTE: AbstractExpectedException encapsulates the base API for FEST's ExceptionAssertion. The code for this is found at the Github site: https://github.com/mike-ensor/fest-backed-expected-exception


public class ExpectedCustomException extends AbstractExpectedException<ExpectedCustomException> {

    private Integer code;

    public static ExpectedCustomException none() {
        return new ExpectedCustomException();
    }

    /**
     * Checks to see if the CustomException has the specified code
     *
     * @param code int
     * @return AbstractExpectedException
     */
    public AbstractExpectedException hasCode(int code) {
        // method telling class that a custom exception is being asked for
        markExpectedException();
        this.code = code;
        return this;
    }

    @Override
    protected void checkAssertions(Exception e) {
        // check parent's exceptions
        super.checkAssertions(e);

        if (getCode() != null) {
            // FEST Custom Assert object
            CustomExceptionAssert.assertThat(e).hasCode(code);
        }
    }

    private Integer getCode() {
        return code;
    }

}

Analysis

In this example, my CustomException has exposed a "code" to store when the exception was created. In order to test this my custom ExpectedException object must look for the proper code on the CustomException object, in a fluent manor.

Here is an example test case to explain how to use your new fluent API Custom Exception test. Take note of the third test case to see the Fluent API in use! (NOTE: Full test cases are available on my github account.

public class CustomExceptionTest {

    @Rule
    public ExpectedCustomException exception =
            ExpectedCustomException.none();

    @Rule
    public ExpectedTestFailureWatcher expectedTestFailureWatcher =
            ExpectedTestFailureWatcher.instance();

    @Test
    public void hasCode_worksAsExpected() {
        exception.hasCode(123);
        throw new CustomException("Message", 123);
    }

    @Test
    @ExpectedFailure
    public void getCode_fails() {
        exception.hasCode(456);
        throw new CustomException("Message", 123);
    }

    @Test
    @ExpectedFailure
    public void getMessageAndCode_codeFailsFirst() {
        exception.hasCode(456).hasMessage("Message");
        throw new CustomException("Message", 123);
    }

}

Summary

Thank you to those of you who have read through this little series covering assertions, how to test exceptions (both the exception flow and custom exceptions) and then on to testing your custom exceptions using FEST assertions. Please come back to my blog in the near future where I will have a REST API checklist to look over when architecting your next REST API.

Those who are reading this blog are most likely a small subset of the software development community, but if you are not, and you find the idea of a fluent-API really cool (as I do), please check out the FEST assertion library. If you are new to test driven development, please take up the practice and try applying to your code immediately. If all developers used TDD as a general practice, the level in quality would grow world wide!

Testing Custom Exceptions w/ JUnit's ExpectedException and @Rule

2012-09-25T23:27:00.001-07:00

Exception Testing

Why test exception flows? Just like with all of your code, test coverage writes a contract between your code and the business functionality that the code is supposed to produce leaving you with a living documentation of the code along with the added ability to stress the functionality early and often. I won't go into the many benefits of testing instead I will focus on just Exception Testing.

There are many ways to test an exception flow thrown from a piece of code. Lets say that you have a guarded method that requires an argument to be not null. How would you test that condition? How do you keep JUnit from reporting a failure when the exception is thrown? This blog covers a few different methods culminating with JUnit's ExpectedException implemented with JUnit's @Rule functionality.

The "old" way

In a not so distant past the process to test an exception required a dense amount of boilerplate code in which you would start a try/catch block, report a failure if your code did not produce the expected behavior and then catch the exception looking for the specific type. Here is an example:

public class MyObjTest {

    @Test
    public void getNameWithNullValue() {

        try {
            MyObj obj = new MyObj();
            myObj.setName(null);
            
            fail("This should have thrown an exception");

        } catch (IllegalArgumentException e) {
            assertThat(e.getMessage().equals("Name must not be null"));
        }
    }
}

As you can see from this old example, many of the lines in the test case are just to support the lack of functionality present to specifically test exception handling. One good point to make for the try/catch method is the ability to test the specific message and any custom fields on the expected exception. We will explore this a bit further down with JUnit's ExpectedException and @Rule annotation.

JUnit adds expected exceptions

JUnit responded back to the users need for exception handling by adding a @Test annotation field "expected". The intention is that the entire test case will pass if the type of exception thrown matched the exception class present in the annotation.

public class MyObjTest {

    @Test(expected = IllegalArgumentException.class)
    public void getNameWithNullValue() {
        MyObj obj = new MyObj();
        myObj.setName(null);
    }
}

As you can see from the newer example, there is quite a bit less boiler plate code and the test is very concise, however, there are a few flaws. The main flaw is that the test condition is too broad. Suppose you have two variables in a signature and both cannot be null, then how do you know which variable the IllegalArgumentException was thrown for? What happens when you have extended a Throwable and need to check for the presence of a field? Keep these in mind as you read further, solutions will follow.

JUnit @Rule and ExpectedException

If you look at the previous example you might see that you are expecting an IllegalArgumentException to be thrown, but what if you have a custom exception? What if you want to make sure that the message contains a specific error code or message? This is where JUnit really excelled by providing a JUnit @Rule object specifically tailored to exception testing. If you are unfamiliar with JUnit @Rule, read the docs here.

ExpectedException

JUnit provides a JUnit class ExpectedException intended to be used as a @Rule. The ExpectedException allows for your test to declare that an exception is expected and gives you some basic built in functionality to clearly express the expected behavior. Unlike the @Test(expected) annotation feature, ExpectedException class allows you to test for specific error messages and custom fields via the Hamcrest matchers library.

An example of JUnit's ExpectedException

import org.junit.rules.ExpectedException;

public class MyObjTest {

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void getNameWithNullValue() {
        thrown.expect(IllegalArgumentException.class);
        thrown.expectMessage("Name must not be null");

        MyObj obj = new MyObj();
        obj.setName(null);
    }
}

As I eluded to above, the framework allows you to test for specific messages ensuring that the exception being thrown is the case that the test is specifically looking for. This is very helpful when the nullability of multiple arguments is in question.

Custom Fields

Arguably the most useful feature of the ExpectedException framework is the ability to use Hamcrest matchers to test your custom/extended exceptions. For example, you have a custom/extended exception that is to be thrown in a method and inside the exception has an "errorCode". How do you test that functionality without introducing the boiler plate code from the try/catch block listed above? How about a custom Matcher!

This code is available at: https://github.com/mike-ensor/custom-exception-testing

Solution: First the test case

import org.junit.rules.ExpectedException;

public class MyObjTest {

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void someMethodThatThrowsCustomException() {
        thrown.expect(CustomException.class);
        thrown.expect(CustomMatcher.hasCode("110501"));

        MyObj obj = new MyObj();
        obj.methodThatThrowsCustomException();
    }
}

Solution: Custom matcher

import com.thepixlounge.exceptions.CustomException;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;

public class CustomMatcher extends TypeSafeMatcher<CustomException> {

    public static BusinessMatcher hasCode(String item) {
        return new BusinessMatcher(item);
    }

    private String foundErrorCode;
    private final String expectedErrorCode;

    private CustomMatcher(String expectedErrorCode) {
        this.expectedErrorCode = expectedErrorCode;
    }

    @Override
    protected boolean matchesSafely(final CustomException exception) {
        foundErrorCode = exception.getErrorCode();
        return foundErrorCode.equalsIgnoreCase(expectedErrorCode);
    }

    @Override
    public void describeTo(Description description) {
        description.appendValue(foundErrorCode)
                .appendText(" was not found instead of ")
                .appendValue(expectedErrorCode);
    }
}

NOTE: Please visit https://github.com/mike-ensor/custom-exception-testing to get a copy of a working Hamcrest Matcher, JUnit @Rule and ExpectedException.

And there you have it, a quick overview of different ways to test Exceptions thrown by your code along with the ability to test for specific messages and fields from within custom exception classes. Please be specific with your test cases and try to target the exact case you have setup for your test, remember, tests can save you from introducing side-effect bugs!

Brief Overview of Java Assertions

2012-09-18T22:55:00.003-07:00

What are asserts?

An assertion is a predicate (a true–false statement) placed in a program to indicate that the developer thinks that the predicate is always true at that place. [wikipedia]

Traditional asserts

Traditional testing frameworks started with the built in keyword assert.

An example of the keyword assert:

assert <condition> value

Assert has a few drawbacks including stopping the test execution and lengthy and hard to describe assert statements.

Second generation assertions

Along comes JUnit's assert framework. Built on top of the assert keyword, JUnit provided developers the ability to be more descriptive about the testing statements.

An example of JUnit's asserts:

// asserts that the condition must be true
assertTrue("This should be true", "abc".equalsIgnoreCase("ABC"));
// asserts that the object must not be null
assertNotNull(new MyObject());
//...
assertFalse(false == true);
//
assertNull(null);
// etc...

While there are some improvements on readability and usability to the basic assert keyword provided by JUnit, they share some of the same drawbacks in that many developers just use the "assertTrue()", "assertEquals()" and "assertFalse()" methods still providing a very cryptic assertion statement.

Third generation assertions

In an effort to guide developers into writing test assertions that are more readable and usable the Hamcrest library was created which switched the philosophy from many assert functions to just one basic function. The fundamental thought is that the assert is the same but the conditions will change. Hamcrest was built using the concepts of BDD (behavior driven design) where the test assertion is closer to that of a sentence.

An example of hamcrest assertions

@Test
public void showoffSomeHamcrestAssertsAndMatchers() {
    // asserts that string "abc" is "ABC" ignoring case
    assertThat("abc", is(equalToIgnoringCase("ABC")));
    assertThat(myObject.getFirst(), is("Mike!"));
    assertThat(myObject.getAddress(), is(notNullValue));
}

Hamcrest is a great improvement on top of the JUnit framework providing a flexible and readable testing platform. Hamcrest + JUnit is a comprehensive testing framework and when combined with Mockito (or other mocking framework) can provide a very descriptive and thorough unit testing solution. One of the drawbacks to using Hamcrest is that while descriptive, multiple assertions must be made to ensure that a test case has been covered. Many TDD purest agree that a test case should contain one and only one assertion, but how is this possible with a complex object? (purest will say refactoring, but oftentimes this is not feasible)

Fourth generation frameworks

And finally we come to the present with the latest assertion frameworks. Fest Assertion framework takes off where Hamcrest stopped providing a fluent style assertion framework giving the developer the ability to test objects with a single assertion.

An example of Fest assertions

@Test
public void getAddressOnStreet() {
    List<Address> addresses = addressDAO.liveOnStreet("main");
    assertThat(addresses).hasSize(10).contains(address1, address2);
    assertThat(stringObj).hasSize(7).isEqualToIgnoringCase("abcdefg");
}

As you can see, FEST exceptions provide a cleaner approach to assertions. FEST is very extensible and easy to use.

Conclusion

Assertions are a key tool in a professional developer's toolbox in which to stress test their code. This post is a part in a series with the culmination being a fluent style ExpectedException mechanism, backed by Fest, in which to add better clarity to your test cases when exceptions are being introduced. Feel free to read the lead up to this post; Allowing known failing JUnit Tests to pass test cases.

NOTE: This blog is providing an overview on assertion frameworks it should be noted that there are various other second and third generation testing frameworks that strive to provide better clarity and usability for testing including notables such as TestNG and JTest

Allowing JUnit Tests to Pass Test Case on Failures

2012-09-09T20:51:00.000-07:00

Why create a mechanism to expect a test failure?

There comes a time when one would want and expect a JUnit @Test case fail. Though this is pretty rare, it happens. I had the need to detect when a JUnit Test fails and then, if expected, to pass instead of fail. The specific case was that I was testing a piece of code that could throw an Assert error inside of a call of the object. The code was written to be an enhancement to the popular new Fest Assertions framework, so in order to test the functionality, one would expect test cases to fail on purpose.

A Solution

Jump to the Code
One possible solution is to utilize the functionality provided by a JUnit @Rule in conjunction with a custom marker in the form of an annotation.

Why use a @Rule?

@Rule objects provide an AOP-like interface to a test class and each test cases. Rules are reset prior to each test case being run and they expose the workings of the test case in the style of an @Around AspectJ advice would.

Required code elements

@Rule object to check the status of each @Test case
@ExpectedFailure custom marker annotation
Test cases proving code works!
Optional specific exception to be thrown if annotated test case does not fail

NOTE: working code is available on my github page and has been added to Maven Central. Feel free to Fork the project and submit a pull request

Maven Usage

<dependency>
    <groupId>com.clickconcepts.junit</groupId>
    <artifactId>expected-failure</artifactId>
    <version>0.0.9</version>
</dependency>

Example Usage

In this example, the "exception" object is a Fest assertion enhanced ExpectedException (look for my next post to expose this functionality). The expected exception will make assertions and in order to test those, the test case must be marked as @ExpectedFailure

public class ExceptionAssertTest {

    @Rule
    public ExpectedException exception = ExpectedException.none();

    @Rule
    public ExpectedTestFailureWatcher watcher = ExpectedTestFailureWatcher.instance();

    @Test
    @ExpectedFailure("The matcher should fail becasue exception is not a SimpleException")
    public void assertSimpleExceptionAssert_exceptionIsOfType() {
        // expected exception will be of type "SimpleException"
        exception.instanceOf(SimpleException.class);
        // throw something other than SimpleException...expect failure
        throw new RuntimeException("this is an exception");
    }
}

Implementation of Solution

Reminder, the latest code is available on my github page.

@Rule code (ExpectedTestFailureWatcher.java)

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
// YEAH Guava!!
import static com.google.common.base.Strings.isNullOrEmpty;

public class ExpectedTestFailureWatcher implements TestRule {

    /**
     * Static factory to an instance of this watcher
     *
     * @return New instance of this watcher
     */
    public static ExpectedTestFailureWatcher instance() {
        return new ExpectedTestFailureWatcher();
    }

    @Override
    public Statement apply(final Statement base, final Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                boolean expectedToFail = description.getAnnotation(ExpectedFailure.class) != null;
                boolean failed = false;
                try {
                    // allow test case to execute
                    base.evaluate();
                } catch (Throwable exception) {
                    failed = true;
                    if (!expectedToFail) {
                        throw exception; // did not expect to fail and failed...fail
                    }
                }
                // placed outside of catch
                if (expectedToFail && !failed) {
                    throw new ExpectedTestFailureException(getUnFulfilledFailedMessage(description));
                }
            }

            /**
             * Extracts detailed message about why test failed
             * @param description
             * @return
             */
            private String getUnFulfilledFailedMessage(Description description) {
                String reason = null;
                if (description.getAnnotation(ExpectedFailure.class) != null) {
                    reason = description.getAnnotation(ExpectedFailure.class).reason();
                }
                if (isNullOrEmpty(reason)) {
                    reason = "Should have failed but didn't";
                }
                return reason;
            }
        };
    }
}

@ExpectedFailure custom annotation (ExpectedFailure.java)

import java.lang.annotation.*;

/**
 * Initially this is just a marker annotation to be used by a JUnit4 Test case in conjunction
 * with ExpectedTestFailure @Rule to indicate that a test is supposed to be failing
 */
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target(value = ElementType.METHOD)
public @interface ExpectedFailure {
    // TODO: enhance by adding specific information about what type of failure expected
    //Class assertType() default Throwable.class;

    /**
     * Text based reason for marking test as ExpectedFailure
     * @return String
     */
    String reason() default "";
}

Custom Exception (Optional, you can easily just throw RuntimeException or existing custom exception)

public class ExpectedTestFailureException extends Throwable {
    public ExpectedTestFailureException(String message) {
        super(message);
    }
}

Can't one exploit the ability to mark a failure as expected?

With great power comes great responsibility, it is advised that you do not mark a test as being @ExpectedFailure if you do not understand exactly why the test if failing. It is recommended that this testing method be implemented with care. DO NOT use the @ExpectedFailure annotation as an alternative to @Ignore

Possible future enhancements could include ways to specify the specific assertion or the specific message asserted during the test case execution.

Known issues

In this current state, the @ExpectedFailure annotation can cover up additional assertions and until the future enhancements have been put into place, it is advised to use this methodology wisely.

Maven 3 Site Docs Part 1: Basic Site Documentation

2012-03-19T17:52:00.000-07:00

What is documentation?

Documentation: The process of documenting knowledge

Documentation for software developers are documents that describe how to run or use the artifact/project. Typically documentation provides any exposed settings, their uses and default values. In addition information about integration, entry and exit points such as JMS queues, JMX MBeans, DataSource objects and/or JNDI variables in order to run and use the artifact. There are many programs and standards to produce this level of documentation including MS word, Sharepoint, doxygen, Latex, Wiki, markdown, textile, etc.

Outside of artifact specific documentation team leads, architects and project managers usually ask for JavaDocs to be included. JavaDocs, if combined with Maven provide more documentation for your IDE and better information for the consumers of your artifact. Other important documentation includes testing reports, test coverage, static analysis tools, dependencies of your artifact, change log and issue management such as Jira or Bugzilla.

Why create docs?

The best reason to care about documentation is to provide information for external developers and team members outside of the project's development environment. Documentation can be utilized inside of IDEs as well and face it, how many times do you wish you had sources and/or JavaDocs for a dependency when you were programming or to send back an email with the site docs so someone can be self-served rather than take your time.

Problems with traditional documentation

The largest complaint with traditional documentation is that it becomes stale over time. Time is never allocated in a project^&dagger to provide or update documentation and stale docs can lead a developer or team down many wrong paths leading to hours or days of misinformation lead rabbit holes.

^{† Agile, SCRUM and Lean processes should allow time to be built-in to handle technical debt like documentation, even thought it is called the "necessary evil". Read more here}

Secondly, developers are the ones who are ~~asked~~forced to maintain the documentation and typically developers are not great in the writing department and unless segments of the documentation process are automated, developers are people and we forget.

Last major problem with traditional documentation is that it is typically not automated and is an extra step during the build/release cycle. Any process in the build cycle that is not automated and tested will break down at some point. Humans are great at abstract thinking, not repetitive processes.

Solution to the problems

As stated above, the main problems are developers maintaining all documentation and documentation that is either built outside of the immediate environment or is not automated. This is where Maven's site documentation comes in. Maven was built with the understanding that documentation needs to be automated and has been progressively improved throughout updates to the maven project.

The most common case for Maven site docs is to deploy generated documentation to an external web server via SCP, FTP, WebDAV or SFTP. The site plugin takes use of Apache's Wagon library for secure transfer of docs. Most of the time the site docs are generated during the release† or deploy phases and generate on the verify phase.

^{† Release isn't a phase in maven, it is a plugin, but it is considered a best practice when publishing versioned artifacts to a maven repository}

Maven 3 vs Maven 2

Maven decided to change the way that reports were collected and generated in between the Maven 2 and Maven 3 releases. In Maven 2, the POM would have a section called >reports< where plugins that handled ReportSet would be handled. In Maven 3, there is now a plugin that handles and collates the ReportSets instead of Maven handling it. The plugin is called maven-project-info-reports-plugin and documentation can be found here.

Sources for this series of blog post

I have provided a link to a fully functional multi-module maven project that publishes to both http://mike-ensor.github.com/multi-module-site-doc-project/index.html and locally to ${env.HOME}/sitedocs/${project.artifactId}.

https://github.com/mike-ensor/multi-module-site-doc-project.

Maven 3 Site Docs

There are two important elements with respect to site docs for Maven. Plugins, which generate ReportSet elements and Site Doc/Descriptors.

Plugins

Plugins are an important part of the documentation process and will typically run throughout the maven build lifecycle generating artifacts for the plugin phase that generates the ReportSet. A good example would be the maven-checkstyle-plugin. The plugin is typically bound to the process-resources or packakge phases so that developers know they have a checkstyle violation early in the build lifecycle. The plugin generates a checkstyle.xml document that is then used in the site phase by the maven-project-info-reports-plugin to generate a checkstyle report page.

Maven Site Plugin

The maven-site-plugin is the main plugin that is used to crate all site documentation. Each plugin that produces and/or consumes a ReportSet will need to be configured inside of the <reportPlugins> section. NOTE: This is where the Wagon plugin is set as a dependency. The Site plugin will use the <distributionManagement><site> line to determine where the site docs are to be published (SCM, WebDAV, SFTP, etc)

<plugin>
 <artifactId>maven-site-plugin</artifactId>
 <version>${plugin.maven-site.version}</version>
 <dependencies>
  <dependency><!-- add support for ssh/scp -->
   <groupId>org.apache.maven.wagon</groupId>
   <artifactId>wagon-ssh</artifactId>
   <version>${plugin.wagon-ssh.version}</version>
  </dependency>
 </dependencies>
 <configuration>
  <attach>true</attach>
  <reportPlugins>
   <plugin>
   ...
   ... Plugins go here
   ....
   </plugin>
  </reportPlugins>
  <attach>true</attach>
 </configuration>
</plugin>

Maven Project Info Report Plugin

The maven-project-info-report-plugin is the main plugin used by the maven-site-plugin that Maven 3 utilizes to collate and produce the common site reporting docs for the artifact

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-project-info-reports-plugin</artifactId>
    <configuration>
        <dependencyDetailsEnabled>false</dependencyDetailsEnabled>
        <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
    </configuration>
    <!-- simpler configuration without
        reportSets available for usual cases -->
    <!-- distribution-management,
        index, dependencies, help,
        issue-tracking, plugins,
        cim, license, dependency-management,
        mailing-list, project-team,
        dependency-convergence,
        scm, plugin-management,
        modules, summary -->
    <reports>
        <report>summary</report>
        <report>dependencies</report>
        <report>project-team</report>
        <report>issue-tracking</report>
        <report>scm</report>
        <report>cim</report>
        <report>modules</report>
        <report>plugins</report>
    </reports>
</plugin>

Take a quick look at the types of reports that are automatically generated using the maven-project-info-reports-plugin. Many of these automatically generated reports provide additional information on top of the redundant information provided by the below listed plugins.

List of automatically provided reports

distribution-management
index
dependencies
help
issue-tracking
plugins
cim
license
dependency-management
mailing-list
project-team
dependency-convergence
scm
plugin-management
modules
summary

Common Plugins

Checkstyle - maven-checkstyle-plugin – Provides static analysis on source files – NOTE: Checkstyle needs to be run during build AND report phases

Example report plugin

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-checkstyle-plugin</artifactId>
 <configuration>
  <skip>${checkstyle.skip}</skip>
  <configLocation>${checkstyle.configUrl}</configLocation>
  <failsOnError>false</failsOnError>
  <enableRulesSummary>true</enableRulesSummary>
  <includeTestSourceDirectory>true</includeTestSourceDirectory>
 </configuration>
</plugin>

NOTE: A best practice is to provide a flag to enable/disable the plugin on a project-by-project basis. Also, notice the failsOnError, this will ensure that the plugin will produce an artifact on error and present that to the user.

FindBugs - maven-findbugs-plugin – Provides static analysis on source files – Plugin does not need to be run outside of report phase

Example report plugin

<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>findbugs-maven-plugin</artifactId>
 <configuration>
  <skip>${findbugs.skip}</skip>
  <xmlOutput>true</xmlOutput>
 </configuration>
</plugin>

Cobertura - maven-cobertura-plugin – Provides code coverage during Unit Test (test) phase. NOTE: Must be run during build AND during reporting phases

Example report plugin

<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>cobertura-maven-plugin</artifactId>
 <configuration>
  <skip>${cobertura.skip}</skip>
 </configuration>
</plugin>

PMD and CPD - maven-pmd-plugin – Provides static analysis (PMD) and code duplication/cut-copy-paste detection for artifact. Read more about the PMD plugin here.

Example report plugin

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-pmd-plugin</artifactId>
 <configuration>
  <skip>${pmd.skip}</skip>
  <targetJdk>${maven.compiler.source}</targetJdk>
  <sourceEncoding>${project.build.sourceEncoding}</sourceEncoding>
  <minimumTokens>100</minimumTokens>
  <rulesets>
   <ruleset>${maven-pmd-plugin.configLocation}</ruleset>
  </rulesets>
  <failOnViolation>false</failOnViolation>
 </configuration>
</plugin>

Surefire and Failsafe - maven-surefire-report-plugin – Creates reports based on tests cases for unit tests and optionally integration tests. NOTE, This is a different plugin than the maven-surefire-plugin, this is the report plugin that operates on the ResultSet for both Surefire and Failsafe. Both Surefire and Failsafe must be run during the build for this plugin to generate reports.

Example report plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-report-plugin</artifactId>
    <reportSets>
        <reportSet>
            <id>integration-tests</id>
            <reports>
                <report>report-only</report>
                <report>failsafe-report-only</report>
            </reports>
        </reportSet>
    </reportSets>
</plugin>

JavaDoc - maven-javadoc-plugin – Bundles JavaDocs into the reporting structure. NOTE, this does not need to be run during the standard build, only during the reporting phase

Example report plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-javadoc-plugin</artifactId>
    <configuration>
        <skip>${javadocs.skip}</skip>
        <failOnError>false</failOnError>
    </configuration>
</plugin>

JXR - maven-jx-plugin – Builds cross-link information for classes. Checkstyle, PMD, FindBugs, JavaDocs and various other plugins can utilize the JXR configuration. NOTE, this plugin must be run at an early place in the Maven lifecycle (suggest process-resources) in addition to the sitedoc plugin section.

Example report plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jxr-plugin</artifactId>
    <configuration>
        <aggregate>true</aggregate>
    </configuration>
</plugin>

Useful Plugins

Other handy plugins to use for site documentation include:

Taglist – Searches source code for specific words like @TODO and FIXME, then collates them into a report per project — http://mojo.codehaus.org/taglist-maven-plugin/
Versions – Uses maven central and/or local mirror to look up each dependency, project properities and plugin versions, then produces a report with what can be updated both minor and major updates — http://mojo.codehaus.org/versions-maven-plugin/
Changelog - Uses SCM listed in the distributionManagement section to pull in a simple history for each file; listing most changed files, top contributors and links up changes with web-based SCM tool like github or web-svn — http://maven.apache.org/plugins/maven-changelog-plugin
Changes - Integrates with Jira, Trac or Bugzilla (or any changelist.xml produced by an Issue Management system) and displays what bugs have been updated in between the current and previous release — http://maven.apache.org/plugins/maven-changes-plugin/

End of Part 1

The next post will cover the developer portion of the documentation; more specifically covering XDoc and APT. The site documentation is robust and the next post will show how to program some of the XDoc portions of the docs by leveraging Velocity templates.

The last part of this blog post will cover multi-module projects. In addition, that post will show how to deploy site-docs to a Github Pages so you can have site docs at http://<username>.github.com/<your-project>.

Colorized Maven Output on Mac OSX

2012-02-21T20:21:00.000-08:00

Overview

Unlike many other open source build tools (for example, git) Maven does not provide any type of colorized output. Maven's excessive amount of output can be hard to read and you tend to get lost in the sea of text. My current client I've been given a Mac to use and quickly found out that the existing Linux scripts did not work with iTerm. My solution, take the existing Linux scripts and modify for the Mac OSX.

Approach

Since Maven does not have colorized output, the only option is to use "sed" (stream editor) that takes the output of maven, matches strings with regex and then adds in color codes. The terminal does not escape ansi colors during the stream editing phase. The solution is to use the "tput" command along with setab and setaf for the background and foreground respectively.

Example of ANSI colors

echo -e "\033[1;34RED TEXT\[0m"

Example of TPUT colors

echo -e "${`tput setaf 1`}RED TEXT${`tput sgr0`}"

The Script

#!/bin/sh

# Written by Mike Ensor (mike@ensor.cc)
# Copywrite 2012
# Use as needed, modify, have fun!
# This is intended to be used for Maven3 + Mac OSX
#
# To use:
# in your ".bashrc" or ".bash_profile" add the following line:
# source ~//colorize-maven.sh
#

#Predefine all variables
c_black=
c_cyan=
c_magenta=
c_red=
c_white=
c_green=
c_yellow=
c_blue=
c_bg_black=
c_bg_cyan=
c_bg_magenta=
c_bg_red=
c_bg_white=
c_bg_green=
c_bg_yellow=
c_bg_blue=
c_end=
c_bold=

xterm_color() {
 # 0 Black
 # 1 Red
 # 2 Green
 # 3 Yellow
 # 4 Blue
 # 5 Magenta
 # 6 Cyan
 # 7 White

    # Yes, this could be a map
    c_bold=`tput setaf 0`
    c_bg_bold=`tput setab 0`
    c_black=`tput setab 0`
    c_bg_black=`tput setab 0`
    c_cyan=`tput setaf 6`
    c_bg_cyan=`tput setab 6`
    c_magenta=`tput setaf 5`
    c_bg_magenta=`tput setab 5`
    c_red=`tput setaf 1`
    c_bg_red=`tput setab 1`
    c_white=`tput setaf 7`
    c_bg_white=`tput setab 7`
    c_green=`tput setaf 2`
    c_bg_green=`tput setab 2`
    c_yellow=`tput setaf 3`
    c_bg_yellow=`tput setab 3`
    c_blue=`tput setaf 4`
    c_bg_blue=`tput setab 4`
    c_end=`tput sgr0`
}

#This is not used (yet)
ansi_color() {
    c_bold=   '[1m'
    c_blue=   '[1;34m'
    c_black=  '[1;30m'
    c_green=  '[1;32m'
    c_magenta='[1;35m'
    c_red=    '[1;31m'
    c_cyan=   '[1;36m'
    c_end=    '[0m'
}

color_maven() {

    # pick color type
    if [ $TERM = 'xterm-color' ]
    then
        xterm_color
#    elif [ $TERM = 'ansi' ]
#    then
#     ansi_color
 else
     echo "${c_red}WARNING:::Terminal '${TERM}' is not supported at this time. Colorized output will not happen for Maven${c_end}"
 fi

 error=${c_bold}${c_red}
 info=${c_white}
 warn=${c_yellow}
 success=${c_green}
 projectname=${c_bold}${c_cyan}
 skipped=${c_white}
 downloading=${c_magenta}

 $MAVEN_HOME/bin/mvn $* | sed -e "s/(\[INFO\]) Building( .*)/ ${info}\1${projectname}\2 ${c_end}/g" \
  -e "s/(Time elapsed: )([0-9]+[.]*[0-9]*.sec)/${c_cyan}\1${c_white}\2${c_end}/g" \
  -e "s/(Downloading: .*)/${downloading}\1${c_end}/g" \
  -e "s/BUILD FAILURE/${error}BUILD FAILURE${c_end}/g" \
  -e "s/WARNING: ([a-zA-Z0-9.-/\\ :]+)/${warn}WARNING: \1${c_end}/g" \
  -e "s/SEVERE: (.+)/${c_white}${c_bg_red}SEVERE: \1${c_end}/g" \
  -e "s/Caused by: (.+)/${c_white}${c_bg_green}Caused by: \1${c_end}/g" \
  -e "s/Running (.+)/${c_green}Running \1${c_end}/g" \
  -e "s/FAILURE (\[[0-9]+.[:0-9]+s\])/${error}FAILURE \1${c_end}/g" \
  -e "s/SUCCESS (\[[0-9]+.[:0-9]+s\])/${success}SUCCESS \1${c_end}/g" \
  -e "s/(\[INFO.*)/${info}\1${c_end}/g" \
  -e "s/INFO: (.+)/${c_white}INFO: \1${c_end}/g" \
     -e "s/(\[WARN.*)/${warn}\1${c_end}/g" \
     -e "s/(\[ERROR.*)/${error}\1${c_end}/g" \
     -e "s/(<<< FAILURE!)/${error}\1${c_end}/g" \
     -e "s/Tests run: ([0-9]*), Failures: ([0-9]*), Errors: ([0-9]*), Skipped: ([0-9]*)/${c_green}Tests run: \1 ${c_end}, Failures: ${warn}\2 ${c_end}, Errors: ${error}\3 ${c_end}, Skipped:  ${skipped}\4 ${c_end}/g"
}

alias mvn=color_maven

NOTE: This script is intended to be used with Green text on Black background. Adjust the colors as you want to come up with something more personal

Download script from Github

Click here to view Gist on Github

How to install/use

In your ".bashrc" or ".bash_profile" add the following line:

source ~/<path to script>/colorize-maven.sh

Mocking with JodaTime's DateTime and Google Guava's Supplier

2012-01-17T21:29:00.000-08:00

Introduction

If you're a seasoned unit tester, you've learned to take note when you see any code working with time, concurrency, random, persistence and disc I/O.

The reason for this is that tests can be very brittle and sometimes down-right impossible to test properly. This post will show how to abstract out "time" by injecting a replacement for it in the consumer. This post will be using Spring 3 as the Dependency Injection container, though Guice, other DI containers or constructor/setters on POJOs would work as well. I will also ignore Locales since the focus is on the injection of the DateTime, not DateTime itself.

Existing code

You've been handed a piece of code to unit test (or you are creating one and this is your first stab at it). Our first piece of code, only one class: (This class is a Spring 3.1 controller and the purpose is to return back the current time as a String)

@Controller
@RequestMapping(value = "/time")
@VisibleForTesting
class TimeController {

    @RequestMapping(value = "/current", method = RequestMethod.GET)
    @ResponseBody
    public String showCurrentTime() {
        // BAD!!! Can't test
        DateTime dateTime = new DateTime();
        return DateTimeFormat.forPattern("hh:mm").print(dateTime);
    }
}

Take note that the class does a "new DateTime()" in the class. Here is the corresponding test class:

What happens when we run the test? How about assuming we have a very slow machine. You could (and most likely will) end up with your comparison DateTime to be different than the returned DateTime. This is a problem!

First thing to do is to remove the dependency, but how are we going to do this? If we make the DateTime a field on the class, we will still have the same problem. Introduce Google Guava's Supplier interface.

Google Guava Supplier

The Supplier interface only has one method, "get()" which will return an instance of whatever the supplier is setup for. An example, the supplier will return a user's first name if they have logged in, and a default one if they have not:

public class FirstNameSupplier implements Supplier<String> {

    private String value;
    private static final String DEFAULT_NAME = "GUEST";

    public FirstNameSupplier() {
        // Just believe that this goes and gets a User from somewhere
        String firstName = UserUtilities.getUser().getFirstName();
        // more Guava
        if(isNullOrEmpty(firstName)) {
            value = DEFAULT_NAME;
        } else {
            value = firstName;
        }
    }

    @Override
    public String get() {
        return value;
    }
}

To your implementing method, you don't care what the first name is, only that you get one.

Refactoring out DateTime

Let's move on. For a much more real example of using a Supplier (and the point of this post) let's implement a DateTime supplier to give us back the current DateTime. While we're at it, let's also create an interface so that we can create mock implementations for testing:

public interface DateTimeSupplier extends Supplier<DateTime> {
    DateTime get();
}

and an implementation:

public class DateTimeUTCSupplier implements DateTimeSupplier {
    @Override
    public DateTime get() {
        return new DateTime(DateTimeZone.UTC);
    }
}

Now we can take the DateTimeUTCSupplier and inject that into our code that needs the current DateTime as the DateTimeSupplier interface:

@Controller
@RequestMapping(value = "/time")
@VisibleForTesting
class TimeController {

    @Autowired
    @VisibleForTesting
    // Injected DateTimeSupplier
    DateTimeSupplier dateTime;

    @RequestMapping(value = "/current", method = RequestMethod.GET)
    @ResponseBody
    public String showCurrentTime() {
        return DateTimeFormat.forPattern("hh:mm").print(dateTime.get());
    }
}

In order to test this, we'll need to create a MockDateTimeSupplier and have a controller to pass in the specific instance we want to return:

public class MockDateTimeSupplier implements DateTimeSupplier {

    private final DateTime mockedDateTime;

    public MockDateTimeSupplier(DateTime mockedDateTime) {
        this.mockedDateTime = mockedDateTime;
    }

    @Override
    public DateTime get() {
        return mockedDateTime;
    }
}

Notice that the object being saved is passed in via the constructor. This will not get you the current date/time back, but will return back the specific instance you want

and finally our test that exercises (slightly) the TimeController we implemented above:

public class TimeControllerTest {

    private final int HOUR_OF_DAY = 12;
    private final int MINUTE_OF_DAY = 30;

    @Test
    public void testShowCurrentTime() throws Exception {
        TimeController controller = new TimeController();
        // Create the mock DateTimeSupplier with our known DateTime
        controller.dateTime = new MockDateTimeSupplier(new DateTime(2012, 1, 1, HOUR_OF_DAY, MINUTE_OF_DAY, 0, 0));

        // Call our method
        String dateTimeString = controller.showCurrentTime();

        // Using hamcrest for easier to read assertions and condition matchers
        assertThat(dateTimeString, is(String.format("%d:%d", HOUR_OF_DAY, MINUTE_OF_DAY)));
    }

}

Conclusion

This post has shown how to use Google Guava's Supplier interface to abstract out a DateTime object so you can better design your implementations with unit testing in mind! Suppliers are a great way to solve the "just give me something", mind you it's a known type of something.

Good luck!

How to setup several GitHub accounts on multiple machines with separate SSH keys

2011-12-15T14:10:00.000-08:00

Problem

So you've got a personal GitHub account, one for one client, one for another and you want to keep them all separate. How do you setup your box to clone private repositories based on which account you're using? Here's the simple guide on how to do this

Assumptions

I have set this up on Linux (Ubuntu 11.10) and Windows, but only via the Git Bash and/or Cygwin. This post will not cover how to setup and where to store SSH keys for Windows or Mac, though Mac should be fairly similar to Linux.

Setup first SSH key

I'm not going to go into depth on how to setup SSH keys, you can read more about that here. However, I will show the steps I used to get there. For first-account: (please replace "first-account" with your GitHub account)

Change to your home .ssh folder
```
cd ~/.ssh
```

Setup your RSA key

ssh-keygen -t rsa -f ~/.ssh/first-account-rsa -C "first-account@example.com"

After following the directions, you should get something that looks like:

The key fingerprint is:
00:ff:00:cc:13:00:bb:ll:aa:hh

The key's randomart image is:
+--[ RSA 2048]----+
|     .+   +      |
|       = o P .   |
|        = * *    |
|       o = +     |
|      o S .      |
|     o o =       |
|      o . E      |
|                 |
|                 |
+-----------------+

Open your GitHub.com account setup page, click on the "SSH Public Keys" tab.
Create a new SSH key and copy the contents of ~/.ssh/first-account-rsa.pub and place into the new text field. Name it something unique so you can remember which machine you're using for your account.

If you plan on each of your accounts needing access, open each of those accounts and add your SSH key to each account. This isn't very common, so only do this if you need to.
To test, type in:
```
ssh -t git@github.com
```
As long as you do not see "access denied", you're good. You might see "failed on channel 0", this is ok.

Adding a second account

Now you should have your first account working, let's setup and add the second account.

Repeat process 1-5 and stop (using the next account)
Create a file named "config" in your ~/.ssh folder
```
echo '' > ~/.ssh/config
```
–or–
```
touch ~/.ssh/config
```

Use your favorite editor to add the configuration information for the SSH client to determine which RSA key to use:

# first-account GitHub account
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/first-account-rsa # Note, this is the private key matching the .pub

# second-account
Host github-second-account  # NOTE: The host is NOT github.com, more on this further down
HostName github.com
User git
IdentityFile ~/.ssh/second-account-rsa # Private key for the second account

# repeat for the third and so on

Save the file and your configuration is done

Let's look at the config file in a bit more detail. The first account is the base github.com host name, and you typically see a git URL as git@github.com:user/project.git. With your first account, you will leave this the domain the same^**.

NOTE: This doesn't stop you from modifying your ~/.ssh/config file and instead of using "github.com" as your default host, setting the value to "github-first-account".

Now, for your second account, you will need to modify the URL you are using to clone the repository. This is done by changing the domain from "github.com" to "github-second-account". For example: git@github-second-account:user/project.git

Summary

You should now have the setup to clone and modify code for multiple accounts on the same machine. Repeat this process for each machine you would like to make available to GitHub.

Spring 3.1 + Cloud Foundry + Local Development

2011-12-03T21:36:00.001-08:00

This post will help you build a Spring 3.1 web application using MongoDB on Cloud Foundry. In addition to pushing to Cloud Foundry, you will also be able to develop in your local environment with a MongoDB instance.

Goals
The goals for this blog posting will be to build the application locally, then publish to your local Cloud Foundry instance. We will utilize the Cloud Foundry runtime, and the new Spring Profiles

Setup

Create an account with Cloud Foundry (https://www.cloudfoundry.com/micro)
Follow instructions to setup your own Micro Cloud

I use VMWare's player
Verify "vmc info" that the micro cloud console matches

Download MongoDB (at least version 2.0)
Install and be familiar with Maven 3 (http://maven.apache.org)
Familiarize yourself with Spring 3.1, Spring Data and Spring MongoDB
Clone or download the source (https://github.com/mike-ensor/first-cloud-app)
Run the app locally with:
```
mvn clean package cargo:run -DskipTests
```
Go to http://localhost:8080/home

Profiles
New in Spring 3.1 are the environment profiles which allow a developer to activate groups of beans based on an environment parameter. There are several "gotchas" that I've discovered, one being an undocumented ordering for beans using profiles.

Take a look at data-services.xml. Notice how the MongoTemplate is defined before the . This is against my intuition because the MongoTemplate takes a reference to the MongoFactory object, which is defined below the MongoTemplate definition.

The second "gotcha" came from when and where to set the parameter to enable Spring's profiles. The documentation and blogs do not explicitly mention that the developer must specify the profile that is active. It was implied by the documentation that "default" was active by default, but this is not true. In order for the default profile to be active, I added it as a system property in my cargo settings. (as long as it is a system environment property, feel free to set it anywhere or any way you'd like). Take a look at the pom.xml file around line 40 for the local Maven property and then around line 253 for the environment variable to be set.

Local development vs. Cloud Development
One of the main goals I had for interacting with Cloud Foundry is that I wanted a local development environment to speed up and ease development and reduce complexity with debugging. Notice that in data-services.xml there is a "cloud" profile and a "default". The point of the "default" profile is to have beans that are constructed when on a local environment. You can see that there are two definitions of MongoFactory, one using Spring Data MongoDB's XML namespace and one using CloudFoundry Runtime's namespace. I am not going to cover why these work the way they do, so if you'd like information, refer to http://blog.springsource.org/2011/04/12/cloud-foundry-for-spring-developers/ and http://blog.springsource.org/2011/11/09/using-cloud-foundry-services-with-spring-applications-part-3-the-cloud-namespace/

Pushing to Cloud Foundry
Now that you have a local running instance of the webapp, you will notice that the artifact is called "first-cloud-app.war", which you can find in the "/target" folder. This is a problem when pushing to the Cloud Foundry instance since the name cannot contain any non-alpha characters. Cloud Foundry's vmc tool is built from the VCAP open source project that is responsible for the open source PaaS services. Another PaaS service includes App Fog, which allows you to basically use the same commands, but replace "vmc" for "af". Both services fall victim to the naming problem.

In order to get around the naming problem, I have created a Maven profile cloud that builds the WAR artifact as "mikeensor.war". Please change this to match your application's name since you won't have the user/password to publish (or the DNS) to publish to my micro instance. The name will need to fit into the URL pattern http://<applicationname>..cloudfoundry.me.

To publish to your local cloud foundry micro instance, goto the root folder and type the following. (this is assuming your micro instance is running and there are no "red" errors.

mvn clean package -Pcloud
vmc push <application name> -path target/

(if you have already pushed before, you will need to type:

vmc update <application name> -path target/

Note: It is possible to use the Maven Plugin for Cloud Foundry, however, I have still not been able to get it to work without changing the name of the artifact.

Enabling and connecting to services
You must create a service(s) so that your application can bind to the data source. The VCAP (vmc) application handles how the configuration works when loading your application into Cloud Foundry. It does this via an environment variable that is consumed in the namespaced configuration element.

In my example, I created a MongoDB service by typing:

vmc create-service mongodb --name <what you want to call your instance>

I named mine "second" (because I had created a first) and you will see that in data-services.xml the cloud XML configuration refers to the name of the service.

Note that if you have multiple MongoDB instances, you will need to do some Spring configuration (@Qualifier) when you want to use different instances. This is not covered by this blog posting.

Now you will need to "bind" the service to your application. This is done by typing:

vmc bind-service <name above> <application name>

Testing it out
You should be able to goto http://..cloudfoundry.me/home (example: http://mikeensor.mikeensor.cloudfoundry.me/home)

Congratulations! You should have not only successfully deployed to Cloud Foundry (micro instance), bound to a MongoDB instance, but should be able to run in your local environment too! As I get time, I will try to add in more detailed features such as multiple types of storage and post other "gotchas" as I find them.

Maven + JavaScript Unit Test: Running in a Continuous Integration Environment

2011-08-02T16:02:00.000-07:00

JavaScript Unit Testing Continued

So you're still interested in unit testing JavaScript (good). This post is an extension of my much more indepth first posting on how to unit test JavaScript using JS Test Driver. Please check it out here.

Recap Last Posting

In the last posting we successfully unit tested JavaScript using Maven and JsTest Driver. This allowes us to test JavaScript when on an environment that has a modern browser installed and can be run.

Problem with typical CI environments

So what happens when the test are passing on your local box, but you go to check in your code and the Continuous Integration (CI) server pukes on the new tests becasue there is no "screen" to run chrome or firefox? As of this posting, none of the top-tier browsers have a "headless" or an in-memory only browser window. There are alternatives to running JavaScript in a browser, such as rhino.js, env.js or HtmlUnit, however, these are just ports of browsers and the JavaScript and DOM representation are not 100% accurate which can lead to problems with your code when rendered in a client's browser.

Approach

What we need to do is to run JSTestDriver's browser in a Virtual X Framebuffer (Xvfb) which is possible on nearly all Linux based systems. The example below uses a Solaris version of Linux, however, Debian and RedHat linux distrubutions come with the simplified bash script to easily run an appliation in a virtual framebuffer. This solution was derived from one posted solution on the JS Test Driver wiki (http://code.google.com/p/js-test-driver/wiki/ContinuousBuild. The given example is also a full working example that is in use at my current client.

Here is the quick list of what we will accomplish. Note, several of these steps are discussed in depth in the previous post and are not covered in depth here.

Create a profile to run Js Unit-Tests
Copy JsTestDriver library to a known location for Maven to use
Copy JavaScript main and test files to known locations
Use ANT to start JsTestDriver and pipe the screen into xvfb

Here is a sample profile to use. You will need to adjust the properties at the top of the profile to match your system.

<profile>
    <!-- Runs JS Unit Test inside of Xvfb -->
    <id>ci-jstests</id>
    <properties>
        <firefox.location>/opt/swf/bin/firefox</firefox.location>
        <js-test-driver.version>1.3.2</js-test-driver.version>
        <xvfb.location>/opt/X11R6/xvfb-run</xvfb.location>
    </properties>
    <build>
        <plugins>
            <!-- Copy JS Test Driver JAR to target folder -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.1</version>
                <executions>
                    <execution>
                        <id>copy</id>
                        <phase>generate-resources</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>com.google.jstestdriver</groupId>
                                    <artifactId>jstestdriver</artifactId>
                                    <version>${js-test-driver.version}</version>
                                    <type>jar</type>
                                    <overWrite>true</overWrite>
                                    <destFileName>jsTestDriver.jar</destFileName>
                                </artifactItem>
                            </artifactItems>
                            <outputDirectory>${project.build.directory}/jstestdriver</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>true</overWriteSnapshots>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <!-- Copy JavaScript files and Tests files  -->
            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <id>copy-main-files</id>
                        <phase>generate-test-resources</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/test-classes/main-js</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>src/main/webapp/scripts</directory>
                                    <filtering>false</filtering>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                    <execution>
                        <id>copy-test-files</id>
                        <phase>generate-test-resources</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/test-classes/test-js</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>src/test/webapp/scripts</directory>
                                    <filtering>false</filtering>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <target>
                                <mkdir dir="${project.build.directory}/js-test-results" />
                                <exec executable="${xvfb.location}">
                                    <arg line="java" />
                                    <arg line="-jar ${project.build.directory}/jstestdriver/jsTestDriver.jar" />
                                    <arg line="--port 4220" />
                                    <arg line="--testOutput ${project.build.directory}/js-test-results" />
                                    <arg line="--config ${project.build.directory}/test-classes/jsTestDriver.conf" />
                                    <arg line="--browser ${firefox.location}" />
                                    <arg line="--tests all" />
                                    <arg line="--verbose" />
                                </exec>
                            </target>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

Possible problems

Although I cannot predict or fix all problems, I can share the one major problem I ran into with Solaris and the script used to fix that. In Solaris (and could happen to other distros) the xvfb-run script was not available and several of the other libraries did not exist. I first had to download the latest X libraries and place them in their appropriate locations on the CI server. Next, I had to re-engineer the xvfb-run script. Here is a copy of my script (NOTE: This is the solution for my server and this may not work for you)

I created a script that contains:

/usr/openwin/bin/Xvfb :1 screen 0 1280x1024x8 pixdepths 8 24 fbdir /tmp/.X11-vbf &

Maven's WAR Overlays: How to manage dependencies

2011-07-12T00:58:00.000-07:00

If you're stumbling on this post looking for a way to manage dependencies with WAR overlays, please make sure you check out part one of this blog posting to get the background about how to apply and use WAR overlays in your POM file. This post picks up from the last post and it's assumed that you understand how to apply a WAR to an existing master project.

Quick recap

Why a quick recap? We're going to go into depth on how dependencies and libraries are applied to the master project. In order to have a WAR overlay, you must include the WAR as a dependency to the master project and must be overlayed during the war-plugin package phase.

Where do the dependency libraries go?

As mentioned in part one of this blog posting, files are placed in their parent's locations if it exists. What this means is that if, in your overlay WAR you have a .jsp file in /WEB-INF/welcome.jsp, the master WAR will then have a file named "welcome.jsp" in the /WEB-INF. Furthermore, if there is already a file named "welcome.jsp", that file will not be overwritten. If we extrapolate this a bit more, the libraries in the /WEB-INF/lib folder from the overlay will be placed inside the /WEB-INF/lib folder of the master project. This is great and all except when different versions of libraries exist causing runtime failures. Good news, there are ways to manage the dependencies in both the overlay WAR and the master project.

How to see your main artifact's libraries?

In order to determine what libraries are in which artifact, there are two simple ways. First, determine what libraries are in your master WAR.

mvn clean package 

ls -al target/YOUR_ARTIFACT_WAR/WEB-INF/lib

NOTE: adding "-DskipTests" when added to the above maven command would skip tests and would make this step a bit faster, though, it is strictly optional

Secondly, add the "workDirectory" parameter to the configuration portion of your war-plugin when applying the overlay. Example: <workDirectory>target/extract_here</workDirectory>

In your master project's POM file:

<plugins>
...
...
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.1.1</version>
        <configuration>
                <workDirectory>target/overlay-war-folder</workDirectory>
                <overlays>
                        <overlay>
                                <groupId>com.yourcompany</groupId>
                                <artifactId>branded-templates</artifactId>
                        </overlay>
                </overlays>
        </configuration>
</plugin>
...
...
</plugins>

This additional property will place each overlay's exploded contents into the specified folder to help debug library conflicts.

How to exclude dependencies/libraries at build time?

It can be a pain to manage the dependencies between each of the overlays, especially if the overlay is not under your development control. One method would be to manage the dependencies within the two POM files by utilizing the "provided" and "optional" dependencies that maven provides. However, if the overlay project can also stand as stand-alone project, this method will not work because the libraries are not considered to be provided or optional when running independently. This is common when trying to test each of the overlays independently. An additional option to control the libraries is to exclude them from the master WAR during the package phase. This is achieved by using the "excludes" parameter in each of the overlay sections. For example, to remove all of the spring libraries, you could exclude "spring*.jar"

<plugins>
...
...
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.1.1</version>
        <configuration>
                <workDirectory>target/overlay-war-folder</workDirectory>
                <overlays>
                        <overlay>
                                <groupId>com.yourcompany</groupId>
                                <artifactId>branded-templates</artifactId>
                                <excludes>
                                        <exclude>WEB-INF/lib/spring*.jar</exclude>
                                </excludes>
                                <excludes>
                                        <exclude>WEB-INF/lib/log4j*.jar</exclude>
                                </excludes>
                        </overlay>
                </overlays>
        </configuration>
</plugin>
...
...
</plugins>

You might start asking yourself, this is going to get complicated really quickly, and you're right, there are other options.

Skinny vs Fat WAR Overlays

Up until now, all of our overlays have been considered "fat" because they contain all of it's dependencies which get overlaid onto the master WAR file. There is an alternative that is more commonly known as the "skinny WAR". A skinny WAR is a combination between most of this post's tactics. NOTE: The method I am showing you is known as a "skinnier WAR" because the dependency libraries do exist in the overlay's WAR file, however, the classes will be externalized and the libraries can be removed during final packaging of the master project.

Achieving the "skinny WAR"

First, in the overlay project, during the package phase, tell maven to build the project's classes as an extra artifact in addition to the WAR file. This set of classes will become a jar file which will be used as a second dependency in the master project's POM.

In the Overlay's POM file (add to the existing war plugin):

<plugins>
...
...
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.1.1</version>
        <configuration>
                <attachClasses>true</attachClasses>
        </configuration>
</plugin>
...
...
</plugins>

After successfully installing/deploying the overlay project, we will add the classes/jar file and the WAR dependency to the master project's POM file.

In the master project's POM file:

<dependency>  
   <groupId>com.yourcompany</groupId>  
   <artifactId>branded-templates</artifactId>  
   <type>war</type>  
</dependency>  


<dependencies>
...
...
<dependency>
        <groupId>com.yourcompany</groupId>
        <artifactId>branded-templates</artifactId>
        <version>1.1</version>
        <type>war</type>
</dependency>
<dependency>
        <groupId>com.yourcompany</groupId>
        <artifactId>branded-templates</artifactId>
        <version>1.1</version>
        <type>jar</type>
        <classifier>classes</classifier>
</dependency>
...
...
</dependencies>

And finally, combine the excludes learned in the last example section above to exclude all .jar files from the overlay WAR.

<plugins>
...
...
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <version>2.1.1</version>
        <configuration>
                <workDirectory>target/overlay-war-folder</workDirectory>
                <overlays>
                        <overlay>
                                <groupId>com.yourcompany</groupId>
                                <artifactId>branded-templates</artifactId>
                                <excludes>
                                        <exclude>WEB-INF/lib/*.jar</exclude>
                                </excludes>
                        </overlay>
                </overlays>
        </configuration>
</plugin>
...
...
</plugins>

NOTE: Dependencies that exist in the Overlay will need to be added as dependencies to the master project because they have been removed from the overlay's /WEB-INF/lib folder. This is the main known risk of using skinny war overlays

Congratulations!

You now have a project with limited library conflicts and can better manager the main project and it's stability.

Thank you for reading part one and this post, part two. Please feel free too leave comments or questions and I will do my best to answer them in a timely manor.

Maven's WAR Overlay: What are WAR Overlays?

2011-06-27T23:02:00.000-07:00

What is a WAR overlay?

def: Overlays are used to share common resources across multiple web applications. The dependencies of a WAR project are collected in WEB-INF/lib, except for WAR artifacts which are overlayed on to the WAR project itself. http://maven.apache.org/plugins/maven-war-plugin/overlays.html

Why and where would you use a WAR overlay?

WAR overlays make up one more tool in your arsenal to combat repetitive code and/or resources. Overlays allow you to combine many similar resources, for example, images, CSS and JavaScript files all into one project. At build time, the overlay will be predictably merged into a master WAR project. A great example of a WAR overlay is a branded website, combined with Sitemesh decorators. Imagine you have a branded company website which consists of multiple web applications. An industry standard approach would be to utilize Sitemesh decorators to decorate the application with brand specific styles and allow each of your applications to focus on the specific purpose of that application. At this time, this blog post will not go into depth on Sitemesh decorators, however, more details are available at http://www.sitemesh.org/.

Leading on with our example, each of the decorators will need a set of CSS, images and possibly JavaScript files to define the application's branded style. If your applications are to be deployed to production to look and act as one cohesive branded site, you will need to make sure that the images, CSS, JavaScript, decorator files and JSP files are all the same in each of your applications in order to keep a consistant brand across your applications. Well, this is duplicated code (currently ranked in my top 5 biggest "code avoid-at-all-cost" list).

How do you solve code duplication using WAR Overlays?

One approach to reducing the amount of duplicated code is to create a separate maven project with the packaging target type of WAR. Then in each of your application's pom.xml file, include the overlay project as a dependency and construct it as an overlay using the maven-war-plugin. This is known as a Fat WAR Overlay. (NOTE: In the next post I will cover other types of WAR overlays to help mitigate transient dependency problems)

More specifics please!

First, let's create a new maven WAR project and name it "branded-templates". (NOTE: This post is not intended to be a comprehensive guide on creating a maven project, only adding to an already working maven project. If you need more assistance, please refer to some of my other post or maven's website for more guidance.)

In our new branded-templates application, we will need to fill this with our resources. Here is my recommended locations to place web-related resources within your project:

/src/main/webapp/styles – used to hold your CSS files, (example: brand.css, reset.css, etc)
/src/main/webapp/scripts/ – used to hold brand-wide JavaScript files, (example: jquery.1.x.x-min.js, jquery-my-company-validation.js, etc
/src/main/webapp/images/ – used to hold brand-wide images, (example: company-logo.png, facebook-icon.png, company-background.jpg, etc)

Naming for now will not be important, but you will have to make decisions here because you have two naming options here that will have development implications. You can name your resources and/or folders the same in both of your applications or you can name them different to get different desired results. How is this going to impact my design and why do you need to think about this? (Answer coming after we add to the containing webapp project, keep following!)

Let's continue. We can now install your branded-templates WAR overlay to your local maven repository:

mvn clean install

If all is good, you should see a "Successful build" along with a 1.0-SNAPSHOT for your WAR's version.

Next, let's crate a new main application called "main-webapp". If you have an existing webapp, feel free to use that and work along with this example. Inside of the main-webapp's pom.xml file, we need to add a plugin to build the two WARs into one WAR and add the overlay as a dependency to the project.

Add to your dependencies section:

<dependency>
   <groupId>com.yourcompany</groupId>
   <artifactId>branded-templates</artifactId>
   <type>war</type>
</dependency>

Note that we are adding this as a WAR type, do not leave the type blank.

Add to your plugins section:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-war-plugin</artifactId>
 <version>2.1.1</version>
 <configuration>
   <overlays>
     <overlay>
       <groupId>com.yourcompany</groupId>
       <artifactId>branded-templates</artifactId>
     </overlay>
   </overlays>
 </configuration>
</plugin>

Next, we've added the war-plugin which will take our application and build a WAR. If you're familiar with maven's artifact types, you'll already know that your project builds a WAR already, so why the plugin? The plugin is where the "magic" of the overlay happens. This is where the overlay is placed into the main-webapp's WAR file.

How many WAR files will I have?

When using a WAR overlay, you will start with two or more WAR files (presumably one or more overlays and a main-webapp), but will ultimately end up with one WAR artifact (your main-webapp) in which to deploy to your environment. Though it is completely possible to run your overlay as a standalone application, I recommended you read my next post (coming soon) on dependency management within overlay projects to avoid dependency problems in your main-webapp project.

So what happens when there is a conflicting resource?

As you can imagine, there can be conflicts between overlay's resource names and the main project's resources. This is what I asked you to think about from the question posed above. If you have the same location and named file in your overlay application as you do in your main application, the overlay version will be ignored for the main application's version. This can work for or against you, so think about how you might use both. How would you want to leverage the customization abilities in the main webapp.

Note, if you plan on utilizing Sitemesh decorators, it's not a bad idea to put your decorators.xml and sitemesh.xml file in your WEB-INF folder and the corresponding JSPs in the WEB-INF folder (suggest placing them in /WEB-INF/decorators folder). Remember to add the sitemesh web.xml information into your containing main-webapp's web.xml!

One more gotcha that might arise out of this would be dealing with the web.xml. The overlay's web.xml file is NOT copied over nor is it merged in any way, so if your overlay has specific web.xml file information you can either manually add it to the web.xml or use a <include file="/WEB-INF/web-overlay.xml"> tag where you've added the contents to a file named web-overlay.xml to the overlay war project.

Tips for rapid development

At this point you should have an overlay and a working master webapp leveraging that WAR overlay. Since you have your resources in the overlay and you are probably working in your main-webapp, and it is probably becoming a pain to keep deploying a new version of your overlay project, you can simply refer to the SNAPSHOT version of your overlay inside your main-webapp pom.xml file and do a local install of your overlay.

Leaving notes

Follow my blog and get the next blog post dealing with dependency management with overlays.

Feel free to leave comments or questions and I'll do my best to get back to you in a reasonable time frame. I would like to thank Dynacron Group (my consulting firm) for allowing me some time to work on this blog post!

Maven + JavaScript Unit Test using JsTestDriver

2011-05-31T23:41:00.000-07:00

What are we doing?
You wake up to the annoying Justin Timberlake ringtone you set for the office only to discover it's the site is on fire" call. A quick glance at the alarm clock and you realize it's 3am. It's this time when you decide that you need to get some test coverage on your Java Script.

What is Unit Testing?
Unit Testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application and in JavaScript, typically it is a function.

Why would you want to unit test JavaScript?
JavaScript has grown from the little scripts that would change button images on rollover to a fully functional, and often application critical, complex applications. There are many interactions with DOM objects, AJAX request/response and user events that make testing your code a must-have. Unit testing will give you the the confidence to add new features and change existing code. In addition, unit test self-document the code so you can be sure that the rad new function you wrote does exactly what you want it to.

What is covered
What the rest of this blog will describe is one approach to integrate maven and JsTestDriver for local development. In a near future blog, this approach will be extended to use on a Linux/Unix based system for a continuous integration system.

Approach
There are many approaches to unit testing JavaScript (QUnit being one of the many), however, most of the unit test frameworks do not easily allow you to integrate your unit test as a part of a maven testing cycle. The one problem with all JavaScript unit testing frameworks is that you need to have a place to run your test. At the time of this blog, there is not a "headless" browser, thus you (or your framework) must open a browser, accept JavaScript loading and running the test cases while providing repeatable testing output. This blog series will discuss a few different methods to run unit test, including a method to run in a virtual framebuffer for continuous integration systems that do not allow you to open a browser in the default display.

What is JsTestDriver?
JsTestDriver is an small application that controls your test cases, your source files and the interaction between the browsers you choose to test against. JsTestDriver allows you to only load the source files you want, along with only the DOM elements you need to unit test your source code, in other words, the browser ONLY loads your JavaScript files and optionally HTML and/or DOM elements needed by your test.

How to integrate with Maven
The approach here is to add JavaScript testing via a profile, so you can create a CI build specifically for your JavaScript unit test.

JsTestDriver Setup

This blog will not discuss in depth how to setup and run JsTestDriver. For more in-depth information, please visit http://code.google.com/p/js-test-driver/ website. This blog is using JsTestDriver 1.3.2.

In order to load your test scripts, you must create a configuration file. For a maven application, place your configuration file in your "/resources" folder.

Here is an example properties file (jsTestDriver.conf)
```
server: http://localhost:4220

load:
 # Header JS Files, all files are loaded top-down
 - main-js/lib/jquery.min.js
 # Body JS files
 - main-js/Person.js
 # Test files
 - test-js/*.js

exclude:
 - main-js/exclude.js
```
The above configuration file assumes your file structure is similar to:
/src/main/webapp/scripts/*.js
/src/test/webapp/resources/jsTestDriver.conf
/src/test/webapp/scripts/*.js

(NOTE: You might notice that the file path is different than the "main-js". Keep reading!)

Maven Dependencies and Profile

Next, let's add the dependencies to the POM file in your project.

Add the following to your "dependencies" section in your pom.xml

<dependency>
 <groupid>com.google.jstestdriver</groupId>
 <artifactid>jstestdriver</artifactId>
 <version>1.2.2</version>
 <scope>test</scope>
 <type>jar</type>
</dependency>

Next, we need to add a profile, which will do quite a few things. First, the JsTestDriver jar file will be copied from the repository to a known folder in the /target folder for use when running the JsTestDriver server. Next, we need to copy the JavaScript files from your project's location to a known folder in the "/test" folder, and also to match the "main-js/*" and "test-js/*" folder structures that we setup in the jsTestDriver.conf file. Lastly, we need to startup the JsTestDriver server so we can run our test.

Here is a profile to start with:

<profile>
    <id>local-jstests</id>
    <build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.1</version>
                <executions>
                    <execution>
                        <id>copy</id>
                        <phase>generate-resources</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>com.google.jstestdriver</groupId>
                                    <artifactId>jstestdriver</artifactId>
                                    <version>${js-test-driver.version}</version>
                                    <type>jar</type>
                                    <overWrite>true</overWrite>
                                    <destFileName>jsTestDriver.jar</destFileName>
                                </artifactItem>
                            </artifactItems>
                            <outputDirectory>${project.build.directory}/jstestdriver</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>true</overWriteSnapshots>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>

            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <id>copy-main-files</id>
                        <phase>generate-test-resources</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${basedir}/target/test-classes/main-js</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>src/main/webapp/scripts</directory>
                                    <filtering>false</filtering>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                    <execution>
                        <id>copy-test-files</id>
                        <phase>generate-test-resources</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${basedir}/target/test-classes/test-js</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>src/test/webapp/scripts</directory>
                                    <filtering>false</filtering>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <configuration>

                            <target>
                                <!-- make output folder -->
                                <mkdir dir="${basedir}/target/js-test-results"/>
                                <!-- start jsTestDriver server -->
                                <exec executable="java">
                                    <arg line="-jar ${basedir}/target/jstestdriver/jsTestDriver.jar"/>
                                    <arg line="--port 4220"/>
                                    <!-- Possibly set this up to point to the same location of your surefire output -->
                                    <arg line="--testOutput ${basedir}/target/js-test-results"/>
                                    <arg line="--config ${basedir}/target/test-classes/jsTestDriver.conf"/>
                                    <arg line="--browser /opt/google/chrome/google-chrome"/>
                                    <arg line="--tests all"/>
                                    <arg line="--verbose"/>
                                    <arg line="--captureConsole"/>
                                    <arg line="--preloadFiles"/>
                                </exec>
                            </target>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>

            </plugin>
        </plugins>
    </build>
</profile>

Sample Unit Test

Now you need to have a sample JavaScript file and a corresponding unit test.

Here is a simple JavaScript file (/src/main/webapp/scripts/Person.js)

NAMESPACE = {};
 
NAMESPACE.Person = function(initObj) {
 this.name = initObj.name;
 this.age = initObj.age;
 this.address = initObj.address;
 this.sex = initObj.sex;
 this.significantOther = initObj.significantOther;
 
 function setSignificantOther(person) {
  this.significantOther = person;
 }
 
 this.setSignificantOther = setSignificantOther;
}

And a sample test case might be: (/src/test/webapp/scripts/PersonTest.js

TestCase("PersonTest", {
 testA:function(){
  expectAsserts(1);

   var peterGriffin = new NAMESPACE.Person({
    name: "Peter Griffin",
    age: 43,
    address: {
     address1: "31 Spooner Street",
     address2: "",
     city: "Quhog",
     state:"RI"
    },
    sex: "Male",
    significantOther: null
   });
   
   assertNotNull("The variable should not be null", peterGriffin);
  };
 }
});

Running your test

In order to run your test, you can now type:

mvn clean test -Plocal-js

Please stay tuned for the expansion to this on how to run in a virtual framebuffer for a CI environment (and of course, updates for found errors)

Unit Testing Named Queues: Spring 3+maven2+Google App Engine

2010-11-17T11:25:00.000-08:00

Intro
Problem, you have a task that you know can take more than 30 seconds to complete, what do you do? What if this task needs to be triggered every day at a specific time? Google provides several mechanisms to to solve just this problem, queues and scheduled task, respectively.

Queues
First, let's explore the "queue" system implemented in GAE. Note, at this time, the "queue" is still experimental which means that the API can change around some, so make sure you have strong unit testing, but don't worry, we'll cover that a bit further down the page. To implement queues, you need to let the web app know that your expecting queues to be used, and to do this, you need to create a file named queue.xml in your WEB-INF folder (example: /src/main/webapp/WEB-INF/queue.xml). The queue.xml file has a specific layout and that will not be covered in depth in this post. Please read up on the queue.xml structure on Google's documentation. We will expand on the knowledge since the documentation is not complete.

Scheduled Task
Scheduled task are task that will trigger by GAE at repeatable times, emulating linux cron, or Quartz application. To implement a cron task, simply create a file called cron.xml and place it in your WEB-INF folder (example:/src/main/webapp/WEB-INF/cron.xml). This post will not go into detail on the implementation of scheduled task so please check Google's documentation to get a firm understanding before reading on. In order to not pack too much into one post, Scheduled Task will be covered in the next post.

Our Problem
The scenario we're going to explain is: You have an birthday tracking software and you want a user's friends to be notified, via email, that their friend is having a birthday. Assume that the code for the services named are tested and fully working.

To solve this we will use a queue and the scheduled task system provided by Google App Engine.

First, let's tackle the queue portion, then we will create the scheduled task. Google's documentation on unit testing queues is very limited and only shows how to retrieve the "default" queue. Since we could have multiple task in our system that need to be run at different rates, we will want to create a new Queue. Let's open (or create) the queue.xml file and insert the following:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
    <queue>
        <name>birthdayemail</name>
        <rate>10/s</rate>
        <bucket-size>10</bucket-size>
    </queue>
</queue-entries>

Next, let's create a simple service (Note: assume that the Autowired resources are working and imports are included at the top of the file, this has been reduced to emphasize the solution)

@Service("birthdayService")
public class BirthdayServiceImpl implements BirthdayService {

    // this is the same name as in your queue.xml file
    private static final String EVENT_REMINDER_TASK_NAME = "birthdayemail";
    
    @Autowired
    private BirthdayDAO birthdayDAO;

    public List handleGuestEmailsForBirthday(Date date) {
        if (date == null) {
            throw new IllegalArgumentException("Date can not be null");
        }
        List<Guest> guestBirthdays = birthdayDAO.getByDate(date);
        for (Guest guest : guestBirthdays) {
            String TASK_URL = "/queue/birthday-emails/" + guest.getId()
            final TaskOptions taskOptions = TaskOptions.Builder.url(TASK_URL);
            // create a unique task name, note, must conform to [a-z] regex
            taskOptions.taskName(guest.getName() + "Task");
            taskOptions.method(TaskOptions.Method.POST);

            Queue queue = QueueFactory.getQueue(EVENT_REMINDER_TASK_NAME);
            // commented out to show the default queue shown in the docs
            //Queue queue = QueueFactory.getDefaultQueue();
            queue.add(taskOptions);
        }
        return events;
    }
}

Let's look at what's here:
First, we have to create a URL that will be run when the task is implemented. This URL will handle the unit of work specified, and should run in under 30 seconds per GAE's restrictions. We will not cover what this servlet does, just assume it uses the ID given in the URL path, pulls up the Guest, then uses that to get the Guest's friends' emails; then uses that to construct an email and pass to an EmailService. (HINT: the EmailService could also implement a queue so each email is separated out to run individually)

Next, notice that we are implementing a "Task Name". The this is optional, but does help when debugging to figure out what task is failing or what is running.

Now we're on to the TaskOptions. TaskOptions are a helper function used to combine URL and model data to pass on to the servlet handling the queue. Just a suggestion, but to follow the REST ideals, setting the RequestMethod to POST or PUT is advisable, depending on what you are trying to do. All Task should be idempotent, meaning that if the task fails or is interrupted, the task can be run again and not harm the data.

Lastly, we have the QueueFactory retrieving the Named Queue (the same name in the queue.xml file). Simply add your taskOptions object to the queue.add and move on.

Unit Testing Named Queues
Next, let's setup a unit test to test our named queue. The Google documentation does not explicitly explain how to properly setup a test case for one, datastore aware context and named queries. Below is an example of a test case written to test our Named Queries. (Note, as before, imports have been removed and the services used have been used before. This is not intended to be the end-all-be-all for unit testing GAE, please refer to previous post for more help, and yes, you might be able to use JMock or Mockito instead of persisting data, but I have not explored the option).

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/testApplicationContext.xml"})
public class QueueBirthdayServiceTest
        extends AbstractJUnit4SpringContextTests {

    @Autowired
    private BirthdayService birthdayService;
    private final LocalServiceTestConfig[] configs = {
        new LocalDatastoreServiceTestConfig(),
    /* NOTE: THE QUEUE XML PATH RELATIVE TO WEB APP ROOT, More info below */
        new LocalTaskQueueTestConfig()
                .setQueueXmlPath("src/test/resources/test-queue.xml")};
    private final LocalServiceTestHelper helper
            = new LocalServiceTestHelper(configs);

    @Before
    public void setUp() {
        helper.setUp();
    }

    @After
    public void tearDown() {
        helper.tearDown();
    }

    @Test
    public void testHandleBirthdayQueueEmails() throws InterruptedException {
        // setup data
        Date birtdayDate = DateUtils.parse("10/16/1900");
        // assume this builds the data for guest/guest's friends
        int guestWithBD = 5; // # of birthdays on date
        int gPerBirthday = 10; // # of emails being sent per birthday
        setupDataForBirthdays(birthdayDate, guestWithBD, gPerBirthday);

        List<Guest> birthdayOnDay = birthdayService.
                handleGuestEmailsForBirthday(birthdayDate);
        assertEquals(guestWithBD, birthdayOnDay.size());
        // pause for a moment to allow queue to fill from previous statment
        Thread.sleep(1000);
        // verify # of birthdays with that day's expire date
        LocalTaskQueue ltq = LocalTaskQueueTestConfig.getLocalTaskQueue();
        final Queue queue = QueueFactory
                .getQueue(BirthdayService.EVENT_REMINDER_TASK_NAME);
        //final Queue queue = QueueFactory.getDefaultQueue();
        QueueStateInfo qsi = ltq.getQueueStateInfo()
                .get(queue.getQueueName());
        assertNotNull(qsi);
        int expectedTaskCount = guestWithBD*gPerBirthday;
        assertEquals(expectedTaskCount, qsi.getTaskInfo().size());
        assertEquals(birthdayOnDay.get(0).getID() + "Task",
                qsi.getTaskInfo().get(0).getTaskName());

    }
}

NOTE: If the syntax or unit testing is very foreign to you, please visit my previous post on Unit Testing on Google App Engine.

Let's explore more
The first half is not too exiting, just setting up data and calling the service that creates the queue. We are creating 5 guest with birthdays on the given date, and each of those 5 guest have 10 friends who we intend to email. This service is already "written" out (above)

What you need to pay attention to is at the top of the file, during the setup of the "helper", we have constructed it with a LocalDatastoreServiceTestConfig and LocalTaskQueueTestConfig objects. The second, the "LocalTaskQueueTestConfig" is the important one to add when using a queue. If you are not using the DefaultQueue, you will need to explicitly state where the queue.xml file is. I suggest that you create a test-queue.xml file and place it in your /src/test/resources folder, so as to not mix production data and testing. NOTE: This file is loaded based relative to the ROOT application folder. The rest of the test should be pretty self explanatory.

In the next post, we will uncover how to wire the service up to a Scheduled Task so you can automate the emails being sent. Stay tuned!

Spring 3 + maven2 + Google App Engine: Part 6 [Integration Testing using HttpUnit]

2010-09-15T22:26:00.000-07:00

In my previous post spring3 + maven2 + Google App Engine: Part 5[Unit Testing] I demonstrated how to create unit test for your Google App Engine application. Unit testing is only one part of the goal of 100% test coverage. The next major topic to cover, and arguably the most difficult to setup is integration tests. Integration test are test that use more than one component to complete the test. In web application testing, this is typically where test that can only be run when there is a running application server.

This post will cover Integration test using HttpUnit and the Failsafe Maven Plugin. Your first question is probably, why use failsafe in addition to the surefire. The quick answer is that the failsafe will continue if your servlets cause an exception allowing you to test all of your files.

Failsafe plugin
We'll get started adding the Failsafe plugin to the POM file. In your pom.xml file, add the following to your section:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.6</version>
    <configuration>
        <includes>
            <include>**/*SysTest.java</include>
        </includes>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
            </goals>
        </execution>
    </executions>
</plugin>

When the goal "integration-test" is run, the this testing plugin will include all files named SysTest.java. This should be changed to your specific package setup. Using "SysTest", "ITCase" or other pre/post fixes and separate packages are other options to segregate your integration test from your unit test.

Application Server
We now have the failsafe plugin, but we have to figure out how to run the application server before the integration test run. This is achieved by running the maven-gae-plugin, yes, the same plugin used to run your webapp. In recent releases of the maven-gae-plugin, developers added the stopPort and stopKey methods which allow failsafe to start and stop the application sever. The plugin can used the phases "pre-integration-test" and the "post-integration-test" to start and stop the GAE server.

Here is sample code to add to your pom.xml file inside of the section (typically right below the failsafe-plugin added above:

<plugin>
    <groupId>net.kindleit</groupId>
    <artifactId>maven-gae-plugin</artifactId>
    <version>0.7.1</version>
    <configuration>
        <scanIntervalSeconds>10</scanIntervalSeconds>
        <stopKey>STOP</stopKey>
        <stopPort>8005</stopPort>
    </configuration>
    <executions>
        <execution>
            <id>start-gae</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>start</goal>
            </goals>
            <configuration>
                <scanIntervalSeconds>0</scanIntervalSeconds>
                <daemon>true</daemon>
            </configuration>
        </execution>
        <execution>
            <id>stop-gae</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>stop</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Notice, we are using the "start" verses using "run" because "start" does not include all of the task to "package" the war allowing the failsafe plugin to stop the GAE server.

Adding HttpUnit
In this posting, we will be adding and using HttpUnit as it's one of the most comprehensive and stable HTML unit testing libraries. HttpUnit not only allows you to query your URLs, but you can unit-test JavaScript, see errors for CSS, JavaScript and malformed HTML. HtmlUnit also allows you to emulate various different browsers including FireFox and Internet Explorer.

The integration test are very similar to the unit test buy using the same testing context file, Spring JUnit helper, Google Local Testing helpers and JUnit.

An integration test using HttpUnit
Here is a very basic example of an integration test:

package com.example.web.controllers;

...imports...

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/testApplicationContext.xml"})
public class HomeControllerSysTest extends AbstractJUnit4SpringContextTests {

    private static final Logger log = Logger.getLogger(
            HomeControllerSysTest.class.getName());
    private final LocalServiceTestHelper helper =
            new LocalServiceTestHelper(new LocalDatastoreServiceTestConfig());

    @Before
    public void setUp() {
        helper.setUp();
    }

    @After
    public void tearDown() {
        helper.tearDown();
    }

    @Test
    public void testHomeController() throws IOException {
        final String url = "http://localhost:8080/movie/test";

        final WebClient webClient = new WebClient();
        final HtmlPage page = webClient.getPage(url);
        assertEquals("The Page Title", page.getTitleText());

        // there are many different methods to query everything on your
        // page. Please refer to the HttpUnit homepage
        HtmlElement header = page.getElementsByTagName("h1").get(0);
        assertNotNull(header);

        String headerValue = header.getNodeValue();
        assertEquals(headerValue, "Hello World!");
    }
}

How to test
Now, it's time to run the test. There are several methods for running integration test without running the unit test, but those tactics will not be covered in this posting. To run the integration test, type the following:

mvn integration-test

That's it! Between this posting and the previous [Part 5] you should be able to test a large majority of your Java Web Application.

Spring 3 + maven2 + Google App Engine: Part 5 [Unit test]

2010-09-15T21:44:00.000-07:00

Unit testing is one of the most important keystones to Agile programming methodologies. Testing with maven and Spring can be a big pain, so here's a quick tutorial on how to set it up. I will not cover proper testing strategies, but I'll give you the ability to setup your own testing.

First, we'll focus on unit testing. Unit testing is easily done using the "surefire plugin". To add the plugin, edit your pom.xml file, add the following inside of the section:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.4</version>
    <configuration>
        <includes>
            <include>**/*Test.java</include>
        </includes>
        <excludes>
            <exclude>**/*SysTest.java</exclude>
        </excludes>
    </configuration>
</plugin>

What this does is to tell Maven that during the phase "test", to run the JUnit test using the include/exclude rules.

Next we need to create a test file to run. Luckily we can use annotations to decorate our test files making it much easier. For this example, I will create a file in the test package called EventServiceTest.java.

package com.example;
...imports...
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/testApplicationContext.xml"})
public class EventServiceTest extends AbstractJUnit4SpringContextTests {

    @Autowired
    private EventService eventService;

    // Google's helper file to put all data storage in memory
    private final LocalServiceTestHelper helper =
            new LocalServiceTestHelper(new LocalDatastoreServiceTestConfig());

    @Before
    public void setUp() {
        helper.setUp();
    }

    @After
    public void tearDown() {
        helper.tearDown();
    }

    @Test
    public void testCreateEvent() {
        Event event = eventService.getByName("pie day");
        assertNotNull("Pie Day should always exist!", event);
    }
}

There are a few things to notice in this example. The LocalServiceTestHelper is a class provided by Google that will allow data to be stored in memory instead of interferring with your local datastore. There are several different types of configurations and customizations that can be passed into the constructor. Visit http://code.google.com/appengine/docs/java/tools/localunittesting/javadoc/ to check out other options.

If you noticed, we are loading a new context in the @ContextConfiguration annotation. You will need to create a new context file (similar to applicationContext.xml) which usually is a copy of the applicationContext.xml file, with changes to use mock services.

I choose to specify the context file by location, which is set in my POM file to be at /src/test/resources (refer to Part 2 to see full POM file). Another option could be to use classpath scanning.

Lastly, notice that the @RunWith(SpringJUnit4ClassRunner.class), this sets up the test file for class loading.

You can now run your unit test by running the following command:

mvn test

Next post will contain information about how to create and run integration test.

Spring 3 + maven2 + Google App Engine: Part 4 [Deployment]

2010-09-10T13:37:00.000-07:00

Now you should have an easy application that is working and you'd like to deploy to Google's servers. Let's take a look at the appengine-web.xml file located in the project's /WEB-INF folder. This file contains all of the settings related to the appengine web application.

Here is an example appengine-web.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <application>my-unique-app-name</application>
    <!--  This variable is defined in your POM file -->
    <version>${gae.application.version}</version>
    <system-properties>
        <property name="java.util.logging.config.file" value="WEB-INF/classes/logging.properties"/>
    </system-properties>
    <sessions-enabled>true</sessions-enabled>
    <!--
    <static-files>
        <include path="/site/**" expiration="1d 12h" />
    </static-files>
    -->
</appengine-web-app>

Notice that the application node is filled out with your unique Google App Engine application name which was created during the creation of the application.

If you take a look at part 2 of this series which goes more in depth on the POM file, you will notice at the bottom of the file there are several properties/variables set. One of these is the application version, which correspond to the different versions you can deploy to Google's servers. Notice that in the POM file, there are several version setup using different profiles. When you deploy your application, you can setup different version numbers for testing, integration builds, staging or other required environments. The deployed version will follow the following URL format:

For a version of "0":
http://0.latest.your-unique-appengine-name.appspot.com/

For a version of "2":
http://2.latest.your-unique-appengine-name.appspot.com/

I recommend using the "0" version for testing, "1" for staging and "2" for production. This allows you to release to test and stage versions of your application, which will still use the same database, without releasing to production.

Now, onto how to deploy. If you have been following this series, you will be using the GAE plugin for Maven, which comes with many handy goals. Using the "gae:deploy" will deploy your application to the Google servers according to the appengine-web.xml file.

NOTE: I recommend creating batch files to deploy your application. Here is a very basic example of a bash script file:

#!/bin/bash
mvn gae:deploy -Dmaven.test.skip=true -Prelease-build

I have set the profile to "release-build" and decided to skip my test when doing my releases. Typically you will need to provide your email and password on the Google App Engine account to deploy.

Lastly, once your have a successful deployment, follow these directions.

Check your application by using the URL format given above
Once you have verified the version has been uploaded, goto http://www.appspot.com, login and goto your application's main page
Look for "Application Versions" and goto those settings
Select the "2" to be your version to be the "default"

Congratulations, you have the necessary skills to create an application using Google App Engine and Spring 3.0!

Later post will give more insights and tricks that I have learned while developing a full application exclusively hosted on Google App Engine.

Spring 3 + maven2 + Google App Engine: Part 3 [Spring Setup]

2010-04-08T23:25:00.000-07:00

This is a continuation of a multi-part series discussing how to get Spring 3 running on Google App Engine using maven. This part discusses in more detail how to get Spring 3 MVC configured using annotations.

If you have ever had to setup a typical J2EE Web Application you remember (or perhaps have suppressed) the massive amounts of XML needed to configure your app. Spring 2.5 took great steps towards reducing the use of XML by introducing annotation based configuration. Annotation based configuration is probably the best achievement towards sustainable RAD. For example, lets take your typical Spring web application. You need a web.xml, dispatcher-servlet.xml, urlrewrite.xml (to have the greatest control of your URLs for SEO purposes), data-config xml files and a build.xml. While you can not completely get rid of all of the XML, using annotations you can reduce the file size and best yet, remove dependencies to your business beans.

To start setting up your application with annotations, create a file called "dispatcher-servlet.xml" and place it in your WEB-INF folder. This file will need to have several namespaces defined.

/src/main/webapp/WEB-INF/dispatcher-servlet.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

</beans>

After saving the skeleton file above, it is time to add the meat of the configuration. First, we will tell Spring to wire Controllers to your POJOs that represent the "C" (controller) of MVC. In order to do this, we will need to place our controllers inside of a package structure so Spring can scan the package looking for the @Controller annotation. For our example, we setup our controller to be under com.example.web.controllers. Where you used to have to wire in a bean that used the class name to guess the controller, now we add the annotation to the class and that's it!

Add this to your dispatcher-servlet.xml file:

<context:component-scan base-package="com.example.web.controllers" />

That's it! Now we have the basics for setting up your first controller. Let's build our first Controller. Create a file called HelloController.java under /src/main/java/com/example/web/controllers

package com.example.web.controllers;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.ui.ModelMap;

@Controller
public class HelloController {

    @RequestMapping(value = "/hello", method = RequestMethod.GET)
    public String helloGet(ModelMap map) {
        // this is your model (in future post, I will show how to use the ModelAttribute and command objects)
        map.put("name", "mike!");
        // for now, this is where your "View" is
        return "hello";
    }
}

For now, we will go with just the basics. Save this file and let's create the "View" so we you can run your application.

Create a file named hello.jsp and place it in /src/main/webapp/WEB-INF/jsp/views folder.

<%@ page contentType="text/html; charset=UTF-8" contentEncoding="UTF-8" isELIgnored="false" %>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt" %>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %>
<-- it is recommended to take the preceding 
four lines and place them in a file to include on 
the top of each page -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
    "http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Hello Controller</title>
    </head>
    <body>
        <h1>Hello ${name}!</h1>
    </body>
</html>

Lastly we must hook up our dispatcher servlet (front-loading servlet) so we can handle web request. Open /src/main/webapp/WEB-INF/web.xml file and add in the following:

<servlet>
    <servlet-name>dispatcher</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
</servlet>

<servlet-mapping>
    <servlet-name>dispatcher</servlet-name>
    <url-pattern>/movie/*</url-pattern>
</servlet-mapping>

Now we're ready to run our app!

mvn clean gae:run

NOTE: If this is the first time you have run Google App Engine application using Maven2, then type the following to "unpack" Google App Engine

mvn gae:unpack

Navigate to http://localhost:8080/movie/hello

Now that you have have checked out your first controller, you can already see things that are static and probably be changed. For example, take a look at your "helloGet" method. The String being returned points directly to the view. Rather than hard-code the entire path, we can setup a ViewResolver so you only need a small part of the view name.

Open /src/main/webapp/WEB-INF/dispatcher-servlet.xml and add in the following:

<bean id="viewResolver"
  class="org.springframework.web.servlet.view.InternalResourceViewResolver"
  p:prefix="/WEB-INF/jsp/views/" 
  p:suffix=".jsp" 
  p:viewClass="org.springframework.web.servlet.view.JstlView"
  />

Now in your HelloController.java file, simply return "hello" (see how the strings concatenated together create the entire path?

You now have the basics of Spring MVC setup and so therefore this concludes this portion of the series on how to get Spring and Maven2 running on Google App Engine

Spring 3 + maven2 + Google App Engine: Part 2 [POM Setup]

2010-04-06T23:52:00.000-07:00

In this post, we will look at the POM file and the dependencies for not only a solid web application using Spring 3, but the dependencies specific to Google App Engine. This is not meant to be a comprehensive overview of Maven and may or may not use best practices for Maven (ie, I'll ignore comments about your maven file, grab a maven book, hopefully off the links to the right!)

Now, let's take a look at the POM file and setup our project. We need to add the Google App Engine and the related dependencies, DataNucleus for enhancing domain objects (though, I am investigating how to use Spring persistence through Annotations to replace this dependency), JUnit and various other utilities needed to construct a working web-app.

Most of the POM file generated from part 1 will be sufficient, but we need to start adding dependencies. We'll start with the Spring 3.

Inside of your <dependencies> section, lets add the following dependencies:

Project Dependencies

Spring 3

 
        <dependency>
            <groupid>org.springframework</groupid>
            <artifactid>spring-webmvc</artifactid>
            <version>3.0.1.RELEASE</version>
        </dependency>

        <dependency>
            <groupid>org.springframework</groupid>
            <artifactid>spring-test</artifactid>
            <version>3.0.1.RELEASE</version>
        </dependency>

Google App Engine 

 
        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>datanucleus-jpa</artifactId>
            <version>1.1.5</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>geronimo-jpa_3.0_spec</artifactId>
            <version>1.1.1</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-1.0-sdk</artifactId>
            <version>1.3.2</version>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-labs</artifactId>
            <version>1.3.2</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-stubs</artifactId>
            <version>1.3.2</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-testing</artifactId>
            <version>1.3.2</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-local-runtime</artifactId>
            <version>1.3.2</version>
            <scope>test</scope>
        </dependency>

Google App Engine Data Store

 
        <dependency>
            <groupId>javax.jdo</groupId>
            <artifactId>jdo2-api</artifactId>
            <version>2.3-eb</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.transaction</groupId>
                    <artifactId>transaction-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>jstl</artifactId>
            <version>1.2</version>
            <type>jar</type>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>javax.transaction</groupId>
            <artifactId>jta</artifactId>
            <version>1.1</version>
        </dependency>

        <dependency>
            <groupId>com.google.appengine.orm</groupId>
            <artifactId>datanucleus-appengine</artifactId>
            <version>1.0.5.final</version>
        </dependency>

        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-core</artifactId>
            <version>1.1.5</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.transaction</groupId>
                    <artifactId>transaction-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Now you have the default minimum dependencies for GAE (still not there quite yet, we need to add the necessary plugins). If you are more familiar with Maven, you might want to add in Ant, Apache Commons Collections and any other libraries you typically use.

Plugins
Once you have the needed libraries it is time to setup the plugins to build and run your project. Add the following to the section:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-war-plugin</artifactId>
    <version>2.1-alpha-2</version>
    <configuration>
        <webAppConfig>
            <contextPath>/movie</contextPath>
            <scanIntervalSeconds>10</scanIntervalSeconds>
        </webAppConfig>
        <webResources>
            <resource>
                <directory>src/main/webapp</directory>
                <filtering>true</filtering>
                <includes>
                    <include>**/appengine-web.xml</include>
                </includes>
            </resource>
        </webResources>
    </configuration>
</plugin>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.6</source>
        <target>1.6</target>
    </configuration>
</plugin>

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>cobertura-maven-plugin</artifactId>
    <version>${cobertura-version}</version>
    <executions>
        <execution>
            <goals>
                <goal>clean</goal>
            </goals>
        </execution>
    </executions>
</plugin>

<plugin>
    <groupId>org.datanucleus</groupId>
    <artifactId>maven-datanucleus-plugin</artifactId>
    <version>1.1.4</version>
    <configuration>
        <!-- Make sure this path contains your persistent classes! -->
        <mappingIncludes>**/model/*.class</mappingIncludes>
        <!-- */This is where your domain objects will be located -->
        <verbose>true</verbose>
        <enhancerName>ASM</enhancerName>
        <api>JDO</api>
    </configuration>
    <executions>
        <execution>
            <phase>compile</phase>
            <goals>
                <goal>enhance</goal>
            </goals>
        </execution>
    </executions>
    <dependencies>
        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-core</artifactId>
            <version>1.1.5</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.transaction</groupId>
                    <artifactId>transaction-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-rdbms</artifactId>
            <version>1.1.5</version>
        </dependency>
        <dependency>
            <groupId>org.datanucleus</groupId>
            <artifactId>datanucleus-enhancer</artifactId>
            <version>1.1.4</version>
        </dependency>
    </dependencies>
</plugin>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin-version}</version>
    <configuration>
        <includes>
            <include>**/*Spec.java</include>
            <include>**/*Test.java</include>
        </includes>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jxr-plugin</artifactId>
</plugin>

<plugin>
    <groupId>net.kindleit</groupId>
    <artifactId>maven-gae-plugin</artifactId>
    <version>0.5.2</version>
    <configuration>
        <unpackVersion>1.3.2</unpackVersion>
    </configuration>
</plugin>

<plugin>
    <artifactId>maven-release-plugin</artifactId>
    <configuration>
        <goals>gae:deploy</goals>
    </configuration>
</plugin>

<plugin>
    <artifactId>maven-antrun-plugin</artifactId>
    <executions>
        <execution>
            <phase>compile</phase>
            <configuration>
                <tasks>

                    <!--
                        This creates the META-INF folder inside of classes, then
                        copies over the jdoconfig.xml required for GAE
                    -->
                    <mkdir dir="${project.build.directory}/classes/META-INF"/>
                    <copy file="${project.build.directory}/classes/jdoconfig.xml"
                          todir="${project.build.directory}/classes/META-INF" />
                    <echo message="Copy:${project.build.directory}/classes/jdoconfig.xml to " />
                    <echo message="${project.build.directory}/classes/META-INF"/>
                </tasks>
            </configuration>
            <goals>
                <goal>run</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Click here to get the full pom.xml file

Spring 3 + maven2 + Google App Engine: Part 1 [Initial Project Setup]

2010-04-01T08:00:00.000-07:00

Recently a co-worker started down the path of Spring 3, Spring MVC and Google App Engine. After hearing him talk about it, I decided it was time for me to figure out what the hype was and figure out how to program something using the "cloud".

This series of post will discuss one approach to getting my favorite Java web-app stack of Spring 3, Spring MVC (both with annotations), maven2 and all developed on Google App Engine. Since unit testing is a corner stone of Agile development, JUnit test will also be included to show how to test your code prior to deployment to GAE servers.

This series assumes that you have and know how to use the following: Google login (Gmail or iGoogle for example), Java 1.6, some IDE (Eclipse, NetBeans, InteliJ, Notepad++, etc) though you do not need to install the suggested "Plugins", maven2, Ant 1.7+, web browser, command line

We will be building a simple "Movie Listing" web application where you can enter a movie, add a comment (single user, for now), add "Actors" to that movie. The model and idea is very simple and easy to define so we can focus on the technology.

Step 1: Sign up for a Google App Engine

First off, you will need to sign up for a Google App Engine account before you can do anything else. Visit http://code.google.com/appengine/ and click on "Sign Up" under "Getting Started" on the right side navigation.

Note: Remember the URL you fill out as the name of your application, this will be used later in the POM file

Step 2: Setup the project
Once you have your GAE account setup, you need to setup your web application.

Suggested directory structure and project layout:
pom.xml
src
    - main
        - resources
        - site
            - img
            - javascript
            - css
        - webapp
            - WEB-INF
        - java
            - com
                - example
                    - controllers
    - test
        - resources
        - java
            - com
                - example
                    - controllers

This is just a standard Spring Web App, nothing special here. As a shortcut, you can use the maven command: (this will create most of the folder structure and default files)

mvn archetype:create -DgroupId=com.example.app -DartifactId=movie-app -DarchetypeArtifactId=maven-archetype-webapp

At this point, you should be able to "clean" and "install" the application. Nothing will happen since we have not setup the project, which is in the next post.

Run the project with:

mvn clean install

Depending on how your maven usage, different plugins may need to download but ultimately you should see "Build Successful" in the output.