The call center agents should use mostly the standard web applications, but with an additional telephony control that allowed them to accept incoming calls, to disconnect calls, or to make consultation calls to other agents or supervisors.

An incoming call

Let’s have a look at the most important usecase first: an incoming call.

The following diagram gives an overview of the flow of events, before the agent’s telephone rings:

The simplified flow of events before an agent receives a call.

The incoming call of the customer is handled by the PBX (Private Branch Exchange). When the agent finally takes the call, a lot of information about the customer has already been collected. In most cases, the caller will already have gone through an interactive voice response system that has collected his account number and verified his PIN (omitted from the picture above).

This is how the agent screen might look like after the agent has taken the call:

Schematic screen of the call center agent UI.

The box on the left are the telephony controls. They are embedded in an iframe and allow the agent to disconnect the call or place consultation calls to other agents. The telephony controls send commands to the gateway (and in extension to the PBX) and receive asynchronous events.

CSTA as a Model for our Protocol

Before it was even determined whether the telephone system should be integrated directly through the PBX or via an integration layer (Genesys), we decided to use the CSTA Phase III communication protocol as an orientation for the protocol between gateway and browser. CSTA (Computer Supported Telephony Applications) is an ECMA standard (like JavaScript) and describes third-party call control using services and events. Third-party call control roughly means that the standard looks at an entire switch and all connected devices (telephones), and not just a single telephone. This point of view is reflected in the naming of the services and events. For example, when a call arrives at a terminal the event is called Delivered. A sample event flow is given in the following diagram.

Exemplary flow of CSTA services and events in our system.

Services are commands to the telephone system. An outgoing call (from any device within the domain of the switch) is initiated by a Make Call Service. But there are also services like Set Agent State.

CSTA is extremely comprehensive; we used only a small selection of its services and events. It is also easy to extend — the transfer of non-standardized key/value pairs within the data part of services and events is explicitly provided for.

CSTA provides an ASN.1 and an XML encoding. Writing an ASN.1 parser in JavaScript was obviously not a good idea and even the XML mapping is quite heavy-weight and we decided to design our own transport encoding on top of JSON and built a REST-inspired web service as a gateway to the PBX.

Requirements

The customers’ technology framework requirements were:

• Internet Explorer 8 as the browser for the call center agents
• Wicket as web application framework for the call center application
• Tomcat 7 as the web application server for both the call center web app and the gateways.

The technical requirements were: minimal latency, high throughput, and high availability. An average delay below 150 ms was required for latency, i.e. a value slightly below the attention threshold. For the call center callers, very low latency is not crucial — most callers will have waited in the queue for minutes rather than seconds to reach a free agent anyway. But the new web application should — if at all possible — not worsen the ergonomics for the call center agents. In the end, this wasn’t a problem: during tests using moderate load latencies below 80 ms could be achieved.

High availability is an obvious requirement: if a call center with about 1000 agents fails, there will be many unhappy customers. On an unlucky day the failure will even be reported in the news. We solved the problem by designing for redundant server components and a low latency failover protocol. The actual web application uses Tomcat’s built-in clustering mechanism. We couldn’t reuse this for the telephony gateway, because the relevant state is distributed across the switches anyway.

The gateway has two essential reliability requirements:

• Commands to the telephone system have to be retried quickly if a gateway fails.
• Telephony events must not be lost.

The functional requirements were straightforward:

• Incoming and outgoing calls (simple call control)
• Call forwarding (single-step/two-step transfer)
• Forwarding to the IVR (Interactive Voice Response) — including customer dependent data — as well as routing back to the same agent that originally took the call
• Setting and displaying the agent status

Architecture

The architecture consists of several interconnected systems as shown in the diagram below:

• The call center agents’ browser with the JavaScript/HTML,
• Telephony-related systems (left): the gateway (a server-side web application running in Tomcat) and the PBX,
• Call center web application (right): Wicket-based web application and its database(s).

The integration of telephony and web application happens in the browser. The web application includes our JavaScript library and a telephony control panel in an iframe.

The architecture of our CTI solution based on web technology.

Sending Server Events to the Browsers

For redundancy, every client connects to both gateways, and keeps the TCP connection open. This means that every application server (Tomcat) of the gateways has to hold nearly 1000 open connections. We use the AIO-Interface of Tomcat 7, so all these connections can be processed by a single thread. This greatly minimizes memory requirements and scheduling overhead.

Overview of our system architecture with redundant gateways and PBX systems.

Server-sent events (a.k.a. server push) was recently standardized as part of HTML5 in the EventSource interface. Another convenient method to implement bidirectional communication is WebSockets. But we couldn’t use any of these due to the use of legacy browsers — we were glad we didn’t have to support IE6 and could rely on at least IE8. So we implemented a COMET variant, which essentially consists of long running XMLHttpRequest through which events are sent as chunked responses.

The asynchronous events from the gateways are decoded by our JavaScript library, which updates the telephony control and forwards the events to the interface part of the browser, which in turn may trigger a server interaction.

Cross-Domain COMET with IE8

The customer wanted to be able to run the web application and the gateways on different application servers. This means that the Javascript XMLHTTPRequests are cross domain, which turned out to be a small challenge on IE8.

Mozilla Firefox, Safari and Google Chrome all support the CORS (Cross-Origin Resource Sharing) specification of the W3C. IE8 supports it as well; however, with IE8 one must use XDomainRequests instead of XMLHTTPRequests, and the API is slightly different. There is a also a subtle buffering bug within IE8 that makes it necessary to set 2 KB of fill characters on every new COMET connection to ensure that the next event is received by the application immediately.

Redundant Gateways and PBX

Each browser keeps two connections to two different gateways. One is active, and the other is a hot standby. When the connection to the active gateway is broken, the hot standby gateway is immediately activated. If necessary,  the last failed command will be retried. As the hot standby gateway has been sending events the whole time as well, it is guaranteed that no event is lost. After this failover, the connection to the failed gateway is retried. When it is active, the previously failed gateway has become the host standby gateway.

Loss of a gateway does not lead to the loss essential state — the gateways hold as little state as possible. All relevant state was either pushed up into the JavaScript library or down into the PBX integration layer. The gateways are also independent of each other and interchangeable. This makes the solution inherently scalable. More gateways can be added at any time.

The PBX (a Genesys installation) itself is also redundant. The fallback on this level is hidden by the Genesys API and the gateway doesn’t have to handle it.

Testing

Our solution was tested with three different methods:

• Javascript unit tests with QUnit,
• A simulator that implements the gateway’s HTTP services and simulates a single agent telephone (with a Swing GUI), and
• Load tests.

Writing the simulator was a substantial effort, but it helped in two ways:

• It made development without the telephony hardware possible.
• It made it easy to test scenarios that were not reliable testable with real hardware (like deliberate race conditions).

In an ideal world, the load tests would have been performed with an external load test tool. We didn’t have one available, so we wrote our own load test generator using the CSTA API to generate and receive calls.

Conclusion

Our solution is light-weight, conceptually simple and scalable. The simplicity is the result of two development iterations and rather long design phases.

The decision to use CSTA as the blueprint for the communication protocol worked well, too. It was helpful that we did not have to re-invent two-step transfer for the umpteenth time. Also, the CSTA vocabulary (which goes down to the text in the log messages) can be understood by personnel that are familiar with CTI.

In the Footprints of Arnold Schwarzenegger

Call center applications always remind me of a slightly silly movie starring Arnold Schwarzenegger as an undercover agent and Jamie Lee Curtis as his unsuspecting wife. His cover story for her is that he is doing something with IT and in one scene she inquires about his day at work. He starts telling her enthusiastically and quite elaborately about a call center integration — and she nearly falls asleep.

I, however, think the combination of a call center and a web application is technically quite fascinating.

(c) 2012 mgm technology partners. This posting "Building a scalable Web-based Call Center CTI Solution" is part of the mgm technology blog. The author of the posting is Lars Immisch.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

The starting point was that we had implemented a class with some fields and needed quick access to its instances. This use case was pretty performance-critical, so we took a HashSet to maintain the class instances. As the instance count was high, we needed a broad distribution of possible hash-values. We used the IDE to generate code for the hashCode() and equals() method. Everything should have been fine with this generated code — at least we thought so. However, our application showed some unexpected results and it took us quite a long time to nail down the root cause.

Here’s a simplified example of code using a HashSet of PhoneNumber instances:

// Class PhoneNumber implements hashCode() and equals()
PhoneNumber obj = new PhoneNumber("mgm", "089/358680");
System.out.println("Hashcode: " +
	obj.hashCode());  //prints "1476725853"

// Add PhoneNumber object to HashSet
Set<PhoneNumber> set = new HashSet();
set.add(obj);

// Modify object after it has been inserted
obj.setNumber("089/358680-0");

// Modification causes a different hash value
System.out.println("New hashcode: " +
	obj.hashCode()); //prints "7130851"

// ... Later or in another class, code such as the following
// is operating on the Set:

// Unexpected Result!
// Output: obj is set member: FALSE
System.out.println("obj is set member: " +
	set.contains(obj));

// Even stranger unexpected Result!
// Output: obj is set member: FALSE
for (PhoneNumber p : set) {
	if (p.equals(obj)) {
		System.out.println("obj is set member: " +
			set.contains(p));
	}
}

Executing the code above surprisingly produces the following output:

obj is set member: FALSE
obj is set member: FALSE

Obviously, what we would expect is a “TRUE”, since obj has been inserted into the HashSet.

What just happened?

The unexpected result from the code above is caused by a trap in the JDK Collections framework into which many developers have fallen: If an implementation of hashCode() uses mutable fields to calculate the value, HashSet.contains() produces unexpected results, i.e. your object seems to be not a member of the set.

For an illustration, let’s look at the class PhoneNumber and its mutable field “number”:

public class PhoneNumber {

    private final String name;
    private String number;

    public PhoneNumber(String number, String name) {
        this.number = number;
        this.name = name;
    }

	// Setter makes "number" mutable!
    public void setNumber(String number) {
        this.number = number;
    }

    @Override
    public int hashCode() {
        int result = name != null ? name.hashCode() : 0;
        result = 31 * result +
			(number != null ? number.hashCode() : 0);
        return result;
    }

	// equals() left out here ...
}

What’s wrong with this class? Well, it’s a bad idea to use mutable fields in hashCode() when its instances are put into a HashSet (or as keys into a HashMap). In general, any hash-based collection is problematic. See also “HashSet contains problem with custom objects” (Stackoverflow) and “hashCode() pitfalls with HashSet and HashMap”.

HashSet.contains() surprisingly uses hashCode()

Part of our problem to spot our bug was that the HashSet.contains() method relies on hash values to stay immutable. Unfortunately, this is not stated explicitly in the HashSet JavaDoc, which only mentions “…returns true if and only if this set contains an element e such that (o==null ? e==null : o.equals(e))“. Actually, this is the same description as the JavaDoc of Set.contains().

A conscientious reader may also find the following note in the JavaDoc of the Set interface, which only mentions equals():

“Great care must be exercised if mutable objects are used as set elements. The behavior of a set is not specified if the value of an object is changed in a manner that affects equals comparisons while the object is an element in the set.”

By the way, the following Sun JDK bug was reported quite some time ago: “(coll) HashSet.contains method violates Set.contains() contract”. The bug is approved (but not fixed) and the last comment was made in late 2007.

Properly coding the hashCode() method

The contract of hashCode() is explained in the JavaDoc of Object. You will also find hints on proper implementations in the very interesting book “Effective Java” from Joshua Bloch. The book covers many interesting topics and especially the items 7 and 8 shed light on good equals() and hashCode() implementation practices. Another book by the same author, “Java Puzzlers”, also contains two puzzlers on this problematic area: specifically, puzzlers 57 and 58 show how these two methods depend on each other.

There are many more discussions about the problems that can occur with hashCode(). For example, in his Google TechTalks presentation “How To Design A Good API and Why it Matters”, Joshua Bloch says that hashCode() is an implementation detail that should not have leaked into the Java API at all (at about 27:30 min).

And don’t forget the lesson learned here: using mutable fields in hashCode() is a recipe for disaster. And disaster strikes when instances of this class are put in a hash-based collection like HashSet or HashMap (as map keys).

Please note that since code usually uses only the respective collection interfaces, e.g. Set and Map, you might not even know about it (as in our case). Or you use a module or library that stores your objects in a collection internally as an implementation detail that’s hidden from you.

Don’t rely on automatic hashCode() Generation

Even with coding rules in mind, a hashCode() implementation that uses mutable fields creeps into our code base faster than you can spell “bug”. This is because developers are reluctant to write the long-winded calculations in the hashCode() methods manually and often generate them with the help of the IDE, as shown in the screenshot below. But it’s just too easy for a developer to press “Generate” without first checking the specific fields that can be included and leaving the mutable fields out. Of all IDEs I tested only NetBeans at least has all fields unchecked, which forces the developer to select them on purpose.

Modern IDEs provide automatic generation of hashCode(). Eclipse and Intellij IDEA by default include all mutable fields.

Code Inspection Tools to the Rescue?

You might wonder if classes of your code base contain an hashCode() implementation that uses mutable fields. One option (besides a manual code review) is using a code inspection tool. Unfortunately, the prominent open source tools like FindBugs, PMD, CheckStyle do not offer such a built-in inspection.

Only IntelliJ IDEA has a built-in code inspection that detects the use of mutable (actually non-final) fields in hashCode().

The only tool support I found was Intellij IDEA. This IDE provides a code inspection named “Non-final field referenced in ‘hashCode()’”. Any violation is highlighted as shown in the screenshot above.

(c) 2012 mgm technology partners. This posting "Consequences when using Mutable Fields in hashCode() - Hacking Java Puzzlers for Fun and Profit, Part 1" is part of the mgm technology blog. The author of the posting is Ulrich Schrempp.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

We develop many applications for e-government, finance and insurance. These usually rely heavily on the presentation and evaluation of data processed within interactive frontends. Legal and business reasons demand that the resulting applications meet very high standards of reliance and safety. For us, this meant that we had to find a way to work close with the customer to understand the domain while simultaneously improving coverage and efficiency of development and QA.

We met this challenge by using Domain Specific Languages to involve the customer in the creation a specification basis for the domain which could be used by our engineers not only as requirement description but also for the implementation of tools for automatic validation or even code generation. The work spent on the definition, development and maintenance of specification language tools proved to be worth it as quality improved and effort decreased. E.g. while we were analyzing and reporting test coverage in terms of project specific formalized test data, we were glad that we could rely on a resilient specification basis.

Specifications based on a Domain Specific Formal Language bridge the gap between Customers and Software Developers.

The proven benefits of formal specifications at the comprehension level of customers encouraged us to bridge the gap in more mgm projects: we motivate more customers to introduce a formal requirement process and more engineers to a domain specific, though formalistic view of requirements so they can benefit from the improvements.

This first blog article introduces our domain specific formal language approach. The upcoming second part will focus on quality benefits derived from formal specifications.

Setting the Scene

The quality of complex interactive applications always includes a high level of conformity to the respective customer’s requirements. It is not enough to have a well-functioning system; the system also has to do what the customer expects! This holds for both, explicit and implicit (sometimes never formulated) requirements. We believe that the only way to accomplish this is to learn from the customer and to struggle for exact and comprehensive requirements consistent with the customer’s domain.

Requirements modeled by Domain-Specific Formal Languages

We believe that the need for exact requirements can be met by using formal languages to describe requirements. To model and implement what the customer needs (and not what we believe that he or she needs), the specification must be formulated using the terms of the respective domain and be comprehensible for the customer. We achieve this through the adaptation and extension of our existing language family to support the customer’s particular domain.

Requirements modeled for Frontends – at least

It is common sense to separate front-end aspects (such as input and output handling for forms) from the full business logic modeled somewhere in the back-end. The main purpose of front-ends is to deliver to the back-end information which is guaranteed to be sound and consistent. The back-end implementers should not be bothered by interactive aspects and user level data consistency.

Our Approach

mgm technology partners’ approach to form-centric applications was to implement a framework supporting specification languages along the full software development chain: editors, code generators, documentation and test data generators for the supported language family.

Whereever possible:

• We use and promote formal specification languages for front-ends and part of the business logics.
• We involve customers in the requirement process, especially through including formal specifications for their domains.
• We tailor the specification framework to each of the customer’s specific application domain.

This yields a high return (for customers and mgm) in terms of delivery time and quality and, in addition, reduces the implementation and quality assurance effort.

Technical Benefits

The benefits of using formal front-end specification techniques are manifold:

1. Consistency of the customer requirements: It becomes very likely, that the system accepts and denies exactly what the customer expected (including implicit requirements which would have never been formulated in absence of formal specifications).
2. Reduction of programming effort and risk: Since formal specifications allow for automatic code generation, a great deal of programming effort and risk simply vanishes.
3. Increasing functional quality standards for the front-end: Obviously, generated code is consistent to specifications (once the code generator is mature and tested). Moreover, due to the formal description of legal inputs, test suites describing valid inputs are automatically generated.
4. Driving the tests of the back-end: Since the front-end specifications define the set of correct inputs, they serve as basis for tests for the back-end. Automatically generated test suites can be run which prove the quality of the back-end in a well defined setting. Test coverage goals for the back-end can be formulated in terms of the existing formal front-end specifications for the front-end. Test coverage becomes measurable in terms of front-end data and front-end use cases.

In this blog post we focus on the topics 1) and 2) described above. We do so by presenting a flavor of the used specification languages. Aspects 3) and 4) will be discussed in a second blog post.

Specifications supporting customer-related formal Requirements

In the following we will demonstrate how to derive a formal specification using mgm’s specification language.

An User Interface Example

Imagine a simple bill writing system based on a simple (e.g. web graphical) user interface where we will also need to consider both, calculations and consistency constraints.

Let us assume that a number of bill positions with a net unit price can be typed in, each of them with its own multiplicity (quantity) of at least 1. The system calculates the net price for each position as well as the baseline including the default or specific VAT (value added tax).

The input form might look as follows (we ignore GUI details in this example):

Sample form with input fields (yellow) and calculated fields (blue).

Input fields are marked yellow and fields which are to be calculated by the system (and thus are “read-only” for the user) are marked blue.

A completed form for two positions, with reduced VAT would look like follows:

Filled form with automatically calculated net and gross amounts.

Formal Specification for the Example

The specification language is now used to describe the properties of the fields, i.e. their types, the relations between their values and other constraints regarding consistency and completeness in a formal way. This approach is the core of the formalization of requirements.

Let us look at some formal specifications for the example above.

Types

Depending on the domain, fields have domain-specific types at user level. These types can be specified as numbers, currencies, date data, strings, enumerations and truth values. Most of them also include a specification of the allowed field length. For instance AlternativeVat is a positive numeric value with 2 digits which leads to the specification:

AlternativeVat: PositiveNumberDigits(2)

Some fields (e.g. all position related fields in the example) might have multiple instances. This is also specified along with the type definition. Multiplicity is specified as follows by using multi:

multi Position:     String(25)
multi UnitPrice:    EurosAndCentsDigits(8)
multi Quantity:     PositiveInteger(3)
multi PosFullPrice: EurosAndCentsDigits(10)

Fields values which are either subject to calculation by the system or which are given or by the specification are marked as calc or constant, respectively. Both kinds are not editable by the user (they are “readonly”).

calc       NetAmount:    EurosAndCentsDigits(10)
constant   NormalVat:    PositiveNumberDigits(2) = 19
calc       AllVat:       EurosAndCentsDigits(10)
calc multi PosFullPrice: EurosAndCentsDigits(10)

Functional Rules

Functional rules specify values in a functional setting.

For the fields AllVat and GrossAmount one gets straight-forward specification formulas:

AllVat = If FieldValueSpecified(AlternativeVat)
            then  AlternativeVat/100*NetAmount
            else  NormalVat/100*NetAmount
GrossAmount = AllVat + NetAmount

The semantics of these rules is intended to be functional rather than imperative, i.e. a field value is not influenced by any state aside from the occurring field values (Similar to formulas in spreadsheets or functional languages). Note that functional rules can also be used to specify mappings within the user interface, e.g. from drop-downs to textual representations.

Constraint Rules

Constraint rules describe constraints which have to be satisfied to ensure consistent inputs. They consist of

• a conditional clause specifying the constraint, which the field values have to satisfy, and of
• a fail-clause including a message which shall be issued if the constraints are not satisfied.

In this example, only specific VATs can be considered: normal VAT, half normal VAT or no VAT at all.

constraint
AlternativeVat == 0
   or AlternativeVat == NormalVat
   or AlternativeVat == NormalVat/2
   => failed: "VAT can only be normal, half normal or zero"

Note that the semantics is similar to assert statements: the failed-clause expresses feedback in case the conditional clause is not fulfilled.

Note further the differences between functional and constraint rules:

• Unlike functional rules, constraint rules specify constraints (operationally expressed: checks) rather than computations. For syntactic differentiation, we use “==” here rather than “=“.
• Functional rules do not have fail-clauses since they enforce values for fields rather than checking their relationship.

Rules for Multiple Instances

The language supports multiple instances of fields (and rules therefore) along two multiplicity dimensions – “All” and “Each”.

All-multiplicity for functional rules

For computations such as for the field NetAmount multiple instances have to be considered

NetAmount = Sum(PosFullPrice.all)

The “.all” next to PosFullPrice stands for an “all instances” semantics. We call this “all” multiplicity.

Each-multiplicity functional rules

Each instance of PosFullPrice is calculated in a homogenous way by multiplying the price with the quantity.

PosFullPrice.each = UnitPrice.each * Quantity.each

This functional rule yields for each row (i.e. instance), respectively and is denoted with the postfix “.each“. We call this each multiplicity.

This defines that the rule holds for each multiple instance of involved fields. Intuitively, one can view this as multiple copies of the rule for each instance.

Each-multiplicity for constraint rules

Consistency constraints for multiple instances of fields are expressed here as well.

In the example, it must be guaranteed, that each position (row) is fully specified. This is expressed with the specific predicate FieldsCommonlyDefined (using “each” multiplicity).

constraint
FieldsCommonlyDefined(Position.each, UnitPrice.each, Quantity.each)
    ==> failed: "All fields (Position , UnitPrice, Quantity) must be specified if one is specified "

All-multiplicity for constraint rules

Obviously, “all” multiplicity is usable in constraint rule as well. In the example, since the system should not print empty bills, one has to specify multiplicity greater than zero at least for one of the fields referring to positions.

constraint
AtLeastOneInstanceExists(Position.all)
    ==> failed: "Please specifiy at least one position"

This completes the example showing some of the specifications means. The full specification is given in the appendix. Since it is complete it can be used for automatic code generations for computations and constraint checking.

The aim of this short example is neither syntactic accuracy nor completeness. It is merely an illustration of important aspects. Please remember that each language of the Specification Language family is tailored specifically anyhow.

Summary: Characterization of the Specification Language Family

In a nutshell, the specification language family described above defines valid inputs and front-end computations for web applications or other form based systems. The formalism can be characterized as a subset of typed predicate calculus defined in terms of the customer domain. For the sake of simplicity and comprehensibility, we do not extend to full predicate calculus or to higher order logic. The pragmatic expressiveness of the language family is more important than the theoretical power of predicate calculus.

Here is a summary of the aspects which can be described.

• Typed field: Valid and invalid field value using field description.
• Field-value-constraints: valid and invalid values for related fields.
• Existence-constraints: validity and invalidity of existent and non-existent input value depending on values or existence of input value for other fields.
• Both kinds of constraints can be interwoven.
• Computation of field values based on other field values. This is expressed by rules including:
  • mapping of external presentations to internal ones (such as mapping drop down selections to values),
  • constant values which show up in the user interface,
  • conditional values.
• Multiplicity: fields may be defined to occur in several instances, controlled by the specification. In terms of constraint control this allows to express iterative aspects on a higher level than for single fields. There are multiplicity aspects expressible by the specification language family (not needed in this example). This, however, is beyond the scope of this paper.

And here’s the full specification example:

# Fields and Types:
multi      Position: String(25)
multi      UnitPrice: EurosAndCentsDigits(8)
multi      Quantity: PositiveInteger(3)
multi      PosFullPrice: EurosAndCentsDigits(10)
           AlternativeVat: PositiveNumberDigits(2)
constant   NormalVat: PositiveNumberDigits(2) = 19
calc       NetAmount: EurosAndCentsDigits(10)
calc       AllVat:  EurosAndCentsDigits(10)
calc multi PosFullPrice: EurosAndCentsDigits(10)

# Functional Rules:
AllVat = If FieldValueSpecified(AlternativeVat)
            then AlternativeVat/100*NetAmount
            else NormalVat/100*NetAmount
GrossAmount       = AllVat + NetAmount
NetAmount         = Sum(PosFullPrice.all)
PosFullPrice.each = UnitPrice.each * Quantity.each

# Constraints:
AlternativeVat == 0  or  AlternativeVat == NormalVat
      or  AlternativeVat == NormalVat/2
      => failed: "VAT can only be normal, half normal or zero"
FieldsCommonlyDefined(Position.each UnitPrice.each, Quantity.each)
      => failed: "All fields (Position , UnitPrice, Quantity) must be specified if one is specified "
AtLeastOneInstanceExists(Position.all)
     => failed: "Please specifiy at least one position"

Development and Testing issues

Due to formal specifications, both, software development and quality assurance become less cumbersome, more comprehensible, easier to track, more scalable and safer. We conclude by shortly sketching these aspects.

Avoiding Implementation Efforts and Risks through Code Generation

Similar to programming languages, specification languages substantially simplify software development. Code generated from specifications eliminates a great deal of complexity and leads to less error-prone systems. Both functional rules and constraint rules can be used as input for generators to generate fully operational code. This is a highly efficient way to automatically produce components for the validation of data in a complex solution environment.

Once the code generators are well tested there is not much need to test for each front-end application again. mgm technology partners developed code generators that generate code for Java, C++ and Javascript, each of them running in a variety of environments.

Test Data Generation and Test Coverage

Since the used specification languages describe well defined inputs they are an ideal prerequisite for test data generation: consistent test cases are generated directly from the specification due to

• the existence of type definitions for fields,
• the existence of constraint rules describing the relationship between fields,
• and the existence of calculation formulas for calculated fields values.

Test data generation occurs with few exceptions fully automatically. For specific test cases it is also possible to automatically generate inconsistent data by deliberately enforcing wrong data. This is done by negating constraints and re-running the test data generator.

mgm technology partners has developed a test data generator suite based on the used formal languages. The tool is used for the generation of big and complex test cases allowing for extensive and well defined test coverage. Some insights can be found in our blog articles “Form Validation with Rule Bases“, and in “Producing High-Quality Test Data“. Test coverage and other quality assurance topics issues will be highlighted in the upcoming second blog article.

(c) 2012 mgm technology partners. This posting "Using Domain Specific Languages to Implement Interactive Frontends - Software Quality driven by Formal DSLs, Part 1" is part of the mgm technology blog. The author of the posting is Dr. Jürgen Knopp.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

In a typical non-caching setup of a web application as illustrated in the figure below, Apache handles static requests for images, scripts, etc. and forwards requests for the HTML pages to an application server like Tomcat or Glassfish. There the dynamic content is generated and then sent back to Apache and finally to the user. In this scenario, the database access is the most critical bottleneck. Even worse, each page request can cause multiple database requests, i.e. SQL statements.

Initial setup without our caching solution: the slow components are shown in red. (Load balancing, which could be done by Apache, is not considered here.)

Let’s assume that without caching, an application server can serve up to 100 dynamic pages per second. Through a bit of vertical scaling, i.e. using two server instances (nodes) and load balancing, this can be increased to about 200 dynamic requests. However, this scalability is not perfect and once it grows to three and more nodes, it already starts to get worse as the sessions have to be distributed among the nodes in the cluster.

The system can of course handle many more simultaneous users than the number 200 suggests, as users do not permanently access links. So the number of users did not really pose a problem during normal operation. However, the situation immediately got critical when newsletters with special offers were sent, as the application server instances were now under “siege”. An overload of the instances led to slower and slower responses and decreasing customer satisfaction. Another reason a shop would want to be responsive, is that search engines consider measured response times during crawling for ranking search results.

So the question we had to solve was: How can we keep the system responsive (ideally with a response time of 1-2 seconds) during high load and peak situations? Please be aware that in the case of online shops, the highest turnover occurs in these situations.

When we analyzed the server log of the www.lidl.de online shop, we noticed an interesting fact, which we used to our advantage later on: the behavior of users is different in these situations. Most users are just browsing and reading. Consider e.g. a newsletter sent to a few million users: most of the readers will just click a few links (which can still easily amount to several million page impressions). Taking a deeper look we found out that most users are viewing absolutely and exactly identical content which has nevertheless been produced exclusively for them. Only a small percentage used the interactive services of the website like shopping carts, ordering etc.

Introducing Varnish

The peak situation described above implies that most content (even though dynamically generated by the web application) is identical for all users. So the obvious idea for a cache is to store the most frequently requested pages. The Varnish manual describes Varnish as a lightweight, efficient reverse proxy server, meaning it is working in front of the web servers (Apache). It acts as a so-called HTTP accelerator which stores (caches) copies of the pages served by the web server (thus the synonym “web cache”). The next time the same page is requested by a user, Varnish will serve the copy instead of requesting the page from the Apache server. Varnish is blazingly fast, since it stores its cached data in memory.

The new architecture with Varnish as a web cache now looks like this:

Varnish in front of Apache acting as a Web cache. It is configured to cache only stateless page requests. Stateful page requests (session) and static resources are forwarded to Apache.

Performance Improvements

Caching with Varnish removes the need for the web application to regenerate the same page over and over again, resulting in a tremendous performance boost. Varnish can easily handle 10,000 requests/s on a single node. Especially in high load situations the hit rate is easily above 90% (and almost 100% for the mostly clicked homepage) so that the setup described above can now handle 50 times the original volume. However, this high performance will only hold for stateless users. Any user with a session will fall back to the 100 requests/s class.

As most of the load is now taken by the Varnish cache servers, the load on the application servers has dropped considerably. Even in high load situations where the Varnish servers handle several thousand requests per second, most of the content comes from the cache and the application servers can concentrate on re-creating expired content (which is then kept in the cache for s-maxage seconds) and handling users with a session (who are hopefully going to order).

Our setup leads to a significantly improved end-to-end performance of the system – even during normal operation. This is interesting as it creates an advantage for users during normal operation and saves money for the website owner at the same time.

Using less hardware means investing less money initially. However, an even more important fact is, that the operating costs will also be much lower. These operating costs are caused by permanent maintenance of the system, like powering servers around the clock, updating, applying patches etc. Since these costs are the main drivers for the total cost of ownership (TCO), the potential savings are also largest in this regime.

Using fewer servers also means consuming less power. By reducing the energy bill this “green IT” approach therefore leads to lower operating costs. Compared to extending the existing system without a cache, an enormous amount of money was saved both in hardware and operating costs, while introducing a “performance buffer” for situations with even higher loads at the same time.

Another effect is that the shop’s marketing division can now act freely without having to keep technical constraints in mind: new campaigns can be planned to increase the turnover significantly, like sending more frequent newsletters, using special offers etc.

Challenges

Before we dive into the details of our Varnish configuration, let’s first discuss three problems we had to solve, specifically handling stateful users, keeping users stateless w.r.t. caching as long as possible, and caching pages with changing content.

Problem: Websites are Stateful

Most websites nowadays are stateful, e.g. a server-side session is created when a user logs in. In case of an online shop, the session might contain the shopping cart, login information etc.

The problem is that as soon as the session contains personalized information, caching must immediately stop. But, as long as state information does not have an effect on the content of generated pages, it can be ignored. This is what we call a stateless or browsing user, and our first objective should be to cache pages suitable for this user class.

Thus, our solution is to classify users, i.e. to carefully distinguish between stateless and stateful users. As the web application did not originally take care of that, it had to be changed in two fundamental ways:

1. The application must only generate and send cookies if it has created some internal state for a user.
2. This state transition can happen at any time. So a user who has not even touched the application server and is completely unknown to the application must be able to become a stateful user at any time.

Two classes of users are distinguished by certain attributes. A user should stay stateless as long as possible. Stateful (red) users will need contact to the application server and experience slower performance.

Fortunately, the web application was already obeying the REST paradigm. HTTP GET requests were used for all content that was just shown to the users. In contrast to this, all user actions which were actually creating some state on the server side were already modeled in HTTP POST requests. This proved to be extremely helpful when we started to configure the cache software.

Keeping Users Stateless

The general goal must be to keep users stateless, at least as long as possible. In a first naive approach, only this facilitates caching.

Keeping users stateless means that the server should never send a session cookie unless really necessary. On the other hand, a lot of web applications require some basic personalization. This dilemma can be solved by using cookies which will be evaluated on the client side only. For example, let’s assume that users can change the background color of the website as a very simple form of personalization. This can be performed by Javascript and, for the sake of caching (and achieving a high hit rate), this should be the preferred way of doing simple personalization. Of course, a server-side cookie for personalized background color could be used to get the same result. But the cache hit rate would then suffer considerably (to be exact, by a factor identical to the number of background colors, since exactly the same amount of cached copies has to be saved).

So one recipe for staying stateless is to keep simple state on the client-side and never send it to the server. This state does not necessarily have to reside in a cookie – you can also use the browser local storage for that, as described in Smashing Magazin’s “Using Local Storage In HTML5-Capable Browsers” article.

Dealing with Content that is Changing

Even if now all stateless users can see the same cached content, this content is changing over time. In an online shop, for example, some products might run out of stock and become unavailable or need to be replaced by other products. Unfortunately, this does not only affect the product pages themselves but sometimes also pages that reference them; e.g. links and thumbnail images will have to be changed or removed. Similar situations often occur in online publishing and in nearly all websites which change over time.

Thus, another requirement for the cache is its ability to partially expire content. And of course, the bookkeeping must be performed externally so that the affected pages can be removed individually.

For the cache to work properly and perform automatic expiration of content, it needs to know how long the currently cached content should be kept (i.e. its maximal age). The web application therefore has to generate this so-called time-to-live (TTL) information.

The HTTP specification has defined HTTP response header fields such as Cache-Control for exactly this purpose a long time ago. These are set by the web application itself, since it knows best how long the content will be considered “current”/”valid”. This setting could even be dynamic , e.g. giving a shorter time-to-live to a product page if stock is low. The Cache-Control directive most suitable for this purpose is s-maxage as it specifies the maximum age of the object in seconds that the response is allowed to be kept in the web cache.

Determining Cacheable Candidates

Not all content can be or even should be cached. Caching on completely static websites is easier by far, however, these tend to be very unattractive, could be pre-generated and then moved to the web server. As the cache will sit in front of the web server, all requests will go to the cache first. It does not make much sense to store pages in the cache which are kept statically in the web server’s file system anyway.

On the other hand, only GET URLs can be candidates for caching. As a POST request transmits information from the browser to the server, it cannot be cached and must always be handled by the application server. This might sound like a big constraint at first but is actually a feature that can be nicely utilized: all URLs which are candidates for performing the state transition of a user from stateless to stateful will be POST requests. And consequently, the application itself can decide whether the POST requests actually qualify for making a user stateful or whether s/he can remain stateless, for example when a wrong login/password combination is entered.

Anatomy of Varnish’s Request Processing

Varnish distinguishes three stages when processing a request:

• The request is received from the browser (vcl_recv).
  At this stage, Varnish calls the subroutine vcl_recv in the configuration file (VCL). Here, the request header can be manipulated e.g. by removing cookies. It can be decided whether the content should be looked up in the cache or be propagated to the backend server.
• The response is received from the backend (vcl_fetch).
  This function is only executed when the content is not delivered from the cache. In this phase, response headers from the backend can be modified (either for delivery or for saving in the cache). The request attributes are also still available and can be used for manipulating several settings.
• The response is sent to the browser (vcl_deliver).
  This stage is passed by all requests and can be used to add headers (like TTL), change cookies etc. The request parameters are available for reading.

Different stages of Varnish's request processing. Everything related to the cache is in red, i.e. all cacheable content is looked up in the cache and possibly delivered; if it's not in the cache, the web server will be asked via vcl_fetch.

Varnish defines additional subroutines which also hook into the Varnish workflow, but they are not as important. See also the VCL tutorial and the VCL reference.

A Sample Varnish Configuration (VCL)

This section contains a simple Varnish configuration that provides caching as required. The challenge is to keep the user stateless as long as possible. In order to achieve this, a simple trick is used: if a request does not contain a JSESSIONID cookie, it is a stateless request and even if the (uneducated) backend wants to set a cookie, it will be removed. Only POST requests will set necessary cookies. Manipulating the TTL compliments the configuration. A lot of logging is used in the example; this is not just for illustrative purposes but also practical for debugging and optimizing the configuration.

import std;

backend default {
    .host = "localhost";  # Varnish is running on same server as Apache
    .port = "80";
}

sub vcl_recv {
  # remove unnecessary cookies
  if (req.http.cookie ~ "JSESSIONID") {
    std.log("found jsessionid in request, passing to backend server");
    return (pass);
  } else {
    unset req.http.cookie;
  }
}

sub vcl_fetch {
  if (req.http.cookie ~ "JSESSIONID" || req.request == "POST") {
    std.log("not removing cookie/passing POST, url " + req.url);
    return (pass);
  } else {
    # remove all other cookies and prevent backend from setting any
    std.log("removing cookie in url " + req.url);
    unset beresp.http.set-cookie;
    set beresp.ttl = 600s;
  }
}

sub vcl_deliver {
  # send some handy statistics back, useful for checking cache
  if (obj.hits > 0) {
    set resp.http.X-Cache-Action = "HIT";
    set resp.http.X-Cache-Hits = obj.hits;
  } else {
    set resp.http.X-Cache-Action = "MISS";
  }
}

Notice the C-like syntax in the Varnish configuration. This is no accident; in fact, the whole configuration code is compiled to a binary shared object at startup and when reloading the script to optimize for performance. As the subroutines in this configuration are called for each request, this helps immensely in creating a fast cache server. Moreover, it is possible to add C code directly to the configuration.

It might seem strange at first to define the configuration in a procedural language, but it proved to be extremely valuable as it enables us to be flexible and to formulate how exactly to handle the requests. Overall, this leads to a much more readable configuration than a declarative approach.

Notice the different “top level” objects in the configuration file:

• req is the request (i.e. the URL including all headers) coming from the browser,
• resp is the response before it is sent to the client, i.e. when it can still be manipulated.
• beresp: The response which Varnish gets from the backend (if the object is not cacheable or not cached) is also available as beresp and can be evaluated.

On a side note, Varnish can use ACLs to restrict the access to certain resources. The same ACLs can also be used to (declaratively) tell Varnish what to cache and what not. This technique is the sometimes used “banning”. Varnish can also (atomically) delete certain elements from the cache. This is accomplished via a “purge” command through the HTTP interface and should be restricted to IP addresses (which is the standard configuration together with a secret).

Configuration Details and Tips

Now that we have seen the basic VCL file and understood how a request is usually processed, let’s dive in even further and discuss the details and lessons learned.

Improving the Hit rate with Header Normalization

Varnish has to be told which HTTP request header fields it should use as a cache index. The index is organized as a hash, thus these selected header fields are often referred to as the hash key.

On a side note, you can select the header fields to be used as a hash by implementing the subroutine vcl_hash. If you don’t implement it, Varnish uses the full URL plus the Host request header field by default. In addition to the hash key computed in vcl_hash, the Vary header field is always automatically added to the hash key . For further information on the hash key, see “What Varnish Does” and “Varnish and http header” on Stackoverflow.

To improve the cache hit-rate, it is crucial that you clean up the request header fields used for the hash key. Cleaning up means to change them to a common denominator (so-called header normalization). Another very good candidate is of course the Host header field, where a normalized version (like “www.sitename.com”) should be used even if “sitename.com” is sent in the request header. In addition to that, removing unnecessary headers is always a good idea.

Be careful that the application server does not send a Vary header field for the user agent as this effectively means that there has to be a distinct copy for each user agent. There are so many different browsers (http://panopticlick.eff.org/) that this will basically make caching useless. See also “Understanding the HTTP Vary Header and Caching Proxies (Squid, etc.)” and the
Varnish Documentation on Vary.

Compression

The Accept-Encoding request header field plays an important role: it can have different values like “plain”, “gzip” or “deflate”. Unfortunately, Internet Explorer prefers the deflate encoding while all other browsers favor gzip. Without intervention, this leads to different copies of the same content in the cache, one in deflate format, the other in gzip format.

Since the request header can be modified on the fly in the vcl_recv
subroutine, we can effectively control that only one variant of the content is cached. In your VCL you can modify the request header field and use gzip exclusively if it is available (which is true for both Internet Explorer and others). This technique is presented in detail in the article “Normalize Accept-Encoding header”. Since both browser families have a market share of roughly 50%, this simple change effectively doubles the hit rate.

Please note that beginning with Varnish 3.0, Varnish supports gzip natively and can modify the Accept-Encoding field by itself, so the measures discussed in the previous paragraph can be skipped.

Handling Cookies

Cookies basically fall into different categories:

• Cookies relevant for caching: These should be kept and their values can be used as part of the hash key for cache index.
• Cookies irrelevant for caching: These should be discarded and not considered by the cache.
• Cookies partially relevant for caching: These should be modified and the irrelevant parts should be removed. The remaining cookie should then be used as part of the hash key for the cache index.
• Session cookies: These cookies must be treated differently as they basically make caching impossible. If such cookies are detected, Varnish should not cache anything but work as a proxy only sending data from the backend server directly to the client.

Consistent Values for TTL and the Expires Field

Varnish has to decide whether and how long to keep elements in the cache. As we have already learned, the Cache-Control header field is utilized here. More specifically, the s-maxage directive part (or maxage as a fallback if s-maxage is not present) is examined to determine the specified maximum lifetime of a cacheable object. Of course, this only works as long as the cache is not full; in the latter case the LRU algorithm is used.

If the web application was not designed with a web cache in mind, it might have conflicting values in s-maxage and the Expires response header field. (See the HTTP specification for a discussion of the Expires versus the maxage field.) This might lead to the bizarre situation that the cached content is sent by Varnish with an Expires header field value that lies in the past if s-maxage has a larger value than Expires.

This weird behavior can be fixed in several ways, e.g. by statically setting the Expires header in Varnish for each request to s-maxage seconds into the future during “vcl_fetch”. This will increase the cache efficiency on the browser side and lead to a more responsive website.

File Descriptors

In our first tests the solution performed well, but not excellently. But even more critical were the many dropped connections, i.e. requests from browsers that did not even reach Varnish.

The reason and the fix were easy – the number of file descriptors had to be increased. This is even more important in real-life situation where connections tend to be slow, as each TCP connection consumes one file descriptor. It does not hurt to allow 32768 descriptors for Varnish.

Monitoring

If you have setup a web cache solution with Varnish, it is important to measure its performance and especially monitor the hit rate of the cache. This turns out to be a bit complicated since the Varnish log files are not written to disk for performance reasons; instead of this, Varnish logs to a circular buffer residing in a shared memory segment. The circular buffer can be read at any time but past values will vanish forever. Since we wanted a monitoring solution that would also allow us to perform a post-mortem analysis in case of a problem, we configured the logging to write the circular buffer to a persistent file.

The most relevant tools for monitoring Varnish are:

• varnishlog: This shows current requests from the logging ring buffer. Usually request phases will be shown in chronological order which mixes up the requests themselves. This can be fixed by using appropriate options though.
• varnishtop: This shows the CPU distribution inside the varnish process and can be used to optimize the configuration if too much time is spent in only a few functions.
• varnishhist: This is easily the most intuitive and graphical tool for analyzing Varnish. It shows a (text) histogram of the response time distribution and thus gives a good overview how the whole system is performing.
• varnishstat: This shows important statistical information about hit rates, total cache hits, accepted connections from clients etc.

Why Varnish is the best Caching Solution (for us)

When we began to investigate ways to speed up the www.lidl.de site, our first choice was to add the Apache mod_cache caching module to the Apache web server already in use. The first hurdle was the declarative configuration; it is well-suited for a web server but not perfect for modeling a caching behavior. After some fiddling around, it was working smoothly. But more serious problems arose from the fact that certain cookies had to be considered and others had to be neglected. It was impossible to find a viable solution, so the cookie was filtered out by the load balancer. Cache invalidation is performed lazily in Apache, i.e. an outdated resource is removed from the cache only after it is requested. Consequently, outdated resources which are not requested will stay in the cache forever and can only be expired externally. As all cached components are distributed in single files, this expiry is slow and the whole process complicated. For our situation, Apache was not a good solution (although it was in use for quite some time) and hit rates were also rather disappointing.

So our search continued. Via dedicated proxy servers, which are more suitable for large client-side installations like Squid, we finally encountered Varnish, an HTTP accelerator specially built for caching purposes on the server-side. Varnish is already used by many big websites like Facebook, Twitter (Search), Hulu.

Varnish is very flexible as it offers procedural configuration of all request stages in a C-like language (which is actually translated to C and compiled at start time to be as efficient as possible). This enables creative cookie handling and all kinds of other tricks which are usually needed in such a scenario. Varnish was specially designed to run on servers with a VM subsystem, so all cached objects live in a single memory-mapped file and can be accessed extremely fast. Varnish handles expiry automatically and correctly and is even much faster than Apache. So the decision was made to go with Varnish.

Other HTTP accelerators were also considered, but proved to be not feasible, like Oracle Web Cache, a commercial software package from Oracle Inc.; the problem here is that the cache cannot grow easily, and that the manipulation of requests and responses is limited. A hardware-based solution is e.g. F5’s BIG-IP WebAccelerator.

Further Optimizations

Below is a discussion of measures that build on a Varnish setup and would speed-up the page delivery even further.

Using a CDN to increase Scale, Reach & Performance

CDNs take care of delivering the static content while the dynamic content is served via the usual stack. They work in an inherently distributed way and have clever algorithms to select the topologically nearest server for each user. Static and dynamic content can be separated by using virtual webservers with different hostnames. The PDF article “Globally distributed content delivery” from Akamai provides an excellent introduction.

Most CDNs offer an API for invalidating all or partial content and respect the expires header field sent from the originating servers. So the Varnish server can work as a central content repository and will be the upstream server for refreshing the CDN.

Almost all traffic would then be served by the CDN. This saves a lot of bandwidth on the Varnish server and the Gigabit interface will not so easily be overloaded. Moreover, as traffic costs in the CDN are negligible, money can be saved as the hosting company does not have to increase its own upstream link. For more information on how to build a CDN see “How to build your own CDN using BIND, GeoIP, Nginx, and Varnish”.

ESI: Caching Page Fragments with diverse TTL

From a technical point of view, only pages which are requested by the GET method can be cached at all. This is due to the fact that – by definition – POST requests change state on the server which then necessarily needs to reach the application server.

However, the solution described above performs less “aggressive” caching since it just stops caching as soon as a session cookie is present. The effect is that stateful users never get cached pages and therefore might have to wait longer for the page to render completely. On the other hand, it does not make sense to cache pages for individual users since it is quite unlikely that the same user will come back to the exactly same page. Even if the user would come back, it would not be safe to assume that the page is still up-to-date (e.g. since the shopping cart might have changed in the meantime).

To speed things up again, a compromise needs to be found between caching invariant fragments of a page and producing personalized content on the fly for stateful users. Fortunately, Varnish offers the correct arsenal to perform exactly this decomposition by leveraging Edge Side Include (ESI).

When Varnish processes ESI tags, the page assembly (out of fragments) is done by Varnish. As these fragments are separate web resources (requested through GET or POST) they can be assigned their own cache settings and handling information. For example, a cache time-to-live (TTL) of several days could be appropriate for the template, but a fragment containing a frequently-changing story or ad may require a much lower TTL. Some fragments may have to be marked uncacheable.

It must be carefully analyzed how the decomposition of the page might look like as getting it right is essential to achieve a high hit-rate and a low overhead. In case of an online shop, the page could e.g. consist of different (graphical) fragments:

Decomposition of a typical page into user-specific, dynamic (red) and static (blue) fragments.

The shopping cart and the login details would then be transferred directly from the application server via an appropriate ESI fragment, whereas the rest of the page is identical for all users and can be stored in the cache. To minimize the number of requests from Varnish to the application server, both fragments can be transferred in one part and integrated in different locations on the page on the client side or via CSS.

Compared to the performance numbers above, the stateful performance is much higher when using ESI. Rates of about 500 stateful requests per second are now easily possible.

Memcached: Caching Session-specific Page-Fragments

If you examine the page diagram above, you might notice that even though the shopping cart and login details are user-specific elements on the page, they are not very dynamic, i.e. they change infrequently.

This leads to an opportunity for further optimization: the user-specific fragments can also be stored, but must of course be associated with the session of the corresponding user. As the information is not persistent (as it becomes invalid with an invalidated session) it can be stored in memory. Memcached is just made for this scenario and therefore a perfect fit, see e.g. the article “Storing Sessions in Memcache”.

Any change in the shopping cart or login details will trigger a regeneration of the HTML fragments which will then be stored in memcached. (This can be done in the same POST request by the application server.) Varnish will include the fragment from Memcached (either via direct integration, via Apache or via Nginx). A SessionListener within Tomcat can take care of removing stale sessions from Memcached.

Memcached is extremely fast. Even for stateful users this leads to a performance of well above 5,000 GET requests/s. POST requests are a different story as they still have to be handled by the application server. As they perform only internal tasks and write both to the database and Memcached, a rate of 500 requests/s is nonetheless realistic.

(c) 2012 mgm technology partners. This posting "Ultra-Performant Dynamic Websites with Varnish" is part of the mgm technology blog. The author of the posting is Dr. Christian Winkler.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

Let’s begin with how JIRA can be utilized during the initial project phases. The main steps here are to prepare a more detailed business modeling and to complete the technical and business concepts. These steps are tightly connected with the compilation of the requirements and the requirement management phase. The requirement analysts of the project and the responsible project manager will interview all the necessary stakeholders to get a complete picture of the required solution that the business modelers and architects need for their work.

Capturing Requirements as JIRA Tickets

During this requirement management phase all the collected functional and non-functional requirements will already be stored as JIRA tickets to control their content and impact and to prioritize them with respect to the solution and its implementation order (planning process). And exactly this phase can be used to create a first contact point for customers with JIRA: Involvement in the compilation of new requirements and detailing of already filed items.

But as the customer is not yet very familiar with JIRA in this very early stage, we typically choose to create all the new requirements tickets ourselves instead of the customer. This is not just to unburden the customer: we also want to avoid the additional work of correcting imprecisely formulated requirements.

The descriptions of requirement tickets should always be unambiguous and complete. Thus, the responsibility to verbalize requirements usually remains with us. But the customer can be involved at any time to contribute details and he can (and should) be an active part during the elaboration phase and deliver his input and expertise through comments to the respective tickets.

Another very important point in requirement management is the used terminology. It is very important to always talk (and to write) the customers’ domain specific language. Use only terms that can be understood by the customer! We find it very helpful to maintain a glossary of all domain specific words and terms together with the customer.

Customer Involvement in the JIRA Requirement Process

In addition to detailing the content of requirements and ensuring their correctness, the customer can take over two other important tasks in the requirement process:

• Assignment: Once a requirement is elaborated, the effort estimated and it is ready for realization, it has to be assigned to the supplier (us) for release planning and implementation.
• Approval: When the requirement is implemented and approved by the development team and our internal quality assurance, the realized requirement is ready for approval by the customer.

Both of these steps (assignment and approval) can be realized as workflow steps for the issue type “Requirement”. Dependent on the character of the project and customer, mgm runs projects with different levels of integration.

Proven Workflow Implementations

Let’s take a look at two requirement workflows that we designed for our projects, each with a different integration level of the mentioned steps.

Workflow 'Requirement' with dedicated customer steps for assignment and approval (Variant 1).

This first workflow (shown above) contains a dedicated step to alert the customer that the requirement is “READY TO ASSIGN”. On his dashboard the customer has a portlet listing all these ‘marked’ requirements as a working queue for assignments!

It is not strictly necessary that the customer himself executes the transition “assign” in JIRA. We have projects where the requirement assignment is officially sent via mail or e-mail by the customer. In these cases, our requirement manager performs the transition on behalf of the customer. But we also have projects where the customer himself pushes the “assign” button in JIRA.

Following the implementation part with the steps “IN PROGRESS”, “RESOLVED” and “CODE-REVIEWED”, the requirement workflow contains the steps “READY FOR TESTING” and “VERIFIED”. During the software approval stage the customer can use these dedicated steps to manage his testing and approval tasks. Once again, the needed filters are integrated into the customer dashboard. The development team will explicitly hand over all implementations that passed internal quality assurance to the customer. The approval transition “VERIFY ISSUE” will then be executed by the customer himself. Usually we convince the customer to do this directly in JIRA.

In this first example, the step “REVISION” of the requirement (the elaboration phase) is separated from the step “ESTIMATE”, thus the customer will keep the control of ordering the effort estimations.

Now let us consider a second workflow example as depicted below:

Workflow 'Requirement' with dedicated customer steps for assignment and approval (Variant 2).

The interesting parts here are the initial step “DRAFT”, the step “ANALYSIS” (before “REVISION”) and the approval step “SIGNED OFF”. The “DRAFT” step is especially useful if customers create requirement tickets by themselves. The distinct step “ANALYSIS” is an independent elaboration and phrasing phase for the customer (typically when they have a dedicated operations department) whereas the step “REVISION” is an elaboration phase for the project team (development). At the end of the whole implementation process the customer can use the step “SIGNED OFF” for the approval process.

Overview: Where and How to Involve the Customer

Requirement management is an obvious area for direct participation of customers, but more traditional areas like “bug tracking” and “change management” are also potential candidates for customer involvement.

Below is a collection of areas where we constantly try to convince our customers to participate directly within our established JIRA processes:

• Requirement management
  • Input of new requirement tickets
  • Direct participation in the elaboration phase (optionally with additional workflow steps)
  • Assignment for realization (workflow steps)
  • Testing and approval of implemented requirements (workflow steps)
• Change management
  • Input of new change request tickets
  • Direct participation in the elaboration phase (optionally with own workflow steps)
  • Assignment for realization (workflow steps)
  • Testing and approval of implemented change requests (workflow steps)
• Bug tracking
  • Input of new bug tickets
  • Testing and approval of fixed bugs (workflow steps)
• Software approval process
  • Execution of dedicated testing tickets
  • Approval of all individual development tickets (requirements, change requests and bugs) (workflow steps)
  • Issue the final software (or release) acceptance (workflow steps)

The “Change Management” process has to be aligned with the customers’ organization structure and change process. We experienced that especially change management is in most cases an already well defined process at the customer side. However, for “Change Management” we can typically apply the same workflow as for “Requirements”.

“Bug tracking” nowadays follows standard workflows. But bug tickets are also development tasks, i.e. changes to the product/source code. Thus, we extended the bug workflows with steps representing the approval and quality assurance parts (“READY FOR TESTING”, “VERIFIED” and “SIGNED OFF”) as well. This applies to all issue types leading to development activities where requirements and change requests just represent the controlling/management part and not the realization part, e.g. Requirements, Change Requests, Bugs and dedicated implementation tasks.

Dedicated JIRA Projects for the Customer and Development

Sometimes the periodic amount of JIRA tickets (e.g. needed for a software release) exceeds the “pain” threshold (typically > 300 per release) and the customer is beginning to loose the project overview and feels lost in the overwhelming amount of requirements, change request, bugs, QA tasks and tasks in general. Our recommendation for these cases is to split it up and create 2 dedicated JIRA projects:

• Customer facing project: Used for all operational bugs and incidents (source: customer and end-user), comprises the complete requirement and change management and the software approval process.
• Development facing project: Used for all bugs during the development phase, all implementation tasks derived from requirements and change requests, development internal quality assurance tasks, all tasks that are related to the project in general, etc.

Splitting up a JIRA project into a dedicated JIRA project for the customer and another one for the development.

The ticket handling is very easy: Assigned requirements within the customer project are just cloned and moved to the development project. The original and the cloned requirements are then automatically linked by JIRA. In the development project you can then create the appropriate division into implementation tasks needed for your team and component diversity. The status update of the customer’s source tickets (linked tickets) has to be done manually.

The development project is typically only visible for the development team and not for the customer. When we did this in the past, the customer’s initial doubts that we just want to hide information from him could always be resolved by simply opening the development project to him and showing him the hundreds of (open) tickets. Normally he would loose interest in this project very quickly because he is not getting any additional benefits out of it. On the contrary, he will be getting rather confused by the amount of information.

We’ve made really good experiences with the concept of 2 dedicated JIRA projects for the customer and the development, respectively. But there is a second way to remove redundant information from an overstrained customer. You can use JIRA’s security level concept. This way you can keep all tickets in one JIRA project. But you will then have to cope with the maintenance of security settings at ticket level due to the fact that security levels have to be set manually for each required ticket. To set a default security level is counterproductive because then every customer created ticket will be automatically hidden from the customer directly after creation.

Conclusion

In our experience the advantages obtained through direct participation of customers in JIRA exceed the disadvantages of for example the increased efforts necessary for JIRA configuration. A well informed customer who is directly involved in his project feels much more comfortable even if something goes wrong or the progress of the project gets stuck. Transparency is the magical keyword.

But you have to accept that every project has its own characteristics. It will be mainly influenced by the customer’s character, organization and stakeholders. You have to find the most appropriate and fitting level for a customer’s direct participation. Try to get the most accurate picture of the stakeholders you have to work with and then decide how they could fit into the process. And keep in mind that the process can always be adapted afterwards in order to achieve the greatest efficiency in project progress and customer satisfaction.

In keeping with agile practice apply continuous improvement to your project management processes: Change something – find out how it went – learn from it – change something again!

Summary of the Key Success Factors

• Let customers create tickets directly in JIRA: requirements, change requests, support inquiries, bugs.
• Incorporate customers’ duties and responsibilities (e.g. assignments, approvals) directly into the issues workflow as dedicated steps.
• Prepare specific filters and dashboards for the customer
  • for his duties (detailing, assignments, approval)
  • for overviews
  • for status
• Split projects with an overwhelming amount of implementation tasks into two separate instances – one for the customer and one for development.
• Give customers a short JIRA training covering all standard actions as well as how to use and adapt dashboards and how to interpret the data and analysis reports.
• Tailor every project set-up individually and don’t try to compress it into the same template.

If you keep all this in mind, you have a good chance that JIRA will become customer’s ’sweetheart’!

There are tons of other interesting topics around JIRA. I will continue to provide you with further ideas, suggestions and mgm experiences. And of course if you have additional questions, ideas, and suggestions around JIRA I would really appreciate any comments and input from you.

(c) 2012 mgm technology partners. This posting "Practical Customer Participation in JIRA Workflows - JIRA beyond Bug Tracking, Part 2" is part of the mgm technology blog. The author of the posting is Alexander Weiss.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

mgm has been using JIRA for project management since 2005 for all its projects, ranging from small projects (< 5 members), medium (< 30 members) ones up to really large projects with up to 500 members and more. Part of my job is setting up JIRA and designing customer-tailored, optimized JIRA workflows. We’re setting up JIRA externally for customers, usually as part of software development projects, and also internally, e.g. for marketing projects, front-office and administrative tasks.

During the past 6 years, our initially defined processes – especially the ones concerning project management and requirement management – passed through several iterations of refinement. We learned a lot and embraced our lessons learned into improved process definitions. And we introduced more and more agile methods into projects fitting to this methodology. The latest step in this direction was to extend JIRA with Atlassians agile plug-in GreenHopper which enables a gradual introduction of agile tools.

Agile task tracking using the GreenHopper extension.

We make sure that every project gets its own tailored project set-up (set of issue types, workflows, custom fields, GreenHopper configuration) optimized for their individual project needs. And although we have several standard configurations sets for new software projects inherited from our best practises, we use them only for the initial project set-up. Once the basic functionality is implemented we try to reflect the distinct project requirements (character of project, product and customer) by creating an individual project configuration instead of trying to condense them into a one-size-fits-all template.

About our largest JIRA Installation with 700+ People

One of mgm’s biggest pool for JIRA project experiences is the set-up and ongoing maintenance of a JIRA instance that supports a huge multi-project environment with more than 700 active members. The set-up for 28 managed software products is split into 60 JIRA projects with approximately 80 workflows definitions and almost 300 custom fields.

The projects form a network of vertical layers (product software development) and horizontal layers (cross-section functions like quality assurance, expertise, strategic product management, support, operation etc). There is also a strong interdependency between the software projects with respect to results and releases that we had to take into account. One of the largest of these software projects generates an average of 2.500 JIRA tickets for every main release (i.e. every 4 months). So we have a lot of different aspects to consider.

Software Project Process Maturity at mgm

As agile project management patterns become more and more prevalent within software projects, we adapted our projects’ lifecycle with the following steps: The typical project starts with more or less agile iterations in which requirements are compiled and analyzed, followed by the design and modelling phase. After several development cycles with accompanying quality assurance activities and a final software approval process, the software moves into operation. Product maintenance and enhancement projects typically follow release plans and put all necessary cycle steps into more or less short time boxes.

Today, mgm has reached a process maturity that allows us to control all aspects of the project lifecycle through JIRA processes:

• Requirement management
• Change management
• Bug tracking
• Task management
• Release management
• Development cycles and sprint planning
• Software approval process
• Quality assurance and test management

Software Project lifecycle and its reflection within JIRA.

Of these aspects, only the requirement engineering part including business modeling (for example using UML tools etc.) and the design of business and technical concepts is typically done outside of JIRA, but even they can be controlled by JIRA tickets with adequate dedicated workflows.

But why do we spend so much effort to squeeze as many lifecycle steps into JIRA processes as possible? The answer is that we have learned our lessons from experiences: we realized there are many advantages in using only one tool instead of a bunch of different ones, thus reducing system discontinuity.

In summary, one tool for the whole project lifecycle has these advantages:

• Creates more expertise and familiarity.
• Avoids disruptions of process and media during transitions to the next lifecycle step.
• Enables fast feedback between
  • Development & Project management,
  • Customer & Project management,
  • Development & Quality Assurance.
• Facilitates a release oriented approach for development and planning.
• Allows cross-processing reports and progress control.
• Helps when introducing agile tools and methods at the appropriate level of need.

Direct Customer Involvement through JIRA

In addition to the benefits to the project as mentioned above, there is one other advantage of using JIRA as the main tool for all phases of the project: it creates a point of access for direct participation of the customer. But why involve the customer at all?

Quite often project managers and suppliers keep away insight into the detailed project work and progress from the customer and just provide him with status reports on a regular basis instead. But our experiences showed that in software maintenance and enhancement projects – especially in large projects – there is less overhead and more benefits for everybody involved if customers aren’t locked out of the JIRA processes.

For example, if the project lifecycle is well defined and well known and all participants know their parts and responsibilities within JIRA, the process runs very smooth. And as all communications can be conducted within this one tool (JIRA) there will be no more bypassed communications, activities and results that have to be painstakingl translated into the necessary JIRA processes and information snippets (and no information should be kept out of JIRA anyway!). Thus, one more cause of problems is be eliminated!

Avoid Creeping Requirements through Process Transparency enabled by JIRA

Furthermore, the customer gets a completely different attachment to ‘his’ project if he is able to see all ongoing activities and if he can generate an actual project state by himself. We have realized that the customer feels much more satisfied and more responsible for his assigned lifecycle parts once he experiences the whole power of process transparency. He also becomes more sensitive for interference factors and usually starts to avoid ‘creeping requirements’ once he realizes the consequences of adding just another “simple” request (but sometimes the exception proves the rules).

Having only one tool to manage and view all processes has the advantage that the customer can concentrate on acquiring only one set of necessary skills. In addition we support the customers in getting all the necessary information by preparing views for his tasks, responsibilities and status reports by creating all needed JIRA filters and dashboards for him. These will then be shared with everyone who needs them on the customer side (or within the project team) – so one set of preparations can be used by any amount of users to get the same results. Another great help are filter subscriptions especially to track and manage time critical process steps. These tend to make the customer’s start in JIRA very smooth and comfortable.

Nevertheless, the customer has to be willing to take an active part in JIRA. If the customer isn’t very technically skilled and has problems with or an aversion to use software tools at all, we can only recommend to stop your efforts to make him an active part in the JIRA process. Compulsion to use JIRA will have the exact opposite effect and creates feelings of unease and dissatisfaction. In such a case stop your efforts to convince him to use JIRA and just claim higher fees for your larger project management efforts.

To be continued…

In the next part of this blog series we will continue with the topic of direct involvement of customers as introduced above and give you some practical examples of when, where and how to introduce and include your customer into JIRA. We will show you some proven real-world workflows that include customer interaction and cover appropriate modes of customer participation.

(c) 2012 mgm technology partners. This posting "Successfully Embracing JIRA in the whole Project Lifecycle - JIRA beyond Bug Tracking, Part 1" is part of the mgm technology blog. The author of the posting is Alexander Weiss.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

Modern web applications often load data via AJAX without leaving the original page; some sites even have only one single page left. The idea behind that was pioneered by Outlook Webclient and, more prominently for Web users, Google Mail in order to create a desktop-like experience. Other benefits include less bandwidth used by the client, faster response time and eventually a more interactive application feeling.

After the first enthusiasm about all these new applications, users found the first serious drawbacks hitting them hard. The beloved back button was not working anymore, bookmarks always led to the first and not the current page and links could not be forwarded.

All these problems have been known for a long time. New techniques now offer very elegant solutions for this problem and I will discuss them in this article.

From Stateless Web Sites to Single-Page Web Applications

In the beginning, the web was stateless. This reflects the stateless nature of the HTTP protocol. Most web sites were purely informational and content-driven like e.g. newspapers and similar. New content was requested via navigating URLs and a whole new page with a new URL was shown.

When the first applications like online shops were created, a state or session would have proved immensely useful; otherwise all URLs had to be dynamic. The web was still young and dynamic, so cookies were invented for this and have become immensely popular since then. The request-response cycle was unaffected; the HTTP requests were still stateless.

The situation changed again when more and more Javascript was used. Javascript now had implications in rendering the local page which were only known to the browser. By leveraging the XMLHttpRequest object, it soon became possible to get information from the server and incorporate that into the document object model (DOM) in the browser (AJAX). A client-side action can trigger a request to the server which brought the displayed URL totally out-of-sync with the state of the web application. This is still the case in most web applications today!

Coding the Application State into the URL

Some clever developers have found out that changes in the URL (also via Javascript) will be interpreted as new pages by the browser. This immediately enables the back button. And if done correctly, the URL once again uniquely designates the current page even if that has been initially created by a single request but modified by a multitude of subsequent AJAX requests.

An URL that is in-sync with the web page state does not come for free though. In order to work correctly, some things have to be taken into account:

1. The URL must be changeable without reloading the page.
2. Each and every action that modifies the current page must trigger a URL change.
3. When the AJAX-changed page is opened using the URL (from a bookmark), the URL must be interpreted to “replay” the changes which are recorded in it.
4. When the back button is hit, Javascript must detect this.

For an example, consider the scenario in the figure below, where a bookmarked page “/url#2″ is called from a browser. Only the part before the “#” (i.e. “/url”) is relevant for the server, so this URL is requested. The Javascript code in the browser needs to interpret the local part (i.e. “2″) by itself and determines which actions to take. In our case, this triggers a subsequent AJAX request. Note that the URL in the browser then does not have to be changed again.

Reconstructing the page state from a bookmark.

Another scenario is shown in the following figure. Here, a user hits the back button. The hashchange event fires and the Javascript code in the browser must decide how to get the appropriate content, in this case also via an AJAX request like in the previous figure.

Reconstructing the previous page state when the user presses the back button.

Unique URLs (AJAX Patterns) has some great details about this technique.

Changing the Local Part of the URL

Although it might sound easy, already the first point in the list above turns out to be complicated. Of course, Javascript can modify the URL in the browser using location.href, but the browser will then load the page from the modified URL. This is apparently not the desired result.

Clever programmers soon found a solution by only modifying the local part of the URL which then does not trigger a page reload. The local part is defined to be everything after the first hash “#” in the URL. You can directly modify this local part by using the location.hash property. The article “Using Javascript’s Location Object to Work with URLs” is a great introduction.

This technique has become very popular. A lot of sites are using this, the most famous one is probably Twitter. If you open the Tweets page of somebody, you will see that the URL looks like “http://twitter.com/#!/myachinghead”. Effectively, this means that there is only a single web page (apart from about etc.) and everything else is loaded via AJAX.

Links on the page to other “pages” are all intercepted via Javascript and trigger changes in the local part of the URL. For example with Twitter, see what happens if you click on other users but also on the tabs on the tweets page. The URL will change in a well-defined way which is suitable for bookmarks and interpreting all actions which have been performed.

By using this technique, bookmarking is easy. The URL structure of Twitter is fortunately quite simple: the local part is interpreted as the username and the corresponding tweets are loaded asynchronously via AJAX. The same is true for subpages like “Favorites”, “Following” etc.

Detecting Changes in the Local Part of a URL

As the technique has become quite popular, an API for changes on the local part of the URL has been designed and is called onHashChange. You can easily catch this event by using something like:

window.onhashchange = function () { ...  }

Unfortunately, older browsers have no easy interface for listening to hash changes. An alternative is to use setInterval in order to check periodically for a changed hash.

Frameworks like jQuery have plugins for hash, see e.g. jQuery BBQ. These plugins degrade gracefully to older browsers and thus hide the messing around with different APIs from your code etc.

The following figures shows the workflow when changing hashes: clicking an active element must only change the hash (i.e. by serializing the current application state there). Immediately after that, a hashchange event fires and will be detected by the browser. The local part of the URL must then be interpreted and in this case generates an AJAX request to the server. Note that if the element is clicked again and the serialized data is unchanged, no event will be triggered (as the hash has not changed) and the application has been accidentally optimized.

Workflow when changing URL hashes.

Twitter is also an excellent example of what problems can arise using this solution:

• Search for the Twitter page of somebody using Google (or Bing, it doesn’t matter). Click the result and see how the URL changes (the “#!/” is inserted). The reason is that search engines will never index pages which only differ in the local part of the URL. For search engines to work properly, some tricks like sitemap.xml, rewrite rules etc. have to be used. See our blog series about SEO to get more information. Also, Google provides an interesting document “Making AJAX Applications Crawlable”.
• Open the Tweets page of a user, click “about” (the company link) and then use the back button. The result is not what you expect! The back button works fine when jumping between tweets of users, though.

Using a real URL and not only one differing in the local part, offers many exciting possibilities.

Example 1: AJAX-ifying a conventional page-based website

For example, if you convert a normal page-based (i.e. request-response based) website into an AJAX-based one, you can keep your current URL structure but generate the URLs on the client side by using pushState. The associated AJAX call will create a request to the server. You can (and should!) use the same URL as for the whole page and detect on the server side that this is an AJAX request and should render as e.g. JSON (which can be pushed as data in pushState) whereas normal requests will render as complete HTML.

Following this approach, your server-side logic needs no modification, only the rendering must be changed. Bookmarking works still in the usual way as the bookmarked page will be requested as a full HTML page even if the application has become a single page application under the covers. This technique has been used successfully by e.g. Github for quite some time. It is so seamless that you might not even have noticed it.

Example 2: Creating an HTML search interface

In a real-world project, we have created a large-scale search application using Apache Solr as a backend. The API is exposed via Tomcat, which intelligently distributes search requests among the cluster, and we created a REST interface with JSON as data transport.

Query Parameters coded into the local part of the URL (example taken from a custom Solr interface).

Putting an AJAX-enabled Web application on top of that proved to be quite easy. JSON data can be used directly in Javascript. Each search creates an AJAX request to the server and updates the results table. Trouble hit us as soon as the first users were testing the application. They were used to working with the back button which didn’t work as expected. A solution had to be found.

As legacy browsers still had to be supported we chose to implement a URL hash. As jQuery was already in use in the project, the choice was to go with jQuery BBQ. The application was changed to “serialize” the state into the URL. URL changes then raise the hashchange event which in turn trigger the search. We got some nice add-ons for free:

• The first search starts automatically.
• Modifying the URL directly also creates a search.
• Hitting search again without changing the parameters does not change the hash and no event is raised, i.e. no search is performed.
• Reload works out of the box.
• Users can (again) use bookmarks e.g. for wrong results and check back later.
• Users can forward URLs to their colleagues for cross-checks.

After this change has been successfully rolled out, the customers were much happier. The application got a more desktop-like feeling without sacrificing the convenience of the Web’s ubiquitous back button and bookmarks.

Solutions for more complex Web Applications

The discussed solution works fine for small websites or dialog-driven applications. As soon as the application gets more complex, new approaches will prove more efficient.

There are quite a few client-side frameworks which all have their individual strengths. The more popular ones are Backbone.js and Knockout.js. For an interesting discussion about their differences see the discussion “Knockout.js vs Backbone.js (vs ?)”. Depending on the requirements, other paradigms like SproutCore, GWT or Vaadin should also be considered.

Using the HTML5 History API

Instead of messing with URLs and onhashchange events and since the functionality is already so important and will be gaining even more relevance in the future, HTML5 provides a different, more elegant solution. This solution is the History API, a dedicated interface that has been designed solely for the forward/backward navigation in the browser. For this post, mostly two methods are interesting:

1. If a URL has to change, the method history.pushState(url, title, data) can be called. For the parameter: Besides the new URL (url) which replaces the current URL without reloading, a title parameter and a data object can be given. The title is more or less irrelevant, whereas the data parameter can be used to encapsulate the state of the page. If the back button is pressed, this state can be retrieved and used to correctly render the page. Using this state opens therefore possibilities of handling navigation more fine-grained than on a pure URL basis (it can also be viewed as a local session storage which is context-sensitive to each navigational step in the page).

   Interesting reads on this topic are: “location.hash is dead. Long live HTML5 pushState!”, Degradable JavaScript Applications Using HTML5 pushState, and the jQuery Plugins BBQ and History.js with their fallback capabilities to the old onhashchange functionality.

2. The History API also defines a callback for the window element which is called onpopstate. You should define a function and assign it to window.onpopstate, which is then responsible for reconstructing the original page. Of course you have access to the URL of the desired page and to the state object. Both can be used for rendering the page, e.g. via performing AJAX requests or using the pushed variables.

Workflow when using the HTML5 History API.

The two figures above show the workflow when using the HTML5 History API. The upper figure describes forward navigation and includes pushing the new state and the AJAX request to the server. The lower figure shows that the URL changes and a popstate event fires when the back button is pressed. The application can either use local state storage or (in our case) use an AJAX call to update the page.

(c) 2012 mgm technology partners. This posting "Must-Know URL Hash Techniques for AJAX Applications" is part of the mgm technology blog. The author of the posting is Dr. Christian Winkler.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

In the previous part of this blog series, we have seen that it is necessary to generate test data containing extreme or special values (ESVs) which are capable of exerting some pressure onto the software under test. We have also seen that it is desirable to squeeze into a single test-data record as many ESVs as possible, in order to reduce the number of records to be fed into the test system. Running functional tests may take considerable time. Therefore we want to achieve a high compression, i.e. a high density of ESVs per data record.

As we mentioned in part 1 of this blog series, the main obstacle in the way of an automated generation of test data for form-centric applications is the presence of cross-field constraints. Before we get to these, let us look at constraints that can be formulated for individual fields alone.

Constraints on single fields — the easy problem

The kind of constraints that one may formulate for a field depends on the type of the latter. Consider a Boolean field. The field may assume only one of two values, true or false. A simple constraint on such a field is that it may not be empty. Additionally there may be attributes governing the external representation of the field values, such as “wahr” and “falsch”, or “1″ and “0″.

Next, let’s look at a field representing an amount of Euro with Cents. Again a simple constraint on such a field is that it may not be empty. Other constraints may be that the amount may not be negative, or that the amount may not be zero. Invariably, there will be a constraint on the length of the field, which entails a lower and an upper limit for the amount the field can hold. Further attributes may involve the external representation such as the presence/absence of a positive sign in front of the amount, or the appearance of the decimal separator (a comma in Germany, a dot in the UK or the USA). Working with these constraints can be separated into a first phase, where a valid value is generated, followed by a second phase where an appropriate external representation is constructed. Our generator typically produces extreme and special values (ESVs), as explained in the previous part 2 of this blog series.

For a string field, we always have a maximum length constraint, plus a global constraint describing the character set that may be used. All we need is a simple string generator that looks at the string fields one after the other, and for each field generates data, while obeying the constraints that may exist (see e.g. [2]).

In our form-centric applications, we usually encounter a type of constraint on string fields that is a bit harder to cope with than the length and character set constraints mentioned above: it is the regular expression constraint. For instance in order to specify a valid German zip code number we may use a simple regular expression. Likewise, a regular expression may be devised constraining valid phone numbers, e.g. “(\d+[ -])?\d+“. A regular expression for valid e-mail addresses is considerably harder to come by.

Matching a given string using a regular expression is a common task in software engineering. The process of generating a valid (or non-valid) string from a regular expression is the inverse of matching. While there are numerous packages that perform matching, it is a lot more difficult to locate a good string-generator that produces a matching (or non-matching) string from a regular expression, while taking into account the side constraints (see e.g. [8]). Fortunately, the theory of finite automata (FA) comes to the rescue, which allows the conversion of any regular expression into a FA [3]. From there it is only a small step to an operational string generator that allows the production of matching or non-matching strings with relatively little effort. Not surprisingly, we have such a string-generator built into our test-data generator.

Figure 1 below shows the transition graph of a finite-state automaton (FA) for the regular expression “ab|cd”. By traversing paths from the start node (marked with an ‘S’) to one of the end nodes (marked with an ‘E’) one can generate the two strings “ab” or “cd” that match the given regular expression.

Transition graph of the corresponding finite-state automaton (FA) for the regular expression ab|cd.

The nice thing is that any valid data item generated for a field not participating in any cross-field constraints can be combined with any valid data item generated for another such field.

Let us now turn to the harder problem, namely to the fields whose values are restricted by cross-field constraints.

Cross-field constraints — the hard problem

Let us begin by looking at a relatively simple problem, namely, how to fill in two fields in an address block of a form. Assume that the fields are the given name and the surname of a person. The single-field constraints that will exist for each field individually may be dealt with in the way described above. Let us assume that there is a single cross-field constraint linking the given name to the surname, and assume that the constraint reads: “if the given name is present, the surname has also to be present”. Formally we might encode this in the assertion

absent(givenName) or present(surname)

Both fields may be empty, but the given name should not occur without the surname.

The simplest approach to solving this miniature constraint satisfaction problem (CSP) is “trial and error”, or “generate and test”. The task of the generator consists in completely filling in the first two columns of the following table:

Then we use a validator to check whether the constraint is fulfilled.

While this simple strategy is effective and efficient enough for a small number of fields, we quickly run into problems as soon as that number grows. For 10 fields filling in a table like the one above might take as little as 1 microsec, but for 100 fields it might easily take 1000 times the age of the universe. We are witnessing the curse of the combinatorial explosion.

Note that the problem of validating the rows in the truth table only takes polynomial time.

What can be done?

A very successful strategy to cope with the combinatorial explosion is to interleave the data generation with the checking/validation steps. This mixing of steps allows an “early pruning” of the search tree.

Early pruning is a hallmark of practically all CSP-solvers, and, perhaps not surprisingly, CSP-solvers have been used for two decades in the area of automated test data generation (see [4], [5], [7], [8]). Also, the increasing importance of CSPs in main-stream computing is witnessed by the on-going attempt to define a Java-API for CSP-solvers [6].

While early pruning is essential, experience shows that it not sufficient to solve real-world CSPs that occur in the area of automated test data generation. As I mentioned in part 1 of this blog series, it is necessary to partition the CSP into independent components before submitting it to a CSP-solver.

The following figure illustrates this concept. It shows the 32 largest of more than 200 independent connected components. Each little square depicts a field, and a connection between two fields indicates that the two fields occur together in a constraint. This primal constraint graph has been derived from the validation rule base of a large form-centric application developed at mgm tp.

This example for partition the CSP into independent components shows the 32 largest independent connected components of more than 200.

One might think that such a partitioning algorithm would be an integral part of a CSP-solver, but in our experience this is not always the case.

Here is, in brief, how a decomposition of a CSP into independent components can be accomplished:

1. Initially create a component for each field, and put the field into a set belonging to that component.
2. Go through the constraints one by one. Identify all fields of the constraint. Identify all components to which these fields belong and merge their sets.
3. Label the resulting sets.
4. Go through the constraints one by one again. Identify a field in the constraint. Find its component, and label the constraint with the label of that component.

Clearly, each field that does not participate in any cross-field constraint ends up in its own independent component, where test data generation can take place undisturbed by the rest of the problem.

Implementation and data representation

The bulk of our test data generator has been implemented in Java. We greatly benefit from representing the CSP’s variables and constraints in Lisp data structures. The representation of a constraint in a Lisp-data structure with prefix notation looks like this:

(constraint (
		2 "Regel_Zahlen_Vergleich"  "4" "Test_Vordruck"
            "Zahlen_Kontext" "Nicht_Negative_Zahl"
		1 1 1 1 1 1)
 (not (and (= 2
              (+ num-def_positive_zahl_$v1_$u1_$z1
                 num-def_nicht_negative_zahl_$v1_$u1_$z1))
           (>= $t_positive_zahl_$v1_$u1_$z1
               $t_nicht_negative_zahl_$v1_$u1_$z1))))

As a matter of fact, we have never regretted the decision of choosing a Lisp representation, which was taken early on in the lifetime of the project. There are certain transformations of the original CSP that are easy to carry out, when the variables and constraints are encoded as Lisp data. Also intermediate results can easily be written back to a file for visual inspection.

A lot of configuration information is necessary in order to control our Rule-Based Test Data Generator, such as the number of forms that should be filled in, the maximum number of rows in the fields, etc. but we spare you the detail.

Our test-data generator produces data records for each component individually. Due to the independence of components, the records of the components can be freely combined.

Finally, we obtain a table with the generated test-data records. The figure below shows a portion of this table with several solutions of a test-data generation process. Each row represents a solution; its value consists of a mixture of extreme or special values (such as -99999999999.99 or 0.01), and of values that are the result of satisfying the validation constraints.

A portion of the table with the generated test-data records.

Most of the desired extreme and special values (ESVs) are present in at least one of the records (high test coverage). Since most of the time many ESVs are present in a single record, we attain a high compression rate (i.e. many ESVs in few records).

Conclusion

We are at the end of this blog series on the Rule-Based Test Data Generator (R-TDG). You must be left with the impression that the R-TDG is a complex machinery, and, undoubtedly, it is. Be assured, however, it is in operational use at mgm tp since about two years.

The constraints which the R-TDG needs are served to us on a silver tablet by the rule-based system whose main purpose consists in generating the validators for form-centric applications. This approach is radically different from earlier attempts in the area of automated test-data generation, where the constraints are extracted from source code. To the best of our knowledge the R-TDG is the only working automated test-data generator world-wide that can properly take into account cross-field constraints.

References

• [1] Bergmann, V. (2008). Databene Benerator.
• [2] Brüggemann-Klein, A. (1992). “Regular expressions into finite automata.” Lecture Notes in Computer Science 583: 87-98.
• [3] DeMillo, R. A. and A. J. Offutt (1991). “Constraint-Based Automatic Test Data Generation.” IEEE Transactions on Software Engineering 17(9): 900-910.
• [4] Feldman, J. (2011). JSR 331: Constraint Programming API, Java Community Process. 2011
• [5] Gotlieb, A., B. Botella, et al. (1998). “Automatic test data generation using constraint solving techniques.” ACM SIGSOFT Software Engineering Notes 23 (2): 53–62.
• [6] Ince, D. C. (1987). “The Automatic Generation of Test Data.” Computer Journal 30 (1): 63-69.
• [7] msdn (2010). “The Regular Expression Generator”, Visual Studio Team System 2008 Database Developer Center, Microsoft Corporation.
• [8] Zhan, Y. (2002). Constraint Solving in Test-Data Generation. Lecture Notes in Computer Science 2470. P. V. Hentenryck. Berlin-Heidelberg: 770-771.

(c) 2012 mgm technology partners. This posting "Effective and Efficient Techniques for a Rule-Based Test-Data Generator - Test Data Generation for Form-Centric Applications, Part 3" is part of the mgm technology blog. The author of the posting is Dr. Hans-Martin Adorf.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

There are many commercial providers for addresses, however not all of them provide a world-wide address base and the Kickz shop is delivering world-wide. It turned out that the best value for money would be to use Google’s Geocoding web service. Geocoding is the process of converting addresses into geographic coordinates, i.e. latitude and longitude.

However, as the response from Google includes not only the longitude/latitude data but also further address suggestions, this service can also be easily used to present refined/suggested addresses to the user. And that was exactly what we needed. So our basic idea of the service usage was to request exact coordinates and refined addresses (suggestions) for a given address. The coordinates were then used to show the location on the map.

The Geocoding web service is part of the Google Maps API and is offered both as free and as commercial. The basic service is free for 2500 requests per day and does not support HTTPS requests. As Kickz requires HTTPS and more requests per day, we decided to go for the commercial service, which allows of up to 100,000 requests per day and provides HTTPS access with signed web service requests.

Using the Geolocation Web Service

Google’s Geocoding web service is quite simple to use. All you need to do is send a simple HTTP request to the specified address and you will receive a response in either XML or JSON format. The response contains zero, one or even several addresses, depending on the precision of the input. It can be empty if Google cannot locate your address — but based on our experience, this rarely happens.

A JSON response is the obvious choice for JavaScript-based validation on the client-side using (AJAX). XML is more suitable for classic server-side validation. For the Kickz project, we chose the XML format. We validate using the Google web service all over the application, where the user enters a address, i.e. during customer registration and checkout. In these scenarios, the request from the browser is first sent to our server-side application, which calls the Google web service with a certificate-signed request. The request for the input “Frankfurstein ring 105a,München, de, 80000″ looks this:

https://maps-api-ssl.google.com/maps/api/geocode/xml?address=Frankfurstein+ring+105a,M%C3%BCnchen,de, 80000,&sensor=false&client=gme-kickzag&signature=VF930KLrbu98sKKLqIjn4adIoTs=

Google returns the following XML document as a response:

<GeocodeResponse>
    <status>OK</status>
    <result>
        <type>street_address</type>
        <formatted_address>Frankfurter Ring 105, 80807 Munich, Germany</formatted_address>
        <address_component>
            <long_name>105</long_name>
            <short_name>105</short_name>
            <type>street_number</type>
        </address_component>
        <address_component>
            <long_name>Frankfurter Ring</long_name>
            <short_name>Frankfurter Ring</short_name>
            <type>route</type>
        </address_component>
        <address_component>
            <long_name>M&#252nchen</long_name>
            <short_name>M&#252nchen</short_name>
            <type>sublocality</type>
            <type>political</type>
        </address_component>
        <address_component>
            <long_name>Munich</long_name>
            <short_name>Munich</short_name>
            <type>locality</type>
            <type>political</type>
        </address_component>
        <address_component>
            <long_name>Munich</long_name>
            <short_name>M</short_name>
            <type>administrative_area_level_2</type>
            <type>political</type>
        </address_component>
        <address_component>
            <long_name>Bayern</long_name>
            <short_name>BY</short_name>
            <type>administrative_area_level_1</type>
            <type>political</type>
        </address_component>
        <address_component>
            <long_name>Germany</long_name>
            <short_name>DE</short_name>
            <type>country</type>
            <type>political</type>
        </address_component>
        <address_component>
            <long_name>80807</long_name>
            <short_name>80807</short_name>
            <type>postal_code</type>
        </address_component>
        <geometry>
            <location>
                <lat>48.1883675</lat>
                <lng>11.5857053</lng>
            </location>
            <location_type>ROOFTOP</location_type>
            <viewport>
                <southwest>
                    <lat>48.1852199</lat>
                    <lng>11.5825577</lng>
                </southwest>
                <northeast>
                    <lat>48.1915151</lat>
                    <lng>11.5888529</lng>
                </northeast>
            </viewport>
        </geometry>
        <partial_match>true</partial_match>
    </result>
</GeocodeResponse>

The XML response is deserialized to Java objects using JAXB 2. After the response is processed, the user is given some options to refine the original address entered in the browser.

It’s convenient that all responses from the Google web service are localized. In the screenshots below, you see the same validation done in the German shop (left) and in the international shop (right). The only difference is the language parameter sent with the web service request and set in the HTTP request header “Accept-Language”. To visually improve validation, we also use static Google maps URL to display the address to the user graphically.

The Geolocation web service is localized: Address data returned for the german shop (left) and for the international shop (right). Click to enlarge.

Conclusion

Google’s Geocoding web service turned out to be an effective tool for the validation of the user addresses in our Kickz online shop. Furthermore, this service can be used free of charge for small projects or sites with lower traffic. In Kickz though, we are using the commercial version of the web service, since the traffic is higher and we need the HTTPS service URL.

(c) 2012 mgm technology partners. This posting "How Google’s Geocoding solves Address Validation" is part of the mgm technology blog. The author of the posting is Petr Skokan.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

The whole conference seemed to have a mantra: A-S-T for Abstract Syntax Tree transformations. A language should have access to its own abstract syntax, as McCarthy claimed back in the days of LISP – Groovy has its access via AST transformations.

AST Transformations

Hamlet D’Arcy, an experienced engineer with Canoo held an in depth workshop on transformations of the abstract syntax tree of Groovy code.

In a nutshell AST transformations are Java annotations allowing to manipulate Groovy code. Practical use cases are best practices patterns such as Joshua Blochs ‘Effective Java’ implemented once in terms of an AST annotation applied to every day Groovy code.

Examples are @Canonical applied to a Groovy class generating correct implementations of Object.hashCode, Object.equals, and Object.toString or @Synchronized, @WithReadLockor @WithWriteLock generating the correct error prone synchronization boiler plate code for synchronization of mutual state access.

Another handy AST annotation is @Log which creates the standard logger and guarding log level check code for log statements.

AST transformation are very powerful but one must not forget that – like AOP – your code will look different in a debugger.

Design by Contract with GContracts

The GContracts project uses AST transformations to implement a feature the Eiffel language has for years: Design by Contract by defining invariants and pre- and post conditions. The infamous example of an integer overflow that caused the spectacular Ariane 5 crash served as the obvious example.

Applying these contract annotations obviously seems like a good idea.

Static Code analysis for Groovy

Java has some mature static code analysis tools such as Findbugs, PMD or the inspections built into the IntelliJ IDE bugging a developer with violations of best coding practices rules. CodeNarc is a tool that offer a similar functionality. It doesn’t seem to be as mature as its senior Java counter parts, but the number of rules is increasing and writing CodeNarc rules is quite easy. Integrating CodeNarc into a Groovy build process and enabling the corresponding IntelliJ plugin should become as mandatory as using Findbugs and friends.

Happy spec’ing with Spock

Spock is a JUnit compliant testing framework allowing to write very expressive test cases. Peter Niederwieser, the creator of Spock uses Groovy’s internal DSL capabilities to provide some very appealing features to write test cases called specifications in the Spock terminology.

A specification follows the given-when-then pattern also known as AAA, arrange-act-assert. JUnit provides the @Before and @Test annotations in combination with a whole bunch of Assert-Methods. Asserts are implict in the “then” section of a spec and the “given” and “then” sections are the arrange and act realizations.

Apart from this basic functionality Spock comes with many more powerful features such as the Power Assert which has been extracted into the standard Groovy GDK, by the way. One of these features is the “where” section allowing to provide test data in a tabular form or to be read from a database in a very convenient way.

A deeper look into Spock is highly recommended. Spock can serve as a replacement for JUnitor TestNG because due to Groovy’s intrinsic Java integration it allows to test any JVM language.

Languages are either concurrent or obsolete

A very good argument to persuade Java purist to consider Groovy as a language is GPars. Vaclav Pech, the project lead of the GPars project gave a very good introduction to GPars by demonstrating its power by coding concrete examples instead of flipping slides.

GPars can be considered as an abstraction layer on top of the Java concurrency packages. It is a library that enhances Groovy in a way that a programmer can safely use all prominent concurrency patterns such as thread pools, actors or data flow variables. Vaclav demonstrated with a simple piece of code sequentially processing a number of images that the eight cores of his machine are not used as they are supposed to. By simply wrapping the same code with a “withPool” clause the process monitor showed all cores at work. The interesting fact was that no threads had to be created manually – the GPars construct figured out how to distribute the work load.

Data flow variables take care of the synchronization of concurrently executed tasks. There are no explicit threads and locks required any longer. Anyone who has to implement concurrent computations should consider using GPars instead of having to code the corresponding concurrency patterns using the lower level Java concurrency packages.

Summary

Gr8Conf was worth a visit. Compared to the heavy weight JCP the evolution of the Java language is tied to, the growth of the Groovy ecosystem is breath taking. Some of the projects presented on this conference should be evaluated for existing and upcoming mgm projects.

(c) 2012 mgm technology partners. This posting "GR8Conf: A gr8t Conference on Groovy based Technologies" is part of the mgm technology blog. The author of the posting is Marcus Olk.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

Spring Roo’s mission is to “fundamentally and sustainably improve Java developer productivity without compromising engineering integrity or flexibility“. It follows the Convention over Configuration principle, and provides rapid application development for Java EE applications.

Spring Roo itself isn’t a framework; it’s more like a software tool for generating Java EE web applications with CRUD functionality. This means the user can create, read/find/list, update and delete entities. Spring Roo offers several technology options, like various providers for the JPA-based persistence (e.g. Hibernate and EclipseLink), Spring MVC controllers and a frontend based on either JSPs, JSF 2.0 (coming in Spring Roo 1.2), Google Web Toolkit (GWT) or Flex. If you want to change your persistence provider, you could easily do this; if you want to add a JSON output (or whatever else is needed) to the controllers, you could add it with only small effort. And everything integrated into the project has a reasonable “default” configuration, so that very little configuration overhead is needed.

The toolset of Spring Roo can generate code and integrate a lot of standard modules into a Java EE project, with very little configuration overhead, leaving you always the choice to modify everything you need. So with some easy commands you can add technologies like JMS for messaging, JSON for REST, Solr for full-text search etc.

How does Spring Roo work?

The heart of Spring Roo is the Roo Shell with a really great help system, which is self-explaining and easy to use. From here you could setup your JPA provider, create entity objects etc.

There are lots of examples where you can quickly try out Spring Roo. In those, Spring Roo creates a complete project for you, including a Jetty server and a in memory database if needed.

We will use our own short example to explain a few features along with the article.

A Short Example

We are going to build a simple address database, where a person can have multiple addresses. Setting this up is a really easy process:

1. First install Roo from the Spring Roo homepage and add the bin folder to you path.
2. Then create an empty folder and start Roo with roo.sh (or roo.bat) from within this folder.
3. Next, follow the following commands to setup the project. You can type “hint” anytime, showing a smart help system which suggests you the next steps.

project --topLevelPackage de.mgm.roo.sample

After this command an initial project will be created.

Here we will choose Hibernate for persistence and the Hypersonic in-Memory Database as specific database:

persistence setup --provider HIBERNATE --database HYPERSONIC_IN_MEMORY

A Spring Roo project is usually created and modified in the Roo Shell.

There are lots of ORM technology options and databases to choose from, nearly all you would find in a normal environment.

Project structure generated by Roo.

This is our first entity, the Address:

entity --class ~.domain.Address --testAutomatically
field string --fieldName street --notNull --sizeMin 2
field string --fieldName city --notNull --sizeMin 2
field number --type int --fieldName postcode --notNull

This is our second entity, the Person, which can have several addresses:

entity --class ~.domain.Person --testAutomatically
field string --fieldName name --notNull --sizeMin 2
field string --fieldName lastname --notNull --sizeMin 2
field set --fieldName addresses --type ~.domain.Address

Pressing Tab during the commands shows you all available options.

The new entity classes are reflected in the Eclipse IDE.

After the following commands, we have a fully working web application, that can be imported into eclipse (other IDEs are available as well):

perform eclipse
controller all --package ~.web

The Web tier and resources are created through the Roo Shell.

This application can be started easily with the command “mvn jetty:run” and you have the following web application:

The out-of-box generated Web application.

The core modules are included and managed via the OSGI implementation Felix, so the Roo-Shell also supports all OSGi Commands, e.g. to load and unload modules. You should take a look at the module list.

But how does Spring Roo manage the generation and even modifications?
Try to add a field to your entity object and Spring Roo will update everything for you.

How does Spring Roo work technically? Spring Roo uses AspectJ mixins called inter-type declarations in order to achieve separation of concerns, because the Spring Roo code is in a different compilation unit, as the code the user writes. So Spring Roo can modify it files as it needs them, without getting in conflict with the user.

Spring Roo as a Rapid Prototype tool

As the headline suggests, we used Spring Roo with a customer for Rapid Prototyping. Because the specific project had a very short timeline, there was not much time for a long requirement phase, so we tried rapid prototyping instead. So the customer got a very early working base, from where we could add and remove requirements on interacting cycles. Another problem were the changing requirements.

The general concept of the application was very clear, but the details like fields and entities changed. This is where rapid prototyping did really help. Rapid prototyping allowed us to show a lot of changes to the customers early in our development cycle so that we began getting the feedback that will be crucial to the success of the project. Not only the showing was successful, also the “usage” by the customer did help.

The requirement phase was followed by an early development stage:

• At first we developed the data model/entity model with the customer and transferred it to Spring Roo, then added
• the web frontend generated and maintained by Spring Roo, and
• the layout and CSS styles for the customer to see.
• We then included the existing databases with DBRE (Spring Roo Database Reverse Engineering),
• showed the system to the customer and added/removed entities and attributes as needed, and
• finally added security.

Adding and Removing Attributes and Entities

There are different ways to add and remove entities in the existing project. The first would be using the Roo shell to add and remove fields of an object. This is the most obvious way. But another, much better way is changing the classes directly, i.e. in the Java code.

Lets take a look at a domain object, generated by Spring Roo. The Address entity looks pretty clean and straight forward. The getters and setters are managed by Roo, but you can easily overwrite and modify them as you wish.

So lets add an field to our address. We noticed that the country is missing, so lets add this! I just added the following code:

    @NotNull
    @Size(min = 2)
    private String country;

As soon as I save the domain object, Roo updates the project and the new field appears in the web application. Here’s an excerpt from the Roo shell output:

Updated SRC_MAIN_WEBAPP/WEB-INF/views/addresses/list.jspx
Updated SRC_MAIN_WEBAPP/WEB-INF/views/addresses/show.jspx
Updated SRC_MAIN_WEBAPP/WEB-INF/views/addresses/create.jspx
Updated SRC_MAIN_WEBAPP/WEB-INF/views/addresses/update.jspx
Updated SRC_MAIN_WEBAPP/WEB-INF/i18n/application.properties
Updated SRC_MAIN_JAVA/de/mgm/roo/sample/domain/Address_Roo_ToString.aj
Updated SRC_TEST_JAVA/de/mgm/roo/sample/domain/AddressDataOnDemand_Roo_DataOnDemand.aj
Updated SRC_MAIN_JAVA/de/mgm/roo/sample/domain/Address_Roo_JavaBean.aj
Updated SRC_MAIN_JAVA/de/mgm/roo/sample/web/ApplicationConversionServiceFactoryBean_Roo_ConversionService.aj

You can keep Roo running when you modifying a class outside of Roo – it should detect changes, or it will do it on next restart.

You can use the same way to create and delete entities – just create or delete the classes as you need. Tip: A good practice is changing the entities and attributes, but all other stuff should be changed through the Roo shell, like adding and removing security etc.

Removing Spring Roo from Project

After this early development phase, we decided to remove Spring Roo, resulting in a regular Spring based web application, and continued with “normal” Java Development. This way is intended by the Spring Roo authors; there is a whole chapter of removing Spring Roo from a project.

From this point on, we refactored the entity layer (just to get more structure in the packaging etc) and got a quite nice Java EE App as a working base.

Spring vs. Grails vs. Ruby on Rails

Here I want to explain shortly why we chose Roo over Grails and Ruby on Rails. The quickest answer would be, because there was a requirement to develop in Java. But I want the show a little bit about the advantages and disadvantages of this three comparable systems.

Ruby on Rails is a very powerful platform with a large community, but you would have to learn Ruby and you would have to understand Rails as a platform. You couldn’t use existing modules (in Java) easily.

Grails has also a very good community, uses standard Java technologies like Hibernate and runs on the JVM. Here, the only effort would be learning the Groovy language. But on the downside, Groovy is always a little bit slower than native Java. Here the main aspect against Groovy was again the language.

Spring Roo has a very small community, but help is available. The advantages here were that Spring Roo creates “regular” Spring Projects as soon as you remove Spring Roo itself.

Another interesting option is AppFuse, which focusses more on the initial scaffold of a new project (a bit like Maven’s archetypes). We had no chance to investigate it, but consider the Discussion “Appfuse vs Roo – what would you use”.

Conclusion

Spring Roo is definitively useable as a tool for Rapid Prototyping, i.e. to quickly and easily create a first usable application. But you should face the fact that the current version 1.1.2 still has a lot of missing features (like composite primary keys) and bugs, where you suddenly need a solid understanding of AspectJ and databases. This was the main reason why we decided, after the requirements phase, not to continue working with Spring Roo.

The real great upside was, that we got a complete application up in running in a few days with a clean codebase where we could easy start from developing all the details needed in the application. The CRUD web pages created by Spring Roo and the RESTful architecture are a great help extending the application clean and fast.

(c) 2012 mgm technology partners. This posting "Rapid Prototyping with Spring Roo" is part of the mgm technology blog. The author of the posting is Christian Belka.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

Remember that one of the benefits of using LiquiBase is that it allows you to keep all the Changes to the database(s) in one place, the Changelog. But what happens if we are dealing with a very large and complex physical data model with frequent changes? Won’t the Changelog grow to such a size that it is completely unmanageable? This is a valid question to which the simple answer is that although LiquiBase will always have only one Master-Changelog per database, this does not mean that all Changesets have to be in that one Changelog file.

Partitioning the Changelog

In fact, it is very advisable to create smaller Sub-Changelogs and use the include script statement to link these into the Master-Changelog:

<databaseChangeLog ...>
   <include file="r1.0/r1.0_changelog.xml"/>

   <include file="r2.0/r1.0_changelog.xml"/>
   ...
</databaseChangeLog>

It is then quite easy and comfortable to create a hierarchically ordered directory of different Sub-Changelogs which allow to bundle all changes that concern a certain release or concept like triggers etc. into separate Sub-Changelogs. If the Sub-Changelogs are already referenced in the Master-Changelog, you only have to update the smaller Sub-Changelogs with any new Changesets while you will always just run the MasterChangelog.

This mechanism allows us to organize the changes for different releases into separate folders to create an easy to understand structure which makes it quite simple to add any new Changesets at the correct space.

Stored Procedures

We have found it particularly helpful to keep stored procedures in separate Sub-Changelog files, since they tend to get rather big. We use a special approach to achieve this (which differs from the way other Changes are handled): for these stored procedures, we only keep a single version of the procedure in the Sub-Changelog file – instead of appending a new Changeset each time the stored procedure changes. To ensure that the changed Changesets are run again, each procedures is wrapped into a single <changeSet> to which we add the attribute runOnChange="true" that forces LiquiBase to check if the Changeset was changed instead of just checking if it was run once before so it can be skipped. This exception to the rule makes it a lot easier to manage and spot changes to stored procedures and as all our Changelogs are stored and versioned in Subversion there is no danger to lose data.

An example how we organize our Changes into a hierarchy of Sub-Changelogs.

Using Relative Paths

We consider it a good practice to use relative paths to ensure consistency and to avoid problems caused by mistakenly reapplied Changesets. This is especially important if a structure of Master and Sub-Changelogs and/or a lot of include statements are used.

A short recap: The “DATABASECHANGELOG” table records each Changeset that has successfully applied by author and id tag and by filename and uses this information to check if a given Changeset has already been applied.

If absolute paths are used they will be recorded as part of the filename and should (which is quite likely) the next application of the Master Changelog be triggered from a different path (e.g. a different developer’s machine, as part of a build process, etc.) the absolute path will no longer be the same and LiquiBase will not recognize the Changeset.

To avoid running into this kind of problems, some of my colleagues force LiquiBase to record the relative path names in the “DATABASECHANGELOG” table by adding the Master Changelog directory to the CLASSPATH so that Liquibase is launched to execute the Master Changelog only by filename (without path). All the include statements are configured to also use only paths relative to the Master Changelog to ensure that no absolute paths are written to the “DATABASECHANGELOG” table. This has so far allowed us to avoid these problems.

For another discussion covering quite broadly why and how to use relative paths with LiquiBase, see the posting “Why does the change log contain the file name?” in the LiquiBase forum.

Custom SQL — Blessing and curse

In the former Installments we mentioned that PowerArchitect does not offer to reverse or forward engineer concepts like check constraints, triggers or stored procedures but you may have noticed that we just explained how we use separate Changelogs for stored procedures. So it is clear that there is another way to create Changesets for these. The concept is called Custom SQL and it enables us to add any SQL statement as a valid Changeset to LiquiBase. We just enclose the respective statement with the tags:

<changeSet author='xxxx' id='x'>
  <sql >
    ADD YOUR SQL STATEMENT HERE
  </sql>
</changeSet>

If it is a stored Procedure the tags will be:

<changeSet author='xxxx' id='x' runOnChange="true">
  <sql >
    ADD YOUR SQL STATEMENT HERE
  </sql>
</changeSet>

Sometimes it is helpful to add rollback information after the </sql> tag using the <rollback> tag, to avoid problems should the custom SQL somehow fail.

Some advanced concepts are dependent on the DBMS (as they use SQL dialects that slightly vary). If it is obvious that the Changelog will have to be applied to different flavors of SQL we use the Include mechanism and create separate Folders to store all the custom sql Changesets for a given flavor of SQL (depends on DBMS). We might for example have one subdirectory for Oracle Custom SQL and one for PostgreSQL.

Using Preconditions in the Master-Changelog we can then control which set of Sub-Changelogs will be run for a given DBMS. Once this is implemented correctly in the Master-Changelog you can safely update your database specific Changelogs in the respective folders.

Applying Changes

One of the major selling points to use LiquiBase for us is that it makes applying Data model Changes very quick, easy and safe. LiquiBase can be set up to be run using Ant, Maven, Spring etc. or by using the Commandline option. As we currently use Maven we can thus integrate LiquiBase into our build management to deal with database changes. So applying LiquiBase is just a click or two.

LiquiBase will keep track of which changes have already been applied (though you can force it to reapply all Changes by using the attributes “runAlways” or “runOnChange”). It does this by means of an internal table “DatabaseChangeHistory”, which registers every applied Changeset by filename, id and author tags. When applying a Changelog to a database, the LiquiBase process scans the “DatabaseChangeHistory” table and skips all Changesets that have already been applied. Once a Changeset has been applied the filename author and id will be logged in the table. This is a major time saver in the long run.

But what if a customer insists on getting a SQL file that his DBA can apply? Though we do not recommend it because it will be potentially less safe, more time consuming and give less control we can use LiquiBase for the creation of traditional SQL scripts as well. And we can easily switch to LiquiBase should the customer at a later stage decide that he wants it.

Special DBMS specific tips and warnings

Let me finish with a few experiences we have had using LiquiBase with various database management systems.

I will start with MySQL, the popular free DBMS, as it has some quirks that set it apart from most other DBMS.

In MySQL it is necessary to select a storage engine like InnoDB when creating new tables. This poses a problem in LiquiBase that we currently solve by a workaround suggested by LiquiBase. We put the create table statement into a separate Changeset which we then adapt using the tag <modifySql>.

For example:

<changeSet id="55" author="xxx">
  <createTable tableName="person">
    <column name="id" type="bigint"/>
    <column name="firstname" type="varchar(255)"/>
    <column name="lastname" type="varchar(255)"/>
  </createTable>
  <modifySql>
     <replace replace="bigint" with="long"/>
  </modifySql>
  <modifySql dbms="mysql">
     <append value=" engine innodb"/>
  </modifySql>
</changeSet>

This is not perfect but it works.

Another MySQL specific quirk is that opposed to most other DBMS MySQL expects the definition of Constraints to be part of the column definition like:

<column name="ID" type="BIGINT"
        remarks="Primary key" autoIncrement="true">
  <constraints nullable="false" primaryKey="true"
               primaryKeyName="KATEGORIE_pk" />
</column>

Instead of adding the key afterwards in a statement as it is generated by the PowerArchitect export:

<addPrimaryKey tableName="KATEGORIE"
               constraintName="KATEGORIE_pk"
               columnNames="ID"/>

To correct this manually is quite tedious but we have found it is possible to use an XSLT transformation to deal with this problem which takes much of the pain out of this.

Another inconvenience we have encountered with MySQL is that it enforces a limit of a maximum of 64 characters on remarks on tables or columns. Currently there is no workaround for this. While this is not a problem introduced by either Power Architect or LiquiBase, my colleague Thomas Kellerer has delivered a solution that can be run with the “Criticize Model” context menu item in Power Architect to check if the comment is compatible with MySQL (See image below). Maybe not a big deal for some but still a little handy solution.

Use the 'Criticize Mode' context menu item in Power Architect to check if the comment is compatible with MySQL.

On Oracle we have discovered that we can run into a situation were we have to use preconditions to ensure the correct username and this database is used when dealing with multiple databases as in Oracle DB user and schema are the same thing. We use Changeset modifications like the following to ensure that we do not run into problems

<changeSet author="xxx" id="5">
  <preConditions onFail="MARK_RAN">
    <or>
      <runningAs username="SHOP" />              <!-- PROD   -->
      <runningAs username="SHOPBUGFIXONLINE" />  <!-- BUGFIX -->
      <runningAs username="SHOPTESTONLINE" />    <!-- TEST   -->
    </or>
  </preConditions>

We also use Liquibase with PostgreSQL but have not yet ran into any major problems there.

Managing Data with LiquiBase?

A question that keeps coming up is: “Can I manage my (test) data with LiquiBase as well?”

The answer is yes and no. While it is often no problem to use LiquiBase to deal with small amounts of relatively static test data (e.g. to fill a smaller test database for the first time) using LiquiBase to maintain large amounts of quick changing data is often not so convenient. We are currently still investigating the best ways to deal with this problem and we will keep you informed about the outcome in another blog post.

Conclusion

Now that we have shared our experiences with you let us conclude to sum it up to answer the question: Why do we do it? Why do we use tools instead of just writing SQL scripts from logical data models?

One answer is that we believe and our experiences support it that well documented data models save us time and money. Many applications evolve over time and our tool-chain supports the adaptation of the physical data model and it simplifies the application of the changes to databases. It also helps us in the test and maintenance phases to create databases in different stages and selective rollbacks of many changes with great precision and little effort. And last but not least it is much easier to test a given data model on different DBMS should a customer want to either change his DBMS or insist on an unsuitable one – nothing is more convincing than a demonstration.

(c) 2012 mgm technology partners. This posting "Getting the most out of LiquiBase - Tool-supported Data Modeling and Change Management, Part 3" is part of the mgm technology blog. The author of the posting is Ursula Bauer.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

Searching is a frequent activity on the web and one of the most important features of every online shop. With a powerful searching facility, customers can easily specify what they want and then find a product they look for. This includes all kinds of standard catalog search (i.e. category) and full-text search.

Traditional web shop solutions implement a catalog search with relational database queries. Full-text search tasks can then be implemented either by native features of the databases, or by using external indexing engines. Lucene is an open-source Java indexing engine library. It is used by a huge number of web sites and applications. Internally, Lucene stores data in a flat storage structure, where each record consists of several fields as key/value pairs. In Lucene terms, such a record is called a Document.

We have had very good experience with Lucene in previous projects and decided to extend its usage beyond standard full-text search. Here are a couple of non-standard usages of Lucene that we’ve used in our projects and that you might find interesting too.

Criteria search with Lucene

Our recent shop project www.kickz.com had high performance requirements for a multi-criteria search. Specifically, we required <100 ms for >200 concurrent queries on average. The application itself is written in Java and can be clustered easily.

We are using RAMDirectory to load an index entirely into RAM (see also later on in the blog). This is especially convenient since we have a pre-built index living on disk and can easily slurp the whole thing into RAM for faster searching. In such a setup, a search index is only limited by the amount of heap memory available to the JVM process. A mid-sized shop contains up to hundred thousands of products what, which should not reach usual memory limits. In our case, the index size is about 15 MByte per language, summing up to 60 MByte in total. However, if the number of indexed items is very high (resulting in big indexes) or a distributed search is needed then Solr could be considered as an alternative. See also our blog series about using Solr/Lucene with Hadoop.

Mapping Relational Data to the Lucene Index

Our first challenge was to represent the relations in the Lucene index, i.e. searchable data that comes from embedded or associated entities (*-to-many associations). For example, when searching products, one might restrict the search on specific categories, such as shoes. In relational databases, a SQL query on an entity and its associations can easily join several tables by primary/foreign keys. Such a join cannot be done with Lucene’s document model in a straight-forward way.

However, there is a workaround at the cost of storing redundant data: a document might contain more fields with the same key, i.e. key/value = “category/shoes” and “category/basketball-shoes”. This can be used for *-to-many associations where joins are represented by its business key (i.e. unique category key from the ERP system) or primary key from the database. Then it is necessary to collect all required data during index creation and build index documents from them. In our case, this means the transformation of the domain model (left side) into Lucene’s flat document structure (right side):

Extracting information from an domain entity and its associated entities into a single Lucene document.

When searching for a product with a given relation, i.e. for products of a specific category, then the search will be performed for all documents that contain the relation’s key with a given relation value, i.e. documents containing the key “category” with the value “basketball-shoes”.

Please note, that this approach is not suitable in cases when the associated entity in a “*-to-many” relation is changed frequently. Because in this case, either the complete index or at least all the documents containing the changed data would have to be updated. This might be time consuming operation. However, for our shop engine, this concern is not relevant, because data is changed only within the underlying ERP system, and each publication triggers a complete rebuild of the index (see below). Thus, the approach can be used without any limitations.

A sidenote: If you’re using JPA with Hibernate, you might want to have a look at Hibernate Search. It uses special annotations and automates the indexing and extracting of data from the entities, and even supports the indexing of embedded and associated JPA entities. For one-to-many and many-to-many relations, it uses the same “trick” as mentioned above.

Lucene as a NoSQL Database

Our first versions of the search returned only IDs of database records that met the given condition. The records were then loaded from the database. This approach, combined with a properly configured Hibernate second-level query cache, perfectly fits to most usages.

However, the performance can still be improved. For example, the search result page does not display all product information — only brand, product name, price is typically displayed, but the detailed product description is not, see example. Lucene allows to store “result values” in the index, which are not processed, i.e. kept untokenized, which can be retrieved for displaying the result. For example like this:

Field field = new Field("brand", brand.getName(),
                        Field.Store.YES,
                        Field.Index.UN_TOKENIZED);
document.add(field);

And after performing the search, the value can be easily retrieved from the query result:

String brand = document.get("brand");

Like brand, we also store product name and price in the index, so that all necessary data to build the search result page can be retrieved from Lucene alone. Specifically, there is no need afterwards to make an extra database query to load them. A similar effect can be achieved by a second level cache containing all products.

Concurrent Rebuilding and Searching with Index Switching

The index creation is running in an external process that is triggered after data from the ERP system has been imported into the SQL database. This is typically done once a day, or at maximum several times a day. The imported data is read-only in the SQL databse. Once the import is finshed, all available products are indexed during this process and thus, the index is always completely rebuilt.

Unfortunately, Lucene has a restriction in that a index cannot be updated while it is open for reading by another process. But this issue can be easily solved by working with two indexes: one for searching, the other for updating. While one index is used by the shop for searching, the other might be recreated in the background. If the new index is ready, the clustered shop application is notified via JMS and opens the index that has just been recreated. The original search index stays untouched and is thus available for the next import and index re-creation.

The principle of Index Switching allow concurrent rebuilding and searching of different JVM processes.

Searching in RAM

The index is normally being stored in the filesystem and performs all operations there. If there is enough JVM heap space, then the index file can be loaded into memory using RAMDirectory and all search operations then do not require any disk access. This will increase the performance, if the file system does not keep recently accessed files in shared memory:

if (ramSearch) {
	searcher = new IndexSearcher(new RAMDirectory(directory));
} else {
    searcher = new IndexSearcher(directory);

Other high-performance options are discussed in chapter 9.4 of the latest edition of the Lucene in Action book.

(c) 2012 mgm technology partners. This posting "Non-Standard Ways of using Lucene" is part of the mgm technology blog. The author of the posting is Michal Perlik.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

Unit tests are written by the developer simultaneously with the production code. If the API or the functionality of a class in the production code changes, then the corresponding unit test class has to be adapted accordingly, otherwise the build executing the unit test will fail. This is no principal problem, since both should be changed by the same developer simultaneously and should be committed only after they are consistent.

How is the situation with functional tests? A functional test with its test data is coupled to the business requirement and the part of the product fulfilling this business requirement. So if the product changes, then the functional tests must change as well.

If the functional tests run in the same build as the product, every change to the product will cause the build process to fail, unless the developer and tester change their code simultaneously. This would require a strong synchronization between developer and tester, which is usually not feasible.

The result is that functional tests almost always fail if they are integrated into the build environment. This calls at least for different builds for the product (code plus unit tests) and the functional tests. But I will go further: I recommend to not run the functional tests in the build environment at all! Because the build environment not only does not facilitate anything, but on the contrary adds a lot of extra complexity to the test environment.

The following diagram compares the two approaches of running functional tests, i.e. in a build environment and in a separate test environment against an independent validation system (or staging system).

System setup for unit tests within a build system and functional tests against a validation system. On the left side: Unit tests within a build system are triggered by the build system (Jenkins/Hudson, Luntbuild, Ant, Maven etc.), and executed locally. The application runs on the build server ideally setup by the build system and unit test framework (JUnit, TestNG) before the execution of the unit tests. On the right side: Functional Tests are executed by test drivers for functional tests on dedicated test clients against a validation system, which is set up and configured like the live system.

How to run GUI-/Browser-based Tests

If you run functional tests in the build environment, then they must be executable in a batch-run, driven by the build framework. Specifically, on the build system, the build framework, e.g. Jenkins (formerly known as Hudson), Luntbuild etc., starts a build script (Maven, Ant), which starts a test framework (JUnit, TestNG), which executes the test case. Of course, this is technically possible. There are functional test tools and build environments, which have such capabilities, but they bring in extra complexity and entail extra development and significant configuration effort (often in the range of 1-2 weeks). But the restrictions are high.

It is very hard to develop GUI-related or browser-based functional tests in an environment without visible screen-output. How do you debug test failures, if they occur only within the build environment? How do you create keyboard and mouse events (i.e. entering text in an applet within a browser)?

If the functional tests are executed on dedicated test clients (often the tester’s PC), then any test and scripting software available for the desktop OS of the tester’s PC can be used. What’s more, the test execution can be easily observed (GUI-Tests) and debugged by the tester, so the test development has less obstacles in its path.

Merits of an Independent Validation System

Another important point is the test of the configuration of the system. If the system under test runs within the build framework, then usually the configuration of the system deviates strongly from the configuration of the live system. An independent validation system can be set up and configured much closer to the live system including hardware, installed software and components and configuration of all parts. Thus, a functional test against the validation system checks a large part of the system configuration as well and finds system specific errors and failures in the deployment of the system.

A test on the validation system almost satisfies the well-known software development paradigm: The work is only done, if the requirement is fulfilled on the live system. Manual reproduction and repetition of failed tests is also easier, if the test is executed from the same client against the same system under test (validation system) with the same product version, the same configuration and the same test data.

If the functionality of the product is accessed by an API instead of a GUI and if the product or parts of the product are already very stable, then it can be useful to put all stable functional tests as regression tests into the build environment.

Stable Builds, or when to Deploy on the Validation System

If a lot of effort is invested in builds with high quality standards, like high coverage of unit tests or code reviews before a commit, then the builds are usually stable enough to be deployed on the validation system as a basis for the execution of functional tests. If however the builds are not very stable, then there should be an additional pre-validation system, which is used to stabilize a build. As soon as a build is deployed on the pre-validation or validation system, functional tests are executed against it.

In order to adapt existing functional tests and to prepare new ones, the testers must be aware of all functional changes and additions that are to be expected in the new build. Depending on the team size, this information exchange can be done in small teams in ad-hoc meetings with the developers or — in larger teams — with the support of detailed feature lists maintained in task management tools like JIRA.

For all failing functional tests, the requirements must be checked (often in collaboration with the developer), to find out whether the test or the product is working incorrectly. After the errors in the functional tests are fixed, the functional tests are repeated. The remaining failing tests should then all be caused by flaws of the product.

A build is considered stable if and only if the functional tests checking the main paths of the business functionality have passed, so that the execution of further tests is feasible and makes sense. In projects with longer development cycles (several months), stable builds should be achieved every one or two weeks and towards the end in even shorter intervals. In projects with short development cycles (several weeks), stable builds should be achieved every one or two days. 10 to 15 stable builds should be achieved per development cycle. Each stable build shows, that the development and test group is in sync and the product is stable. A close collaboration between the functional testers and the developers is essential in such a setup.

Summarizing the argumentation, I emphasize the following points:

• Functional tests are executed on dedicated test clients.
• Functional tests are executed against a dedicated validation system.
• Only stable builds are deployed on the validation system.
• Stable builds should be achieved regularly and in adequate frequency throughout the development cycle.

In a later blog, I will discuss test coverage, when to write and rely on unit tests and when on functional tests and how to derive test coverage. A rough guide for electing the right tools will be given as well.

(c) 2012 mgm technology partners. This posting "Why Functional Tests don’t belong in a Build Environment - A Practitioner's Guide to Successful Software Testing, Part 2" is part of the mgm technology blog. The author of the posting is Dr. Martin Varendorff.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.

First of all, it is important to understand that our project uses the “separate deployment” approach of Orbeon integration. Here, Orbeon is only used as XForms renderer — neither the Orbeon Portlet nor the Formbuilder / Formrunner system was used.

Orbeon XForms supports two modes for a JSR-286 portal integration:

• The Orbeon Portlet uses the Orbeon Form Builder and Orbeon Form Runner, which are Orbeon’s own XForms editing and management toolkits. They allow you, to easily develop your own XForms and XForms screenflows with Orbeon. The catch is that you have to use the Orbeon framework and these toolkits for your whole XForms Application.
• With the “separate deployment” mode, Orbeon is only used as a XForms rendering engine, you have to create and mange the application by yourself. This gives you the ability to create 100% custom applications. As mentioned, this is the mode we chose.

Overview of our Integration Approach

Besides XForms, our existing web application was developed with Spring MVC, therefore the Spring Portlet MVC technology was used for the portlet implementation. The Spring application loads the XForms from a repository corresponding to the required business case. The specific architecture as well as Spring and Spring Portlet MVC were project technology decisions. They are not needed to integrate Orbeon with a portal server.

Orbeon is deployed as a normal web application besides Liferay, but is stripped down (Form Runner/Builder and samples removed). The JavaEE server is JBoss 5.1 with Liferay and Orbeon 3.8 installed on it, and configured with 1,5 GB RAM for the JVM.

Please note that Orbeon, as of October 2010, proclaims that the 3.8 stable version has various issues with the Portlet mode, so a Orbeon 3.8 nightly build from end of October 2010 was used. We also chose the stable release of Liferay 6, since the nightly build of Orbeon 3.8 is currently tested with this version only.

Integration of the different components and the overall architecture.

The Spring Portlet loads the XForms through some services invoked from the controller and uses an own XForms View to send the XForms directly to Orbeon through the portlet filter for further rendering. Other Spring MVC views like the JSTL view (a Spring view to display normal JSPs) are not affected, and can be used normally. The edit mode of this portlet is for example written as normal JSP Page.

The XForms XHTML Portlet Page

Spring MVC uses views to render models which are filled and created by controllers. (Ok, you might have guessed that;). There are many different included views for displaying Velocity templates, XML and JSPs. The “view resolvers” decide which View should be used, to render the requested page. So an own Spring ViewResolver was written, which reacts on our Spring convention that all View requests should start with “XForms::”. All other View requests are ignored by this resolver, so that the other resolvers can fetch it. As mentioned before, Spring is not needed in order to use XForms in Portlets; it was a general project requirement in our case.

Against your expectation, and not common when using Portlet page snippets, the XForms portlet page must be a complete XHTML page including all HTML Tags as shown below:

<html xmlns="http://www.w3.org/1999/xhtml"
  xmlns:ev="http://www.w3.org/2001/xml-events"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:xhtml="http://www.w3.org/1999/xhtml"
  xmlns:xf="http://www.w3.org/2002/xforms"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:n1="http://www.w3.org/2004/11/xmlmime"
  xmlns:xxf="http://orbeon.org/oxf/xml/xforms"
  xmlns:f="http://orbeon.org/oxf/xml/formatting"
  xmlns:c="http://java.sun.com/jsp/jstl/core">
<head>
  <xf:model>
    <xf:instance id="vu">
      <repo:vu xmlns:repo="repos-ns"
               xmlns:partner="partner-ns" xmlns:allg="allg-bs">
        <repo:Versicherungsunternehmen>
          <partner:Name>Versicherungsunternehmen</partner:Name>
          <partner:Anschrift>
            <allg:Strasse></allg:Strasse>
            <allg:Hausnummer>543</allg:Hausnummer>
            <allg:Postleitzahl>32430</allg:Postleitzahl>
            <allg:Ort>Stadt</allg:Ort>
            <allg:Land>D</allg:Land>
        [...]
      </repo:vu>
    </xf:instance>
  </xf:model>
</head>
<body>
  <h2>Adressdaten</h2>
  <xf:group ref="instance('vu')/repo:Versicherungsunternehmen">
    <xf:label>Firma</xf:label>
    <xf:input ref="partner:Name"/> <br/>

    <xf:group ref="partner:Anschrift">
    <xf:input ref="allg:Strasse"/>
    [...]
  </xf:group>
  [...]
</body>
</html>

Screenshot of the XForms web application running in a Liferay portal. The form on the right side is rendered by the XForms portlet. It contains a XForms page with different validation rules (seen by the red exclamation marks).

A Liferay Bug and a Workaround for rendering XForms

Usually, Spring views act like a normal “view” Servlet or JSP, i.e. you can write to the output stream or forward to a JSP to render your model. However, there seems to be a problem with the portlet filtering mode in Liferay 6, so JSPs cannot be used as view technology. The filter only works for direct writing to the response stream, not when a redirection to a JSP happens.

The first work-around was to create the complete XForms pages in the Spring view and to send it out to the response via some Output Writer. But while debugging the bug described above, we discovered the request attribute “oxf.xforms.renderer.document”. Orbeon first checks this attribute in its filter initialization, and, it’s set, uses the content of the attribute to render the page.

In our case, the complete page was filled via a String into this attribute, which seems to be quite a bit faster then the normal response stream filter:

request.setAttribute("oxf.xforms.renderer.document", result);

Receiving POST Data from the Browser

For receiving POST Data, no further configuration needed to be done. The binding URLs get rewritten automatically by the portlet filter. The following binding is sufficient to address the portlet correctly:

<xforms:submission id="gui-submission" method="post" ref="instance('getOffer')" resource="/" replace="all" />

With this binding, the submission resource URL (“/”) gets rewritten to the correct portlet action URL and the XForms model gets converted through the Orbeon filter. This filter is specified in the portlet configuration file “portlet.xml” as shown below:

<filter>
  <filter-name>orbeon-forms-portlet-xforms-filter</filter-name>
  <filter-class>org.orbeon.oxf.portlet.OrbeonPortletXFormsFilter</filter-class>
  <lifecycle>RENDER_PHASE</lifecycle>
  <lifecycle>ACTION_PHASE</lifecycle>
  <lifecycle>RESOURCE_PHASE</lifecycle>
  <init-param>
    <name>oxf.xforms.renderer.context</name>
    <value>/orbeon</value>
  </init-param>
</filter>

<filter-mapping>
  <filter-name>orbeon-forms-portlet-xforms-filter</filter-name>
  <portlet-name>xFormPortlet</portlet-name>
</filter-mapping>

For this to work, the portlet filter JAR “Orbeon-xforms-filter.jar” must be included within the Portlet WAR archive. For greater detail, see the Orbeon Wiki on XForms – Separate Deployment with Portlets.

Conclusion

We have discussed our quick prototype to test whether portal integration could be achieved easily or not. The general impression was that this solution based on the “separate deployment” mode works quite nice. And it seemed faster than the operating mode “Orbeon Portlet”. This might be because, in the portal environment, the portal takes up most of the loading time. A quick JMeter test revealed an increase of 50% load time.

The rewriting of URLs to the portlet addresses works quite well and transparent. The downsides of using Orbeon with portlets using our approach are the following:

• limitations in your application design, through the portlet filter bug (no redirection and JSPs),
• bugs in the 3.8 release of Orbeon XForms,
• the current release of Orbeon Portlet Integration has been tested only with Liferay 6.x.

Another interesting approach using remote portlets (had the same idea) can be found in the Orbean Developer Wiki.

In conclusion, if you are planning to use Orbeon XForms with your portal server, you should use Liferay and also think about getting professional support from the Orbeon Team (which requires a commercial license), because further insight into the Orbeon system is of great advantage here. Other great sources are the Orbeon mailing list, the Development Overview, and the Liferay Forum.

(c) 2012 mgm technology partners. This posting "Portlets with XForms in Liferay - Rethinking Web Forms with XForms, Part 3" is part of the mgm technology blog. The author of the posting is Christian Belka.

We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.