„We are maxed out anyway. We need a solution that helps to make our work more effective and not something that if we are lucky saves about as much time as it needs to set it up and maintain it“.
I hear statements like those again and again from IT administrators. With the effect that APM solutions are mainly used for firefighting by experts.

So, what is needed? A solution which allows to monitor a large number of applications with a minimum of configuration effort and identifies the root cause of problems quickly.

Indeed I found and tested a tool which fulfills those requirements. AppDynamics developed a product which is convincing not least because of its ease of use. I was sceptical at first but haven’t been disappointed in a couple of evaluations. Almost as easy as an iPhone- or Android-App. Simply use it.

The 3 steps towards 24×7 monitoring

Let’s take a look at the steps needed to establish application monitoring and how the AppDynamics solution adds value and saves time.

1. What to measure? – Measuring Points

The definition of measuring points (or sensors, probes) is the first challenge. Most APM solutions for Java or .NET use BCI (bytecode instrumentation) to get performance data. The measuring points need to be determined very carefully to prevent a big influence on the results (overhead) as additional code is executed. This usually asks for the assistance of an expert, an architect or developer. For every application that needs to be monitored.

If agile development processes are used this is an exhausting process as classes can change daily or new frameworks are added. A “trial-and-error” approach in production is prohibitive due to the necessity to restart the application servers most of the times. Additionally the overhead can be increased inadvertently to an unbearable level for the users.

AppDynamics uses a patent pending technology which needs only a minimum of BCI and still is capable of delivering information on method level to identify “loitering” components. And that without any configurations effort. The architect / developer can do his day job without being bothered by the admin.

2. How to get an overview? – Visualization

Dashboards are commonly used to provide an overview over the architecture (which component talks to whom and how often?) and the business transactions (which transaction is behaving cumbersome and who is affected) for all involved applications.

Most vendors use “customizable dashboards” for visualization as a kind of panacea where every view can be adjusted for every type of user. And that is exactly what needs to be done for every detail and every application – so to say “mustomizable dashboards”. Any change in the environment or the business functionalities requires additional effort.

AppDynamics dashboards are created automatically and determine business transactions based on the “inner” values of an application (e.g. strus actions, URI patterns or HTTP parameters). If the default settings are not matching they can be changed with a few clicks and the system is ready for action.

AppDynamics - Application Overview

AppDynamics Application Flow Map

3. Red Alert! Something is going wrong. – Thresholds

What defines a problem in production? Usually something out of the norm, e.g. a user login takes 3-times the time that is normal for that time of the day or a JVM uses excessive amounts of CPU. Such abnormalities are visible with the help of predefined thresholds where a violation thereof results in an incident or alert.

Now what I see in the real world are 100 and more applications with a multitude of different business transactions which have very divers “normal” response times: Sometimes 2 seconds are very good (cost calculation for an isurance policy), sometimes 200 ms are a catastrophe (placing a bet on an online beting patform). Or worse: There are no non-functional requirements defined at all, so that the thresholds have to be set using a dice initially and later adjusted. With only 50 applications with 50 transactions each we have a stunning 2500 thresholds that need to be set and checked. On a regular basis. And we only looked at response times so far…

With AppDynamics this is not needed. A slick baselining and statistical methods like standard deviation are used to automate this work. You can adjust each value individually if needed but 95% of all thresholds are already covered with the default rules. This includes time of day and weekly differences;  e.g. on monday mornings the login process takes longer because of the load and will not raise an alert though the same response time causes an incident 2 hours later or on tuesday morning as it is above the norm for that timeframe.

4. And what about root cause analysis? (Bonusstep)

Alerting in case of problems is nice and needed the admin knows that something went wrong or is about to go wrong in advance but who to notify for remedy? Triage and root cause analysis capabilities complete the monitoring. This means identifiying the responsible person to resolve the problem and additionally given them the details to return to normality quickly.

I stated before that AppDynamics instruments very little bytecode. How are the necessary details retrieved then? AppDynamics uses so called snapshots, which include a call stack with timings and details about the transcation itself. Snapshots are taken automatically of abnormal transactions (too slow, erroneous, etc.), on demand and time based (like every 10 minutes or every 100th occurence). With this technology an administrator is spared a tsunami of data but is equipped with exactly the necessary information when he / she needs it.

In the coming weeks we will publish a series of blog posts on how to diagnose different kinds of performance problems in detail.

Simple and effective

In summary:  AppDynamics created an easy to use and effective solution in which I see the promises of the last seven years kept. A simple to use system which was developed specifically for the monitoring of highly distributed, business critical Java applications.

Revolutionary? No, rather evolutionary. AppDynamics paid attention to the shortcomings of existing solutions and put a lot of thought into automation. “2-3-100” is the goal. 2 administrators take 3 days to setup 100 applications for monitoring.

While the first providers of APM solutions for Java and .NET had the goal to open the blackbox and get some data at all the second generation expanded this to transactions in order to be able to x-ray modern SOA/SBA based applications. What was missing was the usability and automation. How can I effortlessly sort my data and turn it into valueable information?

Let’s take a look into the next generation of APM!

Put an agent into an application (see AppDynamics Lite Screencast by Fabian), let it send data to the central controller and simply wait for the first results to reveal themselves.

So, what’s it about?

I just wanted to point out for those who do not know yet that Dana Bredemeyer will conduct his well-known Software Architecture Workshop at codecentric from Sep-13-2010 to Sep-16-2010. Dana haven’t had an open enrollment for this workshop in Germany for almost 10 years and from all I’ve heard about it this will be a great course. So, if you are interested in that workshop and do not want to travel to the United States for that you might want to take a look at http://www.meettheexperts.de/software-architecture.

Also, I wanted to mention our “Meet the Experts” on Sep-17-2010. “New challenges in architecture today” will be the topic. Five well-known speakers, including Dana Bredemeyer, Dr. Gernot Starke and Prof. Dr. Heinz Züllighoven will share their ideas in insights with you. And, as always there will be a lot of time for discussions, networking, open space for your questions, great catering, and so on. So, if that sounds interesting for you, just have a look at http://www.meettheexperts.de/meettheexperts-architektur.

Okay, that’s all I wanted to talk about. I think those sessions will be great stuff, I will be there for sure. And if you will be there too, I’ll be glad to meet you … and if not, maybe we’ll meet some other time …

Mirko gives a really good description why AppDynamics delivers the right solutions for todays distributed architectures in his Post “Troubleshoot Java in production – introducing AppDynamics Lite“. If you have not read it yet, head over to it now.

The key promise of AppDynamics is simplicity. Easy installation, easy operation and very fast results. In fact this sounds uncommon and hard to believe. But this is indeed the case, as proven by the following uncut screencasts.

Installation of AppDynamics is fast and easy like pie. In this screencast I will be showing to you how to perform the installation in less than 3 minutes (there is a full screen mode in the lower right corner):

Directly after the installation you can start investigating your performance issues. In the following screencast I show you the basic functionality and a possible workflow for finding the root cause of a slow response (there is a full screen mode in the lower right corner):

In a recent article Gunther Dueck from IBM wrote that we can try to romanticize the good old times where everything was easier and better than today, but one day we have to accept SOA, Cloud and Mobile Apps. They are there and will not go away soon.

As “firefighters” for enterprise applications we see also a shift in the problem areas of mainly Java based web and enterprise applications. Some years ago the problems where mainly “code problems” where Hibernate was not configured correctly or a memory leak broke the system down. These issues are still there but we see more and more architectural problems that are a result of the raising complexity and dependency of these applications and higher needs for performance and scalability. An application may handle 5.000 users – but is it designed to handle 100.000? Can it execute a request in 1 second if it aggregates 20 remote service calls? What happens if one service is not available or doesn’t reply in time?

These questions are mostly not answered before the application begins its life in production. And if a problem occurs the application seems to be a big monster or black box and finding the root cause of the problem can be a long and expensive process.

The problem gets even worth with agile development processes where we deploy new applications every 2-4 weeks. Some internet companies like Flickr even deploy more than 10 time a day. So the applications can change every day or week and the half-life period of your knowledge about the application is getting shorter and shorter.

So how can you approach performance or stability issues that occur in production is such an environment? In general a profiler is not the right tool. It has too much overhead, normally works just for a single JVM and is designed for usage in the development environment or the developers workplace. These environments are often just small and locally deployed environments that are not even close to a highly distributed SOA and/or Cloud scenario. In addition to that problems that are a result of high load, concurrency or legacy system can not be easyly reproduced in the development environment.

Classical monitoring or application performance management tools are in most cases not designed to work in highly distributed and loaded environment – so they generally not provide the data we need to analyze the root cause of the problem. In addition to that they normally not respond to change automatically and need a lot of configuration and instrumentation work. (and they cost thousands of dollars per JVM even if you just want to troubleshoot your problem)

AppDynamics is a San Francisco based start-up that developed what they call a “next generation performance management tool”. It is designed from scratch by experienced APM engineers to support exactly these highly distributed SOA/Cloud production environments. AppDynamics can monitor a distributed Java EE application without the need of custom instrumentation or configuration. It automatically creates an application map (see screenshot) which shows the number of calls and response times of any interesting call like database-, webservice- or JMS calls. It also gives you a full call stack down to method level of any interesting business transaction. This is done by an “intelligent” rule base that utilizes statistics of your application to decide which information to show. This means you can get inside information of all of your applications in 15 minutes – this is the time to install the AppDynamics server and agent(s).

You don’t belive this? (I didn’t either when I talked the first time to the AppDynamics team) There is a good news: AppDynamics Lite is a free edition that you can dowload to troubleshoot slow response times, SQLs, erros and stalls in production. The Lite edition has some limits, but will be in most cases sufficient to troubleshoot Java problems in (pre-) production. It is installed in less than 15 minutes – so give it a try and download it for free here: http://www.appdynamics.com/free

Fabian captured some really nice screencasts to demo the features of AppDynamics lite. In his blog you find “hands on” screencast on installation of AppDynamics Lite and how to find production issues.

Even the performance tester himself may be prone to this type of mistake. I can speak out of my own experience here… Either because he wishes for or expects a certain result or simply because of routine he might be seduced to accept his own numbers as valid too quickly. Application systems and their configuration are often very complex, so it is easy to overlook or forget little changes with possibly large effects on the performance test’s results.

The unshakable belief in the validity of numeric data can lead to many other inappropriate follow-ups. There is often a tendency to overanalyze a single data point, e.g. projecting possible future performance improvements or extrapolating the effects on a system n-times the size of the test system and so on. If later tests don’t confirm these projections, sometimes a frantic search for an assumed hidden cause of the “wrong” numbers is started, instead of realizing that incomparable things were compared in the first place. Also sometimes earlier numbers – correctly reported and documented – are rediscovered, but their context forgotten, resulting in similar wrong interpretations, conclusions and activities.

Most of the time it is very hard to argue against this firm belief in numbers, to state that other tests are necessary to confirm them and that rather large variations are typical. Performance measurements as part of QA in software development are definitely not an exact science. As “getting things done” is normally the topmost priority, the preconditions for a controlled experiment as in the academic world are rarely fulfilled. Yet the results of such experiments are often handled as if they are fulfilled.

As a performance tester, one has to be aware of the possible ramifications of the reported test results. Testing – especially performance testing – is done to replace uncertainty with knowledge, but careless handling of test results can easily lead to the opposite outcome. In my opinion, the foremost rule should be to only report data that one understands, can stick by and defend. One should always be aware of the reliability, scope, relevance and comparability of the numbers reported and should document these characteristics and attach them to the report. It is often a good idea to explicitly remark if a measurement is just a statement about one particular test-setup and not suitable for further projections.

If a misinterpretation of test results occurs, one should act swiftly, clearly and forcefully to present the tester’s view of the numbers and their context in order to contain the possible damage. Numbers, once reported, tend to acquire a life of their own. Like a genie, it’s practically impossible to get them back into the bottle. Therefore, one should be very careful what to report and when to report.

One final advice: As a performance tester, always remain skeptical about your own work and your own instincts. Other people should not blindly believe in test results and numbers and neither should the tester.

Finding the culprit

To find out whats wrong the best is to create a heap dump at peak usage to show whats consuming all the memory.
After opening that dump in the wonderful Eclipse MAT you might get a picture like this:

Ouch. 1.8 GB out of 2GB are consumed by sessions. I usually then filter by “StandardSession” for apache sessions to be able to easily browse around the sessions that have high amount of retained heap.

Ouch again… 10MB per session, this cannot scale. Now it could of course be possible that some careless programmer put in much of our data into the session, but thats simply not true, as we can easily find out:

Thats interesting. Almost all of our memory is consumed by AjaxStateHolders. So what does this actually do?

How JSF and A4J work

I try to keep it simple. Before JSF actually renders HTML to the users browser, it builds an internal representation. For each page (or view) this internal representation (called component tree) is created and run through the lifefcycle. Any user events are processed. If there is a component where a user can select one of 3 items, those 3 items are loaded and a string representation put onto the page. The component keeps track of which item is selected.

Now we do not want the user to submit the whole page to select another item, which would cause the component tree to be recreated and a different item to be selected. We want Ajax! For that A4J remembers the state of the components when the page was displayed. So it knows the component tree and those 3 items, and which one was selected. On AJAX actions the component is looked up, the state modified by selecting another item and the partial HTML representation sent back. This is also called partial page rendering.

So how does the remembering work? Well you might have guessed it: It creates a AjaxStateHolder in the user session and attaches the component tree to it.

How this can get big

Ok, this might be the current page. I can have big views but it is only one. No thats unfortunately not true. A4J does store more than one view. But why? Look at your browser, you are quite likely to find a “back” button somewhat in the to-left corner. When you click that you see the last page. But the browser did not send a request, so the server does not know that you are on that page. Imagine you click AJAX functionality: Where should the server get the component tree from for that time you have been on the page? Easy answer: This is as well in the AjaxStateHolder. By default this tracks back 16 views. and because you might come across the same view in your browsing history more than once there are up to 16 variants of a view.

As you can see this can get big. And remember, this is per user.

How to fix

Unfortunately there is no fix. This is how JSF works, and how A4J works.

There is a ticket on the Richfaces bug tracker which describes this: RF-3878 – Session memory leak. And the best answer is: reduce the number of views you store.

<context-param>
 <description></description>
 <param-name>com.sun.faces.numberOfViewsInSession</param-name>
 <param-value>1</param-value>
 </context-param>
<context-param>
 <description></description>
 <param-name>com.sun.faces.numberOfLogicalViews</param-name>
 <param-value>1</param-value>
</context-param>

You will loose the browser back button functionality, but gain a lot of memory. Second way to fix is to reduce the size of the component tree, which is most likely affected by big lists of data and complex structures. However this is easier said than done.

After some online research and several interviews with other project teams of codecentric I would like to describe a very simple, robust and primarily fast way to generate PDFs with iText. With this approach, the ticket mentioned above was solved in a quarter of the estimated time and the fear of iText in the team disappeared.

iText offers a lot of possibilities to create PDFs completly using its API, containing all lines, pictures, fixed texts and much more. But this approach may get very complex and generates a lot of error-prone code. It’s much easier to create a template with defined content-fields. This Template will be saved as a PDF with a formular and can be watched for example in the Adobe Reader at any time. Another advantage is the possibility to show your customer the empty pdf and discuss the layout before implementing the logic which fills in the data.

Part 1 – Creation of PDF-formulars

PDF-formulars can be created very easy with OpenOffice. I used OpenOffice Draw in my example, but the approach also works with OpenOffice Writer or Calc. Just create a new document and fill it with texts, colours or lines as desired. After the layout has been finished, you have to define the formular fields. Therefore it’s helpful to activate the toolbars for Formular Design und Formular Control Elements via View -> Toolbars (Sorry – I don’t know the exact english translations, because I use the german version of Open Office). You can activate the Formular Design Mode using one of the new icons. After that, you can choose for example a textfield from the Formular Control Elements and insert it into your document. The properties of the textfield can be changed with a right-click. The following properties are interesting:

• Name : This name will be used in the Java code to reference the control element and fill it with data.
• Border : Default is 3D-Look – In most cases the setting without Border should work better.
• Background-Colour, Alignment, Font: You can define the layout with these properties. Layout data in Java code is not necessary!
• Text-type: Default should be one-line, but if you need to fill in the field with line-breaks from Java code, you have to use multi-line.

It’s very hard to spot the formular fields as soon as they have no border. The Formular-Navigator helps to solve this problem.

After creating layout and formular fields the document can be exported to a PDF. This can be done by File –> Export as PDF. Be sure to activate the checkbox Create PDF-Formular in the following dialog.

Editing a formular in OpenOffice

OpenOffice - PDF Template

Generated PDF Template

Part 2 – Fill the PDF-Formular with iText

In my example I use iText in version 2.1.7, because in later versions iText has a more strict license.

First read the created PDF template…

PdfReader pdfTemplate = new PdfReader(pdfTemplateFile);

… and create a PdfStamper with a OutputStream.

ByteArrayOutputStream out = new ByteArrayOutputStream();
PdfStamper stamper = new PdfStamper(pdfTemplate, out);

We have to tell the PdfStamper to flatten the form, because there should be no formular fields in the created PDF.

stamper.setFormFlattening(true);

To finish the work we can just fill the fields…

stamper.getAcroFields().setField("name", "Daniel Reuter");
stamper.getAcroFields().setField("adress", "Merscheider Str.1 – 42699 Solingen");
stamper.getAcroFields().setField("dates", "2008\n2009\n2010\n");
stamper.getAcroFields().setField("titles", "JAX\nDevoxx\nJavaOne\n");

…and close all the opened resources.

stamper.close();
pdfTemplate.close();

Conclusion

From my point of view, this is a very simple and easily understandable approach to create PDFs using iText. Certainly there will be scenarios in which the described way won’t work, for example if you have to implement dynamic layouts. However, I would always try to use a PDF template as a basis.

PDF Result

As we do agile software development for our customers I was curious what agile enterprise means. So I asked Google and read some articles about it. A good definition can be found at the Business Dictionary:

Fast moving, flexible and robust firm capable of rapid response to unexpected challenges, events, and opportunities. Built on policies and processes that facilitate speed and change, it aims to achieve continuous competitive advantage in serving its customers. Agile enterprises use diffused authority and flat organizational structure to speed up information flows among different departments, and develop close, trust-based relationships with their customers and suppliers.

Basicly “Agile Enterprise” means that enterprises are forced to change quicker and be more flexible as globalization has changed the way our markets work. Most of the articles I’ve found suggested to use self-organized teams and high interaction between people in an organization. But what does this mean for IT?

Let’s get back to the article. The article is about big ERP systems and about the problems most of the customers have with these systems if raising flexibility and speed of the business hits the IT. The reasons for these problems can be found in the high complexity of these systems, low productivity in development as a result of excessive customizing of the “standard” system. The statement of the article is: “ERP slows business down!”

Most of the arguements in the article are understandable for me and we see them every day in our projects. But not only with ERP systems, but with any kind of big standard software “monsters” – e.g. big insurance core systems. I have always asked myself why people first spend millions for licences and then realize that the “standard” doesn’t fit to their needs and processes. In many cases this leads to big customizing projects that run for some years and change the “standard” so much that you cannot upgrade to newer releases anymore. So every upgrade will be another midsize project. But I still forgot to tell you the best of the story: You pay 20% per year for support of the standard software – which means that you pay the intial millions again -  every 5 years!

Unfortunately the article doesn’t give good advices how to solve these issues. The strategy of the big ERP vendors is criticized – especially that the promices of SOA were not fullfilled.

A possible solution is seen in the cloud or in Software as a Service. This realy irritated me. So if I don’t operate the standard “monster” myself and put it in the cloud all my business problems are solved and the IT is flexible and fast enough to keep the new pace!? The statements of the many experts of big consultancy companies really scared me…

Last week I’ve thought a lot about the  “Why” for our “Agile Software Factory” and this article and the statements of the experts really helped me to find it.

For me the solution is obvious:

If “Agile Enterprise” means that the business has to be innovative  (First Mover) and fast adopting (Fast Follower), than the solution cannot be a standard software. Because “standard” definitly is the opposite of innovation and the monolithic character of the software often means complexity and not flexibility.

In my point of view we should implement the business processes with individual software. The used components can be standard and I don’t want to tell you to write a booking system from scratch. There are many laws etc. that regulate a booking system so that there is no room for real innovation. But the processes around the booking system can be very specific to the business of a company and putting the processes into the internet and offer customers and partners direct access to it could be a competetive advantage. Also a tight integration in other systems can offer more productivity. Wouldn’t it be great to offer one clean and easy to use interface for all applications and processes to the business users?

If software development should be faster and more flexible and the requirments are not fully known at the beginning of a project as we have to deal with a quickly changing business, you have to use a methodology that minimizes the risk of individual software development. Agile software development offers you the needed methodology. We will not be able to deliver business value quickly with waterfall approaches and projects that run for months or years.

And this is exactly why we offer the Agile Software Factory: A proven approach based on Scrum and XP to develop individual software. With years of experience and lots of best practices we can develop software fast and with high quality. We call it a “factory” because we use standardized components and frameworks for our development and do not reinvent the wheel for every new customer.

We use Open Source components as they offer open standards and open sources. This helps us to build software that is very extendable and easy to integrate in the systems of our customers. In addition to that we are not encountered by the politics of the big vendors. Alfresco for ECM, Pentaho for Business Intelligence, Liferay for portal, OpenCms for CMS  or Compiere for ERP and CRM are only examples of stable mainstream Open Source components. The individual processes and applications we develop for our customers are based on a lightweight architecture and well known Open Source frameworks like Spring, Hibernate, Drools, jBpm or Mule.

Our agile process gives us the flexibility to deliver running software to the business very 2-4 weeks! So they have new needed functionality very fast and in high quality. This quality is enforced by automated unit-, regression- and system tests, continuous integration, pair programming and automated code- and design reviews.

If this isn’t what “Agile Enterprises” need – we will move to the cloud

You can see the long way to our goal. But we are close to it. You can see that I would like to get a reward for the fantastic sprints we did, so I painted a little cake right after the finish line.
Below the path leading to the finish, I depicted our architecture and the actual goal of the project. I really like how well the architecture evolves and works our for us, and also like that our customers start to realize that their confusion was not necessary and they can realize their ideas wit it.
At the top left, there is a bad test stability. It really can be improved and we actually started already addressing this. Right below that you can see our burndown for that iteration, which is not bad, but should have run below the line instead above it.
It seems that I was as well pretty pleased with the amount of features we did create and the value (meaning money) it can generate for our customers. last but not least I draw the great sprint demo we did for our customers the day before.

So how do you like my picture (or doing this kind of retrospective at all)?

Here is an often ignored part from java.text.SimpleDateFormat JavaDoc:

 * Date formats are not synchronized.
 * It is recommended to create separate format instances for each thread.
 * If multiple threads access a format concurrently, it must be synchronized
 * externally.

Perhaps that warning isn’t strong enough, because it reads: “It is recommended”.
This leads to this highly dangerous line of code I have seen in almost every project and is likely to produce hangs in concurrently accessed code.

private static final SimpleDataFormat DATE_FORMAT = new SimpleDateFormat("yyyy.MM.dd");

But it gets worse with System.gc().
There is no warning in the JavaDoc:

* Calling the gc method suggests that the Java Virtual
* Machine expend effort toward recycling unused objects in order to
* make the memory they currently occupy available for quick reuse.
* When control returns from the method call, the Java Virtual
* Machine has made a best effort to reclaim space from all discarded
* objects.

Doesn’t sound too bad, does it? You should wonder what “suggests” and “best effert” really means. Perhaps you are tempted to use System.gc(), so you might look on the internet for other people using this, hoping for an explanation what in reality happens. You might find good insights on Stackoverflow. But what exactly this does nobody seems to know.

I think a picture tells more than thousand words:

We are looking at a pretty big system of one of our customers. It has 14GB of heap and we helped them to bring down the memory consumtion so that the app ran quite well. However there were still pretty long pause times (so called stop the world time). The graph was created from a Garbage Collection log and points us to the cause of the pause times. The blue line shows Garbage Collection activity. So why are there exactly 13 spikes with up to 35 seconds? All the other activity seems to be really fast.
The reason for those 13 spikes is simple: They all were caused by an invocation of System.gc().

So what happens? Calling System.gc() seems to effectively stop any optimization the GC is doing, forcing a full collection to start, rather than the optimized CMS Algorithm used.

Note: This system is using a Sun 1.6.0.18 64 bit HotSpot Server VM. GC is set to -XX:+UseConcMarkSweepGC and -XX:+UseParNewGC. System.gc() does not have to produce results like that. Your results might vary. However also others have reported, that when using CMS and System.gc() a Full GC is always performed.

If you cannot find yourself invoking System.gc(), they might hide deeply in a used library. To fix this, you can use the the server argument “-XX:+DisableExplicitGC”. This argument is also recommended by the SUN/Oracle performance tuning guide.