Trip over IT

Heap Visualizer (web app)

2011-09-25T22:57:00.000+05:30

JVM Memory (Heap and Non-Heap) Visualizer

http://code.google.com/p/heap-visualizer-webapp/

You're having some heap-related issues (OOM, long GC pauses) in production and do not have the liberty to fire up jconsole or install glassbox (or something similar) to diagnose the situation. What do you do Jack?

I these situations, I wished that

I could have a view of the Heap and Non-Heap sections simultaneously.
I could then drill down into either or both the sections and view details of the different heap generations and sections as the application progressed
I could do this by simply dropping an app (.war) into the app server to monitor the heap.

Luckily I didn't waste too much time wishing for more and just wrote the darn thing :)

Well, in keeping with the philosophy of this blog I thought I'd share it so that it might be useful to someone someday.

I normally deploy the app as 'heap.war' so I can reach it at http://[server]:[port]/heap

Here's what you see when you first access it:

The screen shows the overall used/committed/max values for the 'Heap' and 'Non Heap' sections of the memory used by the JVM.
The UI is minimal and the colors have been chosen to present a 'console' experience.
There are two graphs, a bullet graph that plots the used, committed and max values for the respective portion of memory, and a line graph that plots the relative movement of the 'used memory' values.

If you click on the 'details' link you'll see something like this:

As you can see, you'll be able to see drill-down details of the 'heap' and 'non-heap' memory components and their respective memory usage and utilization characteristics.

The next version have customizable refresh-rate (it's currently set at 2 seconds), an option to invoke a Garbage Collection and a way to look at the detailed properties of the current JVM

The code is at http://code.google.com/p/heap-visualizer-webapp/
You can get the source and the download (.war) from the menu on this page.

Do post your feedback and feature requests :)

Technical Disclaimers

2011-06-30T10:47:00.087+05:30

I just read an interesting document on software disclaimers by Jarek Zylinski which is, in short, more of an appeal to develop software using educated (read professionally trained) and ethical (read decent-human-beings) personnel.

I took a few minutes to ponder on the impact this kind of thinking could have to so-called intermediate software artefacts/products created during a product's SLDC.

In today's multi-company development environments, this is especially important as experts from one company might create an overall architecture, while experts from another firm create high-level designs and integration specifications and another firm might actually realize the design into a tangible code. During this process, there are several software artefacts, especially documents, to be generated before the final product is realized, and at some level all of these artefacts could benefit from having a way of telling the audience what they are really about and what the audience should really focus on without getting caught up in the "letter" of the document.

You, as a producer of an artefact that is in the form of a document, need to have a way to declare upfront where you've directed your efforts and your wishes to ensure that the audience views the document as you do.
Here is, what I believe to be one of the clearest disclaimers to add to a technical document:

The contents of this document are believed to be accurate. This is not a legal document and should not be treated as such. The statements, information and recommendations mentioned in this document are intended to be read more in spirit than in letter. This document is intended to be objective. Any subjectivity or bias that has crept into this document must be considered unintentional and must be attributed to a human failing on the part of the author.

The author(s) of this document agree to the terms above.

I really don't recall where I got this from, but I do recall seeing something similar in a document I read on the internet. I might have, at the time edited it for relevance to my field of work but have lost the source of this inspiration.

If any reader that stumbles upon this post and has knowledge of its source, please comment and I shall take appropriate action.

Save encrypted PDF as plain / unencrypted PDF

2011-06-30T10:20:00.001+05:30

Note: If you don't have the password for the PDF file, this post is not what you're looking for.

You have a PDF file that's encrypted and you already know the password.
Say you need to send this PDF file to someone without them needing to use a password to open it, or you just need to store it in an unencrypted format. How do you do this?

As always with software, there are options:

Option 1: Buy an commercial product (like Adobe Acrobat or Foxit PhantomPDF )that is able to strip the encryption from the PDF file- Not for El Cheapo types (like me)
Option 2: Use a product to save the file to JPEG (or any image) format, either manually (using screen prints) or using a free/commercial product (like SnagIT) that takes a screen shot of the entire document, not just the visible window. - Not for the lazy (like me)
Option 3: The quick, simple and free and high quality way - Now that's what I like

As you guessed, I'm going for Option 3 :)

Option 3 - A surprisingly simple way of doing this while preserving the quality and format of the document:

Install a free PDF Printer (I'd recommend PrimoPDF, but that's purely a personal choice).
Open the encrypted PDF file and type in the password.
Invoke the 'Print' option (File->Print) and use the PDF Printer as your choice.

Enjoy!

Creating DMZ configurations on Amazon EC2

2011-03-19T07:23:00.001+05:30

This post tries to address an approach to replicating corporate-style DMZ configurations using Amazon EC2 components.

Conventional (non-cloud) Network configuration for hosting internet facing applications
Most internet applications are deployed on servers placed in secured demilitarized zones.
A DMZ's main function is to restrict the extent of an attack should systems be compromized.
Typically, there would be a main firewall that allows traffic from the internet only to web servers on a particular port. Subsequent to that there could possibly be firewalls that isolate application servers, database and reporting servers etc.

An example of this configuration is shown below. It consists of a Web DMZ hosting only web servers accessible from the internet via port 80. The application servers are placed after the second firewall and cannot receive traffic from any other source other than the web servers. The traffic is only restricted to port 8009 and only in one direction. The database servers are behind the third firewall and can only receive traffic from the application servers on port 3128.
For monitoring and management, all machines in the network have been allowed access on port 22 from machines internal to the company.

Cloud Network configuration for hosting internet facing applications

The first thing to realize is that a Security Group is quite similiar to a firewall configuration specifying the allowed protocol (tcp/udp), port/port range, traffic direction and source network. The only real difference is that Security Groups currently only support the ALLOW sematics (not the DENY), but this is quite sufficient as we shall see.

Once a Security Group is created, it can be associated to a server instance. In reality, the Security Group/s have to be specified for an instance before it can be instantiated.

They say that a picture is worth a thousand words, but I don't think that includes code.
The following depicts the configuration of Amazon's EC2 Security Groups to replicate the above DMZ configurations.

The simplest approach to achieve this is as follows:

Map each DMZ to a Security Group
Create the Security Groups in EC2 using the ec2-add-group API
Authorize access (port, port-range, direction, source group/CIDR) using the ec2-authorize API
Create instances using these security groups

Example: Web DMZ, App DMZ and DB DMZ

//create groups / zones
ec2-add-group web-dmz -d "Web DMZ"
ec2-add-group app-dmz -d "Application DMZ"
ec2-add-group db-dmz -d "Database DMZ"

//Allow admin access to all the servers
ec2-authorize web-dmz -P tcp -p 22 -s 0.0.0.0/0
ec2-authorize app-dmz -P tcp -p 22 -s 0.0.0.0/0
ec2-authorize db-dmz -P tcp -p 22 -s 0.0.0.0/0

//Allow access to the site from anywhere on the internet
ec2-authorize web-dmz -P tcp -p 80 -s 0.0.0.0/0

//Allow access from Web DMZ to Application DMZ on port 8009 only
ec2-authorize app-dmz -o web-dmz -u xxxxxxxxxxxx -P tcp -p 8009

//Allow access from Application DMZ to Database DMZ on port 3128 only
ec2-authorize db-dmz -o app-dmz -u xxxxxxxxxxxx -P tcp -p 3128

//Create instances and assign them to DMZs
ec2-run-instances ami-cef405a7 -z us-east-1d -t t1.micro -g web-dmz
ec2-run-instances ami-cef405a7 -z us-east-1d -t t1.micro -g app-dmz
ec2-run-instances ami-cef405a7 -z us-east-1d -t t1.micro -g db-dmz

Add t1.micro support to ElasticFox

2011-03-14T10:06:00.000+05:30

A lot of people, including me depend on the ElasticFox firefox plugin to interact with Amazon Web Services and its easy to see why. It has GUI support for almost everything that you'd need to interact with AWS.
However, it seems that, as of today, ElasticFox has yet to add support for the t1.micro instances that Amazon introduced recently.

Well, I just couldn't stand waiting for the latest release, and if you're just as impatient, you might want to try the following:

Check out the elasticfox code from svn:

svn co https://elasticfox.svn.sourceforge.net/svnroot/elasticfox/trunk elasticfox

Open up

trunk\src\chrome\content\ec2ui\newinstancesdialog.js

in your favorite editor. Go to line 247 and add the following:

if (this.image.rootDeviceType == "ebs") {
   typeMenu.appendItem("t1.micro", "t1.micro");
}

Now, its time to build the code into a plugin.
Download and install the firefox extension developer plugin
Open Firefox. Go to Tools->Extension Developer->Extension Builder, browse for your working directory and hit 'Build extension'.
You can then drag this resulting .xpi file into the browser window to install the plugin

You should now be able to select a t1.micro instance for a corresponding valid AMI

2011-03-12T12:08:00.003+05:30

Distributed Caching Using Ehcache on Amazon EC2

This post elaborates the setup of Ehcache to enable distributed caching in JEE applications. It also contains a specific Ehcache configuration for use in applications deployed on Amazon EC2.

Ehcache, since its takeover by Terracotta®, has evolved to incorporate enterprise features like ‘search’, BigMemory in the Ehcache DX, EX and FX product editions.
However, you can still achieve a high degree of distributed caching by just using the ‘core’ Ehcache library.

This post focuses on a minimalistic approach to incorporate distributed caching into an application. Deployment approaches that involve a large number of instances in a cluster/farm and require cache interactions to participate in JTA-transactions are not covered here. Distributed Caching that caters to these requirements can be achieved using a Terracotta Server Array (TSA).

There are 4 available mechanisms that can be employed to realize a distributed and replicated cache using Ehcache:
RMI                – Default remoting mechanism in Java
JGroups             – Requires adding JGroups to the application stack
JMS     – Requires a Messaging System Provider
Cache Server     – Realized using a Terracotta Server (/Array)

This post will cover the use of Java’s default remoting mechanism – RMI.

Configuring an application for Distributed Caching
Enabling distributed and replicated caching in a Java/JEE application is surprisingly simple using Ehcache. The procedure involves 3 steps:

Download Ehcache (core) from http://ehcache.org

Place ehcache-core-xxx.jar, slf4j-api-xxx.jar, slf4j-log4jxx-xxx.jar into the library folders of your application (e.g. classpath, WEB-INF/lib)

Place the Ehcache configuration file (ehcache.xml) in the application classpath (e.g. classpath, WEB-INF/classes)

Ehcache Configuration File (ehcache.xml)
The configuration file, which configures Ehache’s Cache Manager, must have the following elements to enable distributed caching:

A CacheManagerPeerListenerFactory

A CacheManagerPeerProviderFactory

At least one Cache configuration containing:

A CacheEventListenerFactory
A BootstrapCacheLoaderFactory

Configurations for environments with no multicast limitations:
Using RMI as the replication mechanism, a simple configuration of the Cache Manager would be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="ehcache.xsd"
         updateCheck="false" monitoring="autodetect"
         dynamicConfig="true" name="my cache">

    <cacheManagerPeerListenerFactory     
               class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"/>

    <cacheManagerPeerProviderFactory
                class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
      properties="peerDiscovery=automatic, 
                   multicastGroupAddress=230.0.0.1, 
                   multicastGroupPort=4446, 
                   timeToLive=32"/>
 

    <cache name="cache1"
           maxElementsInMemory="100"
           eternal="true"
           overflowToDisk="false">

        <cacheEventListenerFactory
                class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"/>
        <bootstrapCacheLoaderFactory
                class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"/>
    </cache>
</ehcache>

This configuration file can be copied into all deployments of the application that need to be part of, and have access to the distributed cache.

Configurations for environments with multicast limitations:
If the application is deployed within an environment where multicast is not allowed or not available (like Amazon EC2) then the following configuration can be used:
Example configuration on machine 1 [server1]:

<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="ehcache.xsd"
         updateCheck="false" monitoring="autodetect"
         dynamicConfig="true" name="my cache">


<!-- For Non-Multicast supported environments [for server1] START -->
    <cacheManagerPeerListenerFactory
            class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
   properties="port=40001, 
   remoteObjectPort=40002, 
   socketTimeoutMillis=120000"/>

    <cacheManagerPeerProviderFactory
           class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
           properties="peerDiscovery=manual, 
                        rmiUrls=//server2:40001/cache1"/>
  
<!-- For Non-Multicast supported environments [for server1] END -->

    <cache name="cache1"
           maxElementsInMemory="100"
           eternal="true"
           overflowToDisk="false">

        <cacheEventListenerFactory
                class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"/>
        <bootstrapCacheLoaderFactory
                class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"/>
    </cache>
</ehcache>

Example configuration on machine 2 [server2]:

<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="ehcache.xsd"
         updateCheck="false" monitoring="autodetect"
         dynamicConfig="true" name="my cache">


<!-- For Non-Multicast supported environments [for server2] START -->
    <cacheManagerPeerListenerFactory
            class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
   properties="port=40001, 
   remoteObjectPort=40002, 
   socketTimeoutMillis=120000"/>

    <cacheManagerPeerProviderFactory
           class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
           properties="peerDiscovery=manual, 
                        rmiUrls=//server1:40001/cache1"/>
  
<!-- For Non-Multicast supported environments [for server2] END -->

    <cache name="cache1"
           maxElementsInMemory="100"
           eternal="true"
           overflowToDisk="false">

        <cacheEventListenerFactory
                class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"/>
        <bootstrapCacheLoaderFactory
                class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"/>
    </cache>
</ehcache>

If there are more than 2 servers in a replication group, then additional servers can be specified in the ‘rmiUrls’ property using a pipe character ‘|’ as the delimiter.

This configuration relies on manual peer discovery and requires the enumeration of all machines participating in the distributed cache to be specified.
Another important point to note is that if the machines are separated by firewalls then the RMI remoteObjectPort needs to be specified (as shown in the example above) and firewall configurations would need to be made accordingly.

For the above configurations, create your servers/virtual machines in a security group and authorize the group to itself (this allows all virtual machines in the group free access to each other).

ec2-add-group ehcache -d "ehcache replication ports”
ec2-authorize ehcache -o ehcache –u [aws account id]

If this is not possible and each virtual machine has its own group then allow ports 40001 and 40002 between the groups (you can read this post for details on how to do this).

Phone remains in headset mode - Fix

2010-08-10T07:37:00.002+05:30

For those who own it, the nokia 6230i with its great form-factor, screen resolution and connectivity options is what keeps them from wanting to chuck it for the latest and greatest to arrive in the market. It is a great phone for accessing the internet and your email easily on the go. If you are a power user, the only accessory that you may ever need is a DKU-2 USB Data Cable

Imagine my dismay when I received a call this morning and couldn't hear a thing unless I switched on the loudspeaker. Looking closer at the screen I found that the headset sign was on even though the headset was not connected! Googled, looked up forums and decided to fish for an old toothbrush. They usually come in handy while cleaning my stereo and PC motherboard:)

The headset sign remaining on seems to be a common problem with a lot of phones and it happens because there is usually some accumulated dust/dirt lodged in the connector at the bottom which makes the phone think that a headset is connected.

Fixing this problem is a breeze. All you need to do is remove the front and back covers. Unscrew the 6 screws that you see on the PCB and brush the bottom connector from the inside and the outside throughly with a dry toothbrush. Worked for me!

Memorizing Design Patterns - the fun way

2009-12-15T11:44:00.004+05:30

Just stumbled upon a wonderful way to memorize stuff at memorize.com

Say you want to test yourself or just plain want to memorize Design Patterns for example, just head over here and discover how fun it can be.

Design Patterns
Core JEE Design Patterns

I must admit, this is a really unique way of using a wiki!

*I am in no way affiliated with memorize.com. I just happen to like their stuff

Ready reference for Web Service Types

2009-07-06T09:57:00.001+05:30

TripleDES encryption compatibility when using Java and .NET

2009-06-15T12:05:00.004+05:30

Note: This article shows you how to generate a SecretKey to use with a TripleDES encryption cipher. The shared-secret key can be 24 byte or even 16 bytes long.

For a quick brief of how TripleDES (3DES) works have a look here.

The most common problem related to encrypting something in Java and decrypting in .NET or vice-versa is a misunderstanding of the Keying options that are defined in the standards and those implemented by Java and .NET

A DES key is made up of 56 bits and 8 parity bits (8 bytes)

A 3DES key is made up of a bunch of 3, 8-byte DES keys i.e. a 24 bytes long

If you are going to use a 24 byte key for both Java and .NET, you're safe; then encryption will be compatible.

Java will force you to use only a 24 byte key when using TripleDES; the subtly is that .NET supports both a 16 byte as well as a 24 byte key.

Now If you generate a key from a MD5 hash of a shared secret, it will be just 16 bytes. .NET has no problem with this. It implements Keying Option 2. It will intelligently take the first 8 bytes and append it after the 16th byte - forming a 24 byte key. Java, *sigh* sadly doesn't do this. You'll have to spoon feed it like so:


public SecretKey getSecretKey(byte[] encryptionKey) {
 SecretKey secretKey = null;
 if (encryptionKey == null)
   return null;

byte[] keyValue = new byte[24]; // final 3DES key

if (encryptionKey.length == 16) {
 // Create the third key from the first 8 bytes
 System.arraycopy(encryptionKey, 0, keyValue, 0, 16);
 System.arraycopy(encryptionKey, 0, keyValue, 16, 8);

} else if (encryptionKey.length != 24) {
 throw new IllegalArgumentException("A TripleDES key should be 24 bytes long");

} else {
 keyValue = encryptionKey;
}
 DESedeKeySpec keySpec;
try {
 keySpec = new DESedeKeySpec(keyValue);
 SecretKeyFactory keyFactory = SecretKeyFactory.getInstance("DESede");
 secretKey = keyFactory.generateSecret(keySpec);
} catch (Exception e) {
 throw new RuntimeException("Error in key Generation",e);
}
 return secretKey;
}

Damn Small Linux: Install files with .dsl extension

2009-04-22T08:40:00.003+05:30

The DamnSmallLinux live CD/USB allows you run DSL in memory. It also allows you to run apps that you download from the MyDSL repository. If you install DSL to HDD/USB then you'll want to have these apps permanently on your system. You can do this easily.
If you look carefully, the MyDSL repository, contains a lot of files (apps) with the .dsl extension.

A .dsl extension file is nothing but a compressed archive in the tar-gzip format.

All you need to do is:

download the .dsl file to any location (lets say /home/dsl)
cd /
tar -zxvf /home/dsl/.dsl

and you're through!

JPA and Stored Procedures

2009-03-12T17:02:00.006+05:30

I recently stumbled upon many queries on google about how to make Stored Proc calls via JPA. Something was not right with questions like this.

Stored Procs and JPA are completely different beasts with completely different philosophies behind them. Sure you can call stored procs using JPA but thats using the wrong hammer. JPA deals with persistence of entities; Stored procs, with business logic being closest to the data.

If you have a lot of business logic in stored procs, you have the advantage of speed, security and maintainability, however, you lose portability of your business logic if you change your database. Now, in any large-scale enterprise, the database vendor is not really going to change and a database (oracle/sql server/db2) is going to be a uniform commitment across the organization. Having said that, even if you have to port the procs from one database to another, the vendor documentation will include quite comprehensive migration guides for the same. They have to :) For the record, IMHO, I normally stick with stored procs for the reasons listed above. I always have some real DBA cats working on my team that perform something called ATM (Application Transaction Modelling) on the database. Look it up (ATM). Once they are through with this you can flex your ORM muscles all you like, but in the end what you will be left with will just be UGLY.

Been there, done that, got the T-Shirt.

If there's one book you should read on JPA, let it be this:

HP Laptops with XP - Brightness buttons dont work - Fix

2009-03-08T18:46:00.003+05:30

If you've got a new HP laptop and have formatted it (to probably get rid of the lousy Vista OS)

and installed or downgraded it to XP you'll have noticed something that people all over seem to be annoyed with:

The brightness keys Fn F7 and F8 don't work once the XP OS starts loading into memory.

HP has probably rigged this to work only with Vista. So you're stuck.. right?

Nope :)

Here's the surefire fix:

1. Just download the .NET 2.0 framework from microsoft

2. And then hop across to HP and search for something called HP Quick Launch Buttons or do a google search for "HP Quick Launch Buttons "

3. Download it for your laptop model. Install it and reboot.

After this you should have your laptop running XP with the Fn F7 and F8 brightness buttons working!

BPEL: Get clobbered with the Golden Hammer

2009-02-18T12:08:00.012+05:30

Somewhere in the lobby after an IBM/Oracle seminar showing off their shinny and latest BPEL products

Brainwashed customer: I just met with IBM and Oracle and I'm convinced that I need their BPEL engines since we are going down the SOA road. After all, what am I going to do with my business services?

BPEL Zen Master (to herself): Ah. I see I have to do more penance on this earth by suffering these zombies. Looks like they dont know that SOA is dead

BPEL Zen Master : I agree you need to model and monitor your business processes and also automate some 'processes' that you have. How did they convince you that you need BPEL to do this?

Brainwashed customer: They told me that with BPEL my business activities can be modelled and hence I will have control and I can monitor them and I can tweak them and ....

BPEL Zen Master: Ahem! Have you tried modelling your process in a tool that supports BPMN? Have you tried the same with a tool that does it in BPEL.

Pssst... You will find that the big vendors are using the 'golden hammer'.

You must know that you cannot model your Business Process completely in BPEL as you can do in BPMN.

Contrary to what a vendor might tell you, a business analyst just CANT manage (draw, manage, round-trip) a business process as a BPEL today. Anyway, once you model your processes in the vendors modeller, round-tripping that back is virtually impossible. Besides, BPEL is all very technical...Would your like your Business Analyst to know what a SOAP fault is?

Brainwashed customer: SOAP what?!... Nevermind..But...I can orchestrate my stateless "processes" easily!

BPEL Zen Master: Vendors have been fooling customers into using microflows (an IBM innovation... verbiage-wise) which are nothing but stateless 'programs' that invoke (orchestrate) webservices. You're better off using a mediation module (Take that, IBM.. you can stick your microflows where the sun dont shine). Speaking about Oracle, there's really not much to speak of. After acquiring BEA, they seemed to have dumped their BPEL engine in favor of BEA's. Actually, from a quick look at their conFUSION Middleware stack, they dont know their butt from their heads. Buttheads! For the rest of us, a simple composite business service will do. Not only will you yourself save mucho $$$ by not buying the BPEL engine, but also CPU, storage space, administrator time and most importantly user frustration.

Brainwashed customer: But I will be backed by standards if I go with BPEL, wont I? What about long running processes with human interaction? Is'nt BPEL the best for this? I'm solid with the support on standards aren't I?

BPEL Zen Master: Oh yes! A million of them, non which make sense to any business analyst anyway. And here something to thing about: Any decent business process will probably require some human interation.

Well Long running with Human Interaction... Hmm dont let the BPEL vendors fool you on this one. The moment you get human interaction as part of your BPEL, you'd have to use their extensions to BPEL (which are completely proprietary).

That flushing sound you are now hearing is the standards going down the toilet.

In case your still wondering about which way to go, this should enlighten you

http://www.infoq.com/news/2009/01/WhoNeedsBPEL

BPEL Zen Master: Meditate on this koan from the one-who-has-been-enlightened

I will reproduce it here for your convenience.

Hermann Schmidt:

BPEL is advertized as a high-level process design language. That's a lie. BPEL exposes the designer to lowest-level issues like communication failures, SOAP faults, and XML unmarshalling problems. Drawing a BPEL process naively will not produce any usable artifact. It is a programmers job. Picking up the recent discussion about "BPMN is not software engineering", I think it is fair to say that BPEL is one reason why BPMN still is software engineering.

I have seen tutorials from SUN that show how to access rows in a database table with BPEL. What a brilliant idea. Let business people drag their data from databases directly! That about says it all how some vendors treat BPEL. It's just yet another programming language with a fancy graphical interface. "Hey, look what I can do with BPEL! Isn't that neat?". Hello? Service orientation?

BPEL has no modularity. BPEL has no concept of reusable patterns or code fragments (besides functional decomposition into yet more processes - did someone say high-level?). Reusing patterns in BPEL means cut&paste of sourcecode. No signs of object orientation anywhere. It is a low-level web-service caller with a functional decomposition design model and a little event dispatching with parallel execution. Oh, and you can shuffle data around in XML-based structures.

The ubiquitous GUIs give the illusion of a high abstraction. Ironically, they are necessary because BPEL code is a pain to write as text. The graphics are merely the syntactical representation of a very basic programming language.

Vendors pimp up BPEL engines with additional features to alleviate weaknesses of the language. This may even result in practically usable products, if done well. However, that makes BPEL non-portable in all but trivial cases and renders the whole standard useless.

I am deeply frustrated by the level of intelligence in this standard. I believe that it will not take the software industry any closer to more efficiency or excellence.

I don't need it. It doesn't get the job done.

Brainwashed customer: Master, that was enlightening. During my meditation my mind drifted to a scene from the movie 'The Matrix' and then... then it hit me

There is no BPEL

Hung Threads with BO SDK - Solution

2008-09-30T06:54:00.006+05:30

Many developers seem to be facing the dreaded 'Hung threads' issue when they you the BO SDK in their java applications.

The symptoms:
The issue usually manifests itself with a line in the SystemOut.log or equivalent reporting the following:
WSVR0605W: Thread "THREAD NAME : ID" (55c8824f) has been active for 600,112 milliseconds and may be hung. There are 1 threads in total in the server that may be hung.

If you do a thread dump in your App Server at this point, you will see something like so:

- waiting on <0x93d29020> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at com.crystaldecisions.thirdparty.com.ooc.OB.Downcall.waitUntilCompleted(Downcall.java:831)
- locked <0x93d29020> (a java.lang.Object)
at com.crystaldecisions.thirdparty.com.ooc.OB.GIOPClientWorkerThreaded.receive(GIOPClientWorkerThreaded.java:327)
at com.crystaldecisions.thirdparty.com.ooc.OB.GIOPClientWorkerThreaded.sendReceive(GIOPClientWorkerThreaded.java:353)
at com.crystaldecisions.thirdparty.com.ooc.OB.Downcall.request(Downcall.java:336)
at com.crystaldecisions.thirdparty.com.ooc.OB.DowncallStub.invoke(DowncallStub.java:583)
at com.crystaldecisions.thirdparty.com.ooc.CORBA.Delegate.invoke(Delegate.java:579)
at com.crystaldecisions.thirdparty.org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:125)
at com.crystaldecisions.enterprise.ocaframework.idl.OCA.OCAi._InfoStoreEx3Stub.queryEx3(_InfoStoreEx3Stub.java:62)
at com.crystaldecisions.enterprise.ocaframework.j.a(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.j.find(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.AbstractServerHandler.buildServerInfo(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.AbstractServerHandler.buildClusterInfo(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.aa.for(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.for(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.p.a(Unknown Source)
at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.getManagedService(Unknown Source)
at com.crystaldecisions.sdk.occa.managedreports.ras.internal.CECORBACommunicationAdapter.connect(Unknown Source)
at com.crystaldecisions.sdk.occa.managedreports.ras.internal.RASReportAppFactory.a(Unknown Source)
at com.crystaldecisions.sdk.occa.managedreports.ras.internal.RASReportAppFactory.a(Unknown Source)
at com.crystaldecisions.sdk.occa.managedreports.ras.internal.RASReportAppFactory.openDocument(Unknown Source)
at com.crystaldecisions.sdk.occa.managedreports.ras.internal.RASReportAppFactory.openDocument(Unknown Source)
at com.talic.pi.utils.ReportingEngine.generateReport(ReportingEngine.java:86)
at com.talic.pi.cms.dao.ReportsDAO.generateReport(ReportsDAO.java:31)
at com.talic.pi.cms.component.ReportGeneratorComponentImpl.generateReport(ReportGeneratorComponentImpl.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.ibm.ws.sca.internal.java.handler.JavaReflectionAdapter$2.run(JavaReflectionAdapter.java:152)

The cause:
This is mainly caused by reports that take a great deal of time to execute causing the application server to report that the processing thread is 'hung'. A worse side effect of this is when the application server runs out of threads and is unable to process any further requests due to these hung threads.

The solution:
The problem is due to the fact that the Corba Timeout has not been set. You've missed the clientSDKOptions.xml in your deployment. This causes the RAS SDK to probably never timeout its call to the RAS.

Create a file called clientSDKOptions.xml and place it in your application's WEB-INF/classes folder, or alternatively, you could place it in a server wide classpath folder like 'applib' on OC4J or $WASPROFILE/properties on WebSphere.
The contents of the file should be as follows:


<CrystalReports.ClientSDKOptions
 xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"
 version="2"
 xsi:type="CrystalReports.ClientSDKOptions">
   <CORBARequestTimeout>120000</CORBARequestTimeout>
</CrystalReports.ClientSDKOptions>

Restart your app and your done. Your call should now timeout within 2 minutes (something with you can change) and you can say goodbye to your Hung Threads issue.

Install and configure CVS on Solaris 10

2008-09-02T14:43:00.009+05:30

Download the appropriate binary from
http://www.nongnu.org/cvs

# create a "cvs" user. This user will be used to impersonate cvs' users

$ mkdir  /export/home/cvsroot
$ useradd -d /home/cvs -c "CVS Owner" cvs

# create the CVS repository

$ cvs -d /path_to_repository

#Add the following to /etc/services

cvspserver 2401/tcp # CVS Client/server operations
cvspserver 2401/udp # CVS Client/server operations

#Place this following line in a file called "cvs_inetd"

cvspserver stream tcp nowait root /path_to_cvs_executable -f \
--allow-root=/path_to_cvs_repository pserver

# import the service into smf

$ inetconv -f -i ./cvs_inetd
cvspserver -> /var/svc/manifest/network/cvspserver-tcp.xml
Importing cvspserver-tcp.xml …Done

#Add the following lines to /var/svc/profile/inetd_services.xml

<service name='network/cvspserver-tcp' version='1' type='service'>
<instance name='default' enabled='true'/>
</service>

#Restart the network

$ svcadm restart svc:/network/inetd:default

You're done. Your pserver should be listening on port 2401

#Script to create cvs users


#!/usr/bin/perl

$cvsroot="/path_to_cvs_root";

$user = shift @ARGV || die "cvspasswd user\n";
print "Enter password for $user: ";
system "stty -echo";
chomp ($plain = <>);
system "stty echo";
print "\n";
@chars = ('A'..'Z', 'a'..'z', '0'..'9');
$salt = $chars[rand(@chars)] . $chars[rand(@chars)];
$passwd = crypt($plain, $salt);
open(PASSWD,">>$cvsroot/CVSROOT/passwd") || die("Cannot Open File");
print PASSWD "$user:$passwd:cvs\n";
close PASSWD;
#-End

$ perl createcvspasswd.pl your_cvs_user

Improve windows networking (especially if hosting webservers)

2008-08-20T14:53:00.003+05:30

First of all, I forgive you for having to use windows to host your web server :)

Now that thats out of the way, here's how you speed up networking performance on windows (especially if you happen to host Apache, IBM HTTP Server, or any other web server on a windows machine)

Create a .reg file (eg: perf.reg) with the contents that follow:


Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"MaxUserPort"=dword:00008000
"TcpTimedWaitDelay"=dword:0000001e
"TcpMaxDataRetransmissions"=dword:00000005

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces]
"TcpAckFrequency"=dword:00000001
"TcpDelAckTicks"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters]
"EnableDynamicBacklog"=dword:00000001
"MinimumDynamicBacklog"=dword:00000020
"MaximumDynamicBacklog"=dword:00001000
"DynamicBacklogGrowthDelta"=dword:00000010
"KeepAliveInterval"=dword:00000001

Double click on the file to merge its contents into the registry. Restart the machine.
You're done.

SSH Port Forwarding, X11 fowarding, RDP tunneling and much more

2008-08-09T00:43:00.012+05:30

The only firewall port you'll ever need opened - PORT 22

SSH Tunneling/Port Forwarding

Pre requisites:

1. Putty (or an equally good ssh client that allows creation of ssh tunnels)
2. Xming (X Server)
3. root access to the ssh server (if you need to modify the /etc/ssh/sshd_config) file

On the server:
Edit /etc/ssh/sshd_config and confirm that the configuration is as below:

AllowTcpForwarding yes
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

Close the file and execute the following command (if you’ve changed the file)
$ svcadm restart ssh

SSH Tunneling is a feature that lets you create 'tunnels of TCP traffic' from the client through to the ssh server over your SSH connection.

See the diagram below:

You can create as many 'tunnels' as you like from the client. All you need to create this tunnel is:
1. The local port -> Local end of the tunnel
2. The remote host and port -> Location where you want your TCP packets to finally reach.

Note: The remote end of the tunnel is the SSH server. This implies that your communication channel is encrypted only from your client to your SSH server (which shouldn't be a problem if your SSH server and the servers you want to finally connect to are all on the same trusted network)

Now for some fun:

RDP Tunneling
Lets say you want to access the machine running XP using windows using Remote Desktop client from your local machine that's outside the firewall.

Step 1: CREATE THE TUNNEL

The general syntax is:
D:\>putty –L [local port]:[remote machine]:[remote port] [ssh server]

a. Choose a local port say 30001, for the local end of the tunnel. See that it is not already in use.
b. Execute the command to create the tunnel:
D:\>putty –L 30001:mymac:3389 mysshserver
Enter the username and password in the putty window that pops up.

You're done! You now have an encrypted tunnel from your machine to the ssh server to send tcp packets to the remote machine.

Step 2: USE THE TUNNEL

a. Connect to the remote machine from the local end of the tunnel :
D:\>mstsc –v localhost:30001

If you want to be extra naughty, you can even share you local disk drives with the remote computer ? like so:

X11 forwarding
If you want to do X11 forwarding, its really very simple.
Just ensure that the login on the ssh server doesnt have the DISPLAY environment set (in the .login/.bashrc_profile)

After that you'll need to do the following:

Now, start XMing on your desktop, open the putty session, login and type the following:
$ xclock &

You should see the XWindow pop up on your desktop.

Enjoy!

WebSphere Process Server - File Adapter errors

2008-07-24T22:06:00.004+05:30

Once you deploy a mediation module containing a file adapter on a websphere process server and turn on global security, things start going bad.

This mess could manifest itself in very many ways, some of which I encountered are listed below along with their solutions

Problem 1:
Error message : SECJ5010E: Could not create default AuthenticationToken during propagation login. The following exception occurred: com.ibm.websphere.security.auth.WSLoginFailedException: Validation of LTPA token failed due to invalid keys or token type.

Fix: Upgrade WAS to at least 6.0.2.11 (WAS is currently at 6.0.2.27)
Create an entry in Admin Console->Global Security->Addition Properties->Custom properties :
com.ibm.ws.security.createTokenSubjectForAsynchLogin=true

Problem 2:
Error message: CWWBF0060E: Cannot find a specific session EJB component.Process for the process

Fix: Go to the resource view in WID, and clean the respective EJB project of the Module. Then go back to the business integration view and 'clean' the entire module.
You might have to shutdown and restart WID after this. Rebuild the project and deploy. The error will go away.

Problem 3:
Error message: com.ibm.websphere.sca.ServiceRuntimeException: caused by: java.lang.IllegalStateException: No implementation or binding handler was found to process this interaction

Fix: Uninstall the following iFix: 6.0.2.0-WS-WPS-IFJR28515.pak
Better still, upgrade to 6.0.2.4

General Warning about iFixes:
Expect every iFix emitted by IBM to break something in the existing product. You'll often hear of iFixes for iFixes and it is not funny, especially if you are in production. Your best bet is to avoid iFixes if possible.
Better still, upgrade to 6.0.2.4

Problem 4
The file adapter (inbound) doesn't work well at all on Solaris (or any unixy systems). The file adapter doesn't know when the file transfer to the inbound folder is completed and starts reading it the moment its poll interval time is up. The adapter works well on windows as windows locks the file during transfer.

Fix: There is none at this time, unless you just want to just transfer the file to another location without parsing it. (Create a 0 byte .done file for file that is placed in the inbound folder and programatically copy the original file to the destination location. The .done file will be transferred to the archive folder (if archiving is turned on))

Check out the following book that could help you work better with WPS:

Update files in .war and .ear directly

2008-07-13T10:25:00.003+05:30

I have seen many applications (.war and .ear) that have their property files or resource bundles stored within the .war or .ear files.

When the time comes for deploying them into production you usually have to unzip the entire war/ear file, modify the property files or resource bundle file and zip the whole thing back up again. Sometimes, some lazy developers just want to sneak in a .class file too :)
Trust me, this can be a pain.

This is where a utility (open source, of course) called 7-zip (www.7-zip.org) comes in handy. It allows you to do all of the above, plus provides the best compression in this industry, does almost any compression format on the planet and the best-est of all, its free and open source.

Use it. Trust me, its great.

WebSphere Portal Server migration Issues - Part I

2008-06-14T10:42:00.008+05:30

You've got WebSphere Portal Server 6.x comfortably hooked up to Oracle / Oracle RAC and then someone wants you to move to another Oracle installation (on a different machine possibly).

IBM has told you that all you need to do is export your database and import it into the other installation.

You're confident. You try it. You do a full=y export and do a full=y import into the new database, change your datasources to point to the new oracle instances, check all the connections and everything is fine.... start your server .... and BOOM!

Part I - tells you about this stack trace that you receive:

Unable to locate default library category Error while calling a function retrieveItemsByCMId of PLS data manager.
Unable to locate webContent library category Error while calling a function retrieveItemsByCMId of PLS data manager.
javax.jcr.RepositoryException: Error while calling a function retrieveItemsByCMId of PLS data manager.

What do you do Jack? Don't raise a PMR for one. The answer is simple for this one.
You're DBA has forgotten to check the tablespaces that were involved with the Oracle Instance for the Portal.

You'd need to tell the DBA (should you be so unlucky to have one who couldn't figure this out) to create the following tablespaces BEFORE the import.:

# These are for Oracle RAC. For plain, standalone Oracle, you'd specify the datafile location after the 'datafile' keyword

create tablespace ICMLFQ32 datafile size 300M autoextend on
next 10M maxsize UNLIMITED extent management local autoallocate;

create tablespace ICMLNF32 datafile size 25M autoextend on
next 10M maxsize UNLIMITED extent management local autoallocate;

create tablespace ICMVFQ04 datafile size 25M autoextend on
next 10M maxsize UNLIMITED extent management local autoallocate;

create tablespace ICMSFQ04 datafile size 150M autoextend on
next 10M maxsize UNLIMITED extent management local autoallocate;

create tablespace ICMLSNDX datafile size 10M autoextend on
next 10M maxsize UNLIMITED extent management local autoallocate;

Run the database import again and you'll be fine.

Little's Law - Verify Performance Test Results

2008-05-30T06:44:00.004+05:30

Here is something I often use (and need to lookup) when I'm analyzing performance test reports.
Most of the time, these reports are nothing but nebulous and I have no choice but to go back to the basics.

Here's the simple procedure I use to verify the sanity of the test:
Run a step load test. Get the following data from your load testing tool (averaged) per step:
1. Transactions per second
2. Hits per second
3. Think time
4. Response time
5. Number of virtual users being simulated.
6. Throughput (bytes)

Here's where the math starts:

Using Little's Law, you then then deduce the following:
Actual Number of Users in the System = (Response time + Think time) * Transactions per second

You can also verify the page size:
Page size = Throughput / Hits per second

You can tell if there is something wrong with your test bed If the actual number of users in the system given by Little's Law is different from the number of virtual users being pumped in by your load testing tool. i.e. if the number of users simulated by the testing tool is more than what you deduce by Little's Law, your system is probably just queuing those extra users. You'll also notice your response time will start to 'knee' once this starts to happen.

The main thing here is to do this calculation for an average of all points data points per step change in number of virtual users simulated.

Change CVS default port on Solaris 10

2008-05-30T06:36:00.002+05:30

Changing the default port (2401) of CVS on Solaris 10 is real easy.

1. Edit your /etc/services file
Change the following line:
cvspserver 2401/tcp #cvs pserver
to
cvspserver {new port}/tcp #cvs pserver

2. Execute the following commands:
# svcadm disable svc:/network/cvspserver/tcp:default
# svcadm enable svc:/network/cvspserver/tcp:default

and you're done!

Find out how to install and configure CVS for Solaris 10 here.

WebSphere Process Server / Portal Server with Oracle RAC

2008-05-28T21:23:00.001+05:30

If you're thinking that you'll get out-of-the-box Oracle RAC support for an IBM WebSphere Process Server or Portal Server installlation, you're sadly mistaken.

All the scripts that are generated by the installation process (for the oracle database) are for a standalone installation and will fail or make your DBA very angry if they are run on one of the RAC nodes!

Well, there's a way out, but you'd have to edit the scripts that these monstrous products generate.

For example:
You'll find a line like this in Portal server scripts:

create tablespace ICMLFQ32 datafile '/oracle/tb/ICMLFQ32_01.dbf' size 300M reuse autoextend on next 10M maxsize UNLIMITED extent management local autoallocate;

you'll have to change it to :

create tablespace ICMLFQ32 datafile size 300M autoextend on next 10M maxsize UNLIMITED extent management local autoallocate;

i.e. remove the location of the datafile specified.

There's another thing that you'd need to do if your using Oracle RAC as the datastore for WebSphere Process and WebSphere Portal Servers... change your JDBC string.

I got this tip from the an IBM developerworks article on Oracle RAC with Process Server

Change your datasource from:


jdbc:oracle:thin:@{hostname}:{port number}:{DBName}

to:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST= myoraclehost1.ibm.com)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST= myoraclehost2.ibm.com)(PORT=1521))
(FAILOVER=on)(LOAD_BALANCE=on)
(CONNECT_DATA=(SERVER=DEDICATED)
(SERVICE_NAME=dbservice)))

I should be posting more gotchas with the IBM stack .... check the 'IBM' tags

Java - Create Zip file in memory

2008-04-12T10:45:00.005+05:30

I find myself writing and rewriting this piece of code whenever I want to zip a set of files (in memory) and return the zipped file back as an object in memory. I often use this when the user requests a download of multiple reports and the deployment environment doesn't allow for disk access.

I thought I'd post it here so that I could copy-paste it the next time I need it :) If you've stumbled upon this page, you're free to use the code below too!


private static byte[] createZip(Map files) throws IOException {
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    ZipOutputStream zipfile = new ZipOutputStream(bos);
    Iterator i = files.keySet().iterator();
    String fileName = null;
    ZipEntry zipentry = null;
    while (i.hasNext()) {
        fileName = (String) i.next();
        zipentry = new ZipEntry(fileName);
        zipfile.putNextEntry(zipentry);
        zipfile.write((byte[]) files.get(fileName));
    }
    zipfile.close();
    return bos.toByteArray();
}