Java 2 Go!

Oracle ADF with SSO – The Definitive Guide

2012-05-14T17:29:00.000-07:00

by Fábio Souza & Eduardo Rodrigues

Introduction

We know. It’s been a looooooong time again. But once you read this post, we are quite sure you’ll be happy we took the time to write it. And it’s also our very first post officially signed by 2 authors. As they say: two heads think better than one.

Recently, we’ve been challenged with the task of setting up an internal “production” environment intended to run and support some internal applications. As part of this challenge, we were required to somehow integrate this environment with our company’s LDAP and SSO solutions. Since this was the first time we were facing those requirements, we started to do a lot of research (googling) on the topics involved. For our great surprise (well... not really), we could not find one single place or document that would give us a birds-eye view in an objective and consolidated form. We did find multiple blogs and documents scattered throughout the Net, each one talking about different pieces. But we had to do all the hard work of filtering, refining and compiling all this data, discarding all wrong/outdated information while keeping only what’s really relevant, correct and up-to-date. And believe me when I say... it’s a lot of information!

That’s the motivation behind this post. Share the final outcome of this research with readers, in the form of a “step-by-step” guide, so that nobody should need to go through all the hassle again.

So, let’s cut to the chase.

Environment Scheme

This diagram tries to give a nice overview of all the pieces involved in this post.

Access Flow with SSO

SSO Configuration

This section describes the steps to integrate the existing environment with the selected SSO Solution. In this example, the SSO solution is based on the following products:

Oracle Access Manager (OAM)

Provides access control services with centralized authentication, policy-based authorizations, and auditing with rich identity administration functionality such as delegated administration and workflows. It protects resources at the point of access, delegating authentication and authorization decisions to a central authority.

Oracle Internet Directory (OID)

This is our Identity Store and plays the role of our LDAP server at the same time. This is where all users are authenticated against and their security profiles are fetched from.

Oracle Weblogic Server (WLS) with these security providers

Oracle SSO Assertion Provider

This component does the mapping between the SSO-authenticated user and his/her identity in the identity store (OID in our case)

OID Authentication Provider

This component is responsible for the communication between Weblogic and the OID server. It is able of authenticating users against OID and also of creating its JAAS security context, which will be used by JavaEE applications. Because our objective here is to integrate with SSO, this particular component will not be responsible for authentication but for the creation of security contexts only.

Oracle HTTP Server (OHS) with these modules

weblogic_module

This OHS module does the bridge between OHS and Weblogic for specified locations (URIs). It's also capable of mapping multiple Weblogic nodes (cluster) and performing simple but handy Round Robin load balancing.

mod_osso

This module is designed to work with both Oracle SSO and OAM. Based on a configured set of protected locations (URIs) it can identify when a user needs to be authenticated and then redirect the user to the SSO login page and then back to the originally requested URL.

The instructions below have been carefully written in order to isolate each of the components’ configuration. If you have a different environment but some of your components are the same, you can still follow the configuration instructions for matching components. For example, even if you are using Webgate instead of mod_osso, you can still follow the OID instructions.

At the end of this section there is a Weblogic Authentication Provider review. Many errors can be avoided by following the instructions described there. Don't skip that review!

Configuring weblogic_module

This module is usually installed and loaded out-of-the-box with OHS. It can be manually configured in file $ORACLE_INSTANCE/config/$COMPONENT_TYPE/$COMPONENT_NAME/mod_wl_ohs.conf. Bellow is a sample:


LoadModule weblogic_module "${ORACLE_HOME}/ohs/modules/mod_wl_ohs.so"



# This empty block is needed to save mod_wl related configuration from EM to this file when changes are made at the Base Virtual Host Level

<IfModule weblogic_module=>

     # list all nodes in our Weblogic cluster

     WebLogicCluster mynode1:80,mynode2:80

     # log/debug options
     # Debug ON
     # WLLogFile /tmp/weblogic.log

</ifmodule>


<Location /myapp>

     SetHandler weblogic-handler

</Location>

As you can imagine, the setup above will take care of redirecting all requests coming in OHS under /myapp to one of the nodes in our Weblogic cluster.

Configuring mod_osso

You'll need to create an intermediate text file called osso.txt with the properties bellow. All values will have to be obtained from your OAM/SSO administrator, based on the URL being protected:


sso_server_version =

cipher_key =

site_id =

site_token =

login_url =

logout_url =

cancel_url =

sso_timeout_cookie_name =

sso_timeout_cookie_key =

Encrypt (obfuscate) file osso.txt into file osso.conf by running the following command (don’t forget to include “root” at the end):
$ORACLE_HOME/ohs/bin/iasobf osso.txt osso.conf root

In file $ORACLE_INSTANCE/config/$COMPONENT_TYPE/$COMPONENT_NAME/httpd.conf under the include section there is a commented line including mod_osso.conf. Uncomment this line so that the include takes effect. Here is an example of what that line looks like:

# Include the configuration files needed for mod_osso

include "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/mod_osso.conf"

Go to $ORACLE_INSTANCE/config/$COMPONENT_TYPE/$COMPONENT_NAME and copy file mod_osso.conf from subfolder "disabled" and then edit the file like this:


LoadModule osso_module $ORACLE_HOME/ohs/modules/mod_osso.so


<IfModule mod_osso.c>


# In this example we are not using SSL, thats why OssoSecureCookies  is off

OssoSecureCookies off

OssoIdleTimeout off


# OssoIpCheck should be set to off when protected resources are accessed via proxies or VPNs

OssoIpCheck on 

# make sure file osso.conf is in the same directory as mod_osso.conf

OssoConfigFile osso.conf


# Location is the URI you want to protect.

# /myapp is set here just as an example.

# you must set your own

<Location /myapp>

   Require valid-user

  AuthType Osso

</Location>


</IfModule>

And don't forget to restart OHS! :)

If the mods are messing up the interception of requests, it might be a good idea to have the Include statement for mod_osso.conf come before the one for mod_wl_ohs.conf in httpd.conf so the mods will intercept the requests in the correct order. Unpredictable errors may occur if URLs that must be interpreted by mod_osso are being intercepted by the mod_wl_ohs first.

Creating the SSO Assertion Provider in your Weblogic Domain

Open the Weblogic Console application
Go to Security Realms > myrealm > Providers
Select New under the Authentication Providers table
Give a name for the new provider, select its type, and click OK. For example:
Name: OSSO Identity Asserter
Type: OSSOIdentityAsserter
Click OK
Click on the name of the newly added provider
In the Common tab, set the appropriate values for common parameters and set the Control Flag to OPTIONAL and then save the settings

Creating the Oracle Internet Directory Authentication Provider

The OID Authentication Provider creation is pretty simple. You just need to fill up a form with the LDAP server info (there are tons of how-tos in sites and blogs about it). The tricky part is to put it to work with an ADF application when you are not using the default “username attribute”. OK, I will translate that. Imagine that in your company “mail” is the LDAP attribute used to authenticate an user. Normally you would just need to configure the “User Name Attribute” in the OID Authentication Provider configuration screen with value “mail” but that alone won’t be enough for ADF to be aware of this change. To make it work, you must properly configure the file jps-config.xml that’s being used by your Weblogic domain, which will most likely be located in <domain_path>/config/fmwconfig. You’ll need to modify this file with something like:


(...)

<serviceInstance name="idstore.ldap" provider="idstore.ldap.provider">

   <property name="idstore.config.provider" value="oracle.security.jps.wls.internal.idstore.WlsLdapIdStoreConfigProvider"/>

   <property name="CONNECTION_POOL_CLASS" value="oracle.security.idm.providers.stdldap.JNDIPool"/>

   <!-- The lines below define which attribute will be used as username -->

   <property name="username.attr" value="mail"/>

   <property name="user.login.attr" value="mail"/>

</serviceInstance>

(...)

This same change must be replicated on each and every node when using a Weblogic cluster.

Since we’re very nice guys and our intention is make readers’ life easier, we’ve created a wlst script that should do most of the work automatically: create the Authentication Provider and configure the jps-config.xml. It can be found here and instructions may be found in comments inside the file.

Reviewing Authentication Providers in Weblogic

Oracle SSO Asserter: set as the first one in the providers list and set its Control Flag to "Optional"
OID Authentication Provider: set as the second one and set its Control Flag to "Sufficient"
Weblogic's DefaultAuthenticator: set as the third one and set its Control Flag set to "Optional"
Weblogic's DefaultIdentityAsserter: must be the last one in the providers list

The idea here is to guarantee that Oracle SSO Asserter and OID Authentication providers are triggered in the right order, before any other provider. The User/Role API in ADF is able to query data from only one provider. Oracle Platform Security Services (OPSS) initializes the identity store service with the LDAP authenticator chosen from the list of configured LDAP authenticators according with the following algorithm:

Consider the subset of LDAP authenticators configured. Note that, since the context is assumed to contain at least one LDAP authenticator, this subset won't be empty.
Within this subset, keep only the providers configured with the higher control flag. The flag ordering is: REQUIRED > REQUISITE > SUFFICIENT > OPTIONAL
Within the remaining subset, keep only the first provider configured in the context.

The LDAP authentication provider singled out will be the one chosen to initialize the identity store service.

Configuring the Application to work with SSO

Independently of which products you are using to enable SSO in your environment, you still have to configure your application to use this infrastructure. In our example we are using OAM + mod_osso, but what this section covers can be used in any environment using a Weblogic Server with the following items:

Application Development Framework (ADF) Runtime installed (for installation, check this link)
Authentication Provider
Identity Asserter

In this section we'll cover the scenario in which the authorization is defined by the application itself.

Enabling ADF Security

In JDeveloper, select "Application > Secure > Config ADF Security"
Select "ADF Authentication and Authorization" and click on finish.

You will notice that files jazn-data.xml and jps-config.xml were auto generated for you. It is really important to understand their roles so check this article from our friend, Mr. Andre Correa.

ADF Security uses jazn-data.xml to setup application authorization (or security policies). Follow the guideline below to configure it:

Create Entitlements to define access for a group of resources. For example, "Basic Access" entitlement could include "view" permission on every page/taskflow that regular authenticated users can see.
Create application roles to group users by access level. For example: "Application Users" could group all users that are granted with "Basic Access" entitlement. "Application Administrators" could group only those users granted with "Basic Access" and "Admin Section" entitlements.
Create Enterprise Roles that correspond 1:1 with LDAP groups and then associate them with your application roles. For example: Enterprise Role with name = "VP's Org" could be mapped to application role "Application Users".

It is highly recommended use of wildcards when configuring resources in jazn-data.xml. Unfortunately, this cannot yet be done through the "Overview" tab in JDeveloper. Nevertheless, it is more than worth it to manually change the XML source. For example:


<!-- Using wildcards to configure resources -->

<resources>

   <resource>

      <name>oracle.appbuild.teamcal.pageDefs.*</name>

      <display-name>All Pages</display-name>

      <description>oracle.appbuild.teamcal.pageDefs.*</description>

      <type-name-ref>RegionResourceType</type-name-ref>

   </resource>

   <resource>

      <name>/WEB-INF/.*</name>

      <display-name>All Task Flows</display-name>

      <description>/WEB-INF/.*</description>

      <type-name-ref>TaskFlowResourceType</type-name-ref>

   </resource>

</resources>

Using the Weblogic Identity Store

When we started looking for the best approach to use the JavaEE container's Identity Store, we’ve found this example:


import oracle.security.idm.IdentityStore;

import oracle.security.idm.UserProfile;

import oracle.security.jps.JpsContext;

import oracle.security.jps.JpsContextFactory;

import oracle.security.jps.service.idstore.IdentityStoreService;



// THIS IS JUST AN EXAMPLE, WE ARE NOT CONSIDERING PERFORMANCE AND THREAD SAFETY ISSUES.

private UserProfile findUserProfile(String username) throws Exception {

   JpsContextFactory ctxFactory = JpsContextFactory.getContextFactory();

   JpsContext ctx = ctxFactory.getContext();

   IdentityStoreService idstoreService = ctx.getServiceInstance(IdentityStoreService.class);

   identityStore = idstoreService.getIdmStore();

   UserProfile userProfile = identityStore.searchUser(username).getUserProfile();

}

Even though this approach works fine, it is completely tied to the underlying security implementation (JPS in our case). Fortunately, we were able to find a more elegant and easy way to achieve the same thing, using only ADF APIs:


import oracle.adf.share.security.identitymanagement.AttributeFilter;

import oracle.adf.share.security.identitymanagement.UserManager;

import oracle.security.idm.UserProfile;



// THIS IS JUST AN EXAMPLE, WE ARE NOT CONSIDERING PERFORMANCE AND THREAD SAFETY ISSUES.

private UserProfile findUserProfile(String username) {

   // UserManager is automatically configured to use the container's Identity Provider and it is not thread-safe

   UserManager userManager = new UserManager();

   AttributeFilter[] filter = { new AttributeFilter("USER_ID", username) };

   ArrayList userProfiles = userManager.getUserProfileList(1, filter);

   UserProfile userProfile = (UserProfile)userProfiles.get(0);



   return userProfile;

}

Configuring Logout

In our environment, using OAM + mod_osso, the application would have to execute a well defined procedure to perform a proper single sign-off. Thankfully, ADF can take care of all logout details by itself. Using ADF this task becomes transparent to the application. There are two ways to implement the logout:

Using adfAuthentication servlet

Call /adfAuthentication with the following parameters:
logout: must be "true"
end_url: URL where the browser will be redirected to after the logout

Programatically

import oracle.adf.share.security.AuthenticationService; import oracle.adf.share.security.authentication.AuthenticationServiceUtil; (...) AuthenticationService service = AuthenticationServiceUtil.getAuthenticationService(); String logoutURL = "/face/home/jspx"; // the end_url service.logout(logoutURL, null); (...)

Deploying the SSO-protected Application

When you deploy an application using JDeveloper Application's default settings, many things are done automatically such as database connections deployment and credentials update. This is fine when you are making local tests, but when you want to deploy to production server the situation changes. In our environment we already have a system-level DataSource setup in Weblogic for our application to use. We also have an identity store that contains all users and groups that we need. That said, our deployment is configured as follows:

Inside Application Properties, go to Deployment > Weblogic
Inside “Security Deployment Options”, select "Update weblogic-application.xml with the following..."
Check the box "Application Policies" (this will make all security policies configured in your jazn-data.xml migrates to the application server)
Uncheck "Credentials" (this will prevent the copy of your local cwallet.sso to the application server)
Uncheck "Users and Groups" (we are using the identity store already configured in WLS, there is no need to upload any local users and groups)
Uncheck "Auto Generate and Synchronize WebLogic JDBC Descriptors During Deployment" (the DataSource is already configured in the Weblogic so there is no need to deploy a new one)

Make sure that the right authentication method is configured in your application's web.xml:


(...)

<login-config>

   <auth-method>CLIENT-CERT</auth-method>

</login-config>

(...)

When you have a database connection in your project and you deploy your application with box "Auto Generate and Synchronize WebLogic JDBC Descriptors During Deployment" checked, a datasource named “<db_connection_name>DS” will be deployed together with your application.

Reference

OAM Home Page
OID Home Page
Application Development Runtime
Article about User/Role API
Article about OPSS Artifacts
OID Configuration WLST Script

Special Thanks

We would like to give special thanks to Mr. Andre Correa for his help during our research and also for kindly reviewing this post.

Web 2.0 Solutions with Oracle WebCenter 11g (book review)

2011-01-26T12:38:00.001-08:00

by Fábio Souza

Hello People! This was supposed to be a post to celebrate the new year, but, as you all can notice, the things didn't happen the way I was expecting (again haha). Today I will talk about this strategic product and (as promised) I will make my review about the book Web 2.0 Solutions with Oracle WebCenter 11g.

About WebCenter

The Goal

Today we interact with content/information on the internet not just as a spectator, but also as a collaborator: we help on the knowledge construction in a dynamic way. This can be done when you put “Like” in something that your friend is saying on the Facebook, when you write an article on Wikipedia or when you post your comment on your favorite blog. All this meta-information, in some way, helps to spread the knowledge and to make it be easier to be found.
To make this possible in the business world, we need a next-generation enterprise workplace: a portal that leverages Web 2.0 technologies, user-focused and community-centric. The goal is to tie together people, ideas, content, processes, systems, and enterprise applications.

The Product

Oracle WebCenter 11g is Oracle's strategical product to deal with E2.0. It provides services such as: discussion forums, wikis, blogs, and tagging, directly from the context of a portal or an application. It worth to mention that Oracle WebCenter is integrated and certified with Oracle Universal Content Management (UCM), Oracle Identity Management Suite, Oracle SOA Suite, Oracle Enterprise Manager, and Oracle eBusiness Suite, as well as PeopleSoft, Siebel, and JD Edwards enterprise applications.

The Components

Oracle WebCenter Framework:

It is an ADF extension that, besides including all its functionalities/facilities, contains several components that make possible (and easy) to construct a portal, as if it was a common web application. The framework uses the Oracle Metadata Services (MDS), that offers a layer to store customization data, avoiding a mix between code and metadata (The MDS is widely explored on Oracle Composer).
To put the Webcenter Framework to work, you just have to download the extension on you JDeveloper, it will make all the framework components available to be used on your own applications. Its good to remember that JDeveloper provides an integrated development environment for composite J2EE applications, business processes, BI applications, and enterprise portals.
There is some good information about JDeveloper on my previous post, check it out!

WebCenter Services:

This is the component that give access to the infrastructure that, not only turns your corporate environment into a “social network”, but take it beyond.

Social Networking Services - Enables users to maximize productivity through collaboration.
People Connection – Enables users to assemble their business networks like linked-in
Discussions - Provides the ability to create and participate in threaded discussion. This is an embedded version of Forums provided by Jive Software.
Announcements - Enables users to post, personalize, and manage announcements.
Instant Messaging and Presence (IMP) - Provides the ability to observe the online presence status of other authenticated users (whether online, offline, busy, or idle) and to contact them.
Blog - Enables blogging functionality within the context of an application. This feature comes from an embedded version of open source package jzWiki.
Wiki - Self-service, community, oriented-content publishing and sharing. This is an embedded version of open source package jzWiki.
Shared Services - Provides features for both social networking and personal productivity.
Documents - Provides content management and storage capabilities, including content upload, file and folder creation and management, file check out, versioning, and so on. This functionality requires licensing of a JCR (JSR-170) compliant document repository product.
Links - Provides the ability to view, access, and associate related information; for example, you can link to a solution document from a discussion thread.
Lists - Enables users to create, publish, and manage lists. (Available only in WebCenter Spaces)
Page - Provides the ability to create and manage pages at runtime.
Tags - Provides the ability to assign one or more personally relevant keywords to a given page or document. This feature is similar to the del.cio.us website.
Events - Provides group calendars, which users can use to schedule meetings, appointments, and any other type of team get-together. This feature requires deployment of a separate calendaring server, which may be Oracle Beehive or Microsoft Exchange. (Available only in WebCenter Spaces)
Personal Productivity Services - Focuses on the requirements of an individual, rather than a group.
Mail - Provides integration with IMAP and SMTP mail servers to enable users to perform simple mail functions such as viewing, reading, creating, and deleting messages, creating messages with attachments, and replying to or forwarding existing messages.
Notes - Provides the ability to "jot down" and retain quick bits of personally relevant information. (Available only in WebCenter Spaces)
Recent Activities - Provides a summary view of recent changes to documents, discussions, and announcements.
RSS - Provides the ability to publish content from WebCenter Web 2.0 Services as news feeds in RSS 2.0 and Atom 1.0 formats. In addition, the RSS service enables you to publish news feeds from external sources on your application pages.
Search - Provides the ability to search tags, services, an application, or an entire site. This makes use of a license limited version of Oracle's search product.
Worklist - Provides a personal, at-a-glance view of business processes that require attention. These can include a request for document review and other types of business process that come directly from enterprise applications.

The WebCenter Services benefits can be reaped by the whole Webcenter Suite Portal stack*:

Oracle Portal
Oracle WebLogic Portal
Oracle WebCenter Interaction

* There are some restrictions

Oracle Composer:

Oracle Composer is an innovative component that enables any application or portal to be customized (changes that affects the general visualization) or personalized (changes made by a person that affects the that person's own visualization) after it has been deployed and is in use. The personalizations/customizations are managed by Oracle Metadata Services (MDS). Oracle Composer is designed to be extremely easy to use so that business users and end users can edit any page with a few clicks of the mouse. Oracle Composer runs in all modern browsers and provides a rich, dynamic way to edit JSF application and portal pages.
The developers can use the Webcenter Framework to design portal applications that uses Oracle Composer.

Oracle WebCenter Spaces:

A ready-to-use application that pulls together all the WebCenter features:

Webcenter Services
Webcenter Composer
Oracle MDS
Easy to integrate with applications developed with Webcenter Framework
Easy to integrate with any other application (actually this can be achieved using the Oracle ADF Model layer)

The focus is to empower teams to quickly and efficiently manage their information, applications, projects, and people. Oracle WebCenter Spaces is prepared to work effectively with project teams and work groups, including teams that span multiple geographies or include external members.

Notes

A new release of Oracle Webcenter Suite 11g is already available. This post does not cover the new features/changes.

Web 2.0 Solutions with Oracle WebCenter 11g – Book Review

Introduction

Oracle Webcenter 11g is a very new product and because of that there are few books talking about it. Web 2.0 Solutions with Oracle WebCenter 11g is a very practical book that presents the main concepts of WebCenter. The book has around 250 pages, 10 chapters and lots of illustrations that make the reading very soft, fast and easy. There is also a source code (for download) that may help to accomplish the exercises that come with the book.

Some concepts (overview) that can be found in the book:

Web 2.0
Enterprise 2.0
Oracle WebCenter Suite
ADF Faces
ADF Business Components
Forum
Wiki
Blog
Search in enterprise applications
Organization with Tagging and Linking
RSS

Public

The book aims Java/Portal Developers who want to try the very first steps with WebCenter. The book helps to understand how the components work and how the main available tools can be used. It is a great book for those that are starting with ADF and WebCenter. If you already have tried by yourself the development of simple applications with Oracle WebCenter, certainly you will skip some chapters, but, if you have a good experience with the product, the book definitely is not for you.

What caught my attention

The book is practical guide to give the first steps in the WebCenter World
The “Hello World” application is very “sophisticated”, and its idea is to show a typical enterprise application architecture.
The book comes with a good material about Discussions, Wiki, and Blog Services

What I missed the most

Although it is not the focus of the book to go into questions of sophisticated software architecture, it could indicate sources to readers who want to take the next steps.
The book does not goes inside the ADF Task Flow subject, it only mentions it.

Chapters Overview

Chapter 1, What is Oracle WebCenter?

Introduction to Web 2.0, Enterprise 2.0 and Oracle WebCenter Suite

Chapter 2, WebCenter Installation and Configuration

Presents the "Webcenter Component Building Blocks"
Walkthrough to install the WebCenter Suite in a development environment.

Chapter 3, Setting up the Development Environment

Walkthrough to install JDeveloper + WebCenter Plugin + SVN

Chapter 4, Getting Started with WebCenter Development

Development of a sophisticated “Hello World” application, composed of a webservice, consumed by a portlet, consumed by a WebCenter application. This application gives a good idea of a real WebCenter application.

Chapter 5, Design and Personalization

Presents the ADF Faces Rich Client
ADF Faces components overview
Presents the ADF Business Components
Shows the development of a WebCenter Application:
- View Layer: Constructed with ADF Faces, it is composed by a template and a JSF page
- Model Layer: Constructed with ADF Business Components, it uses the schema that comes with the book (can be downloaded at the packt website)

Chapter 6, Portlets

Talk about the types of portlets you can build with WebCenter
Shows the portlet development with ADF and JSF Bridge
- View Layer: Constructed with ADF Faces, it is composed by a JSF page, exposed as a portlet –> The page uses ADF Data Visualization Components! Very cool!
- Model Layer: Constructed with ADF Business Components, it uses the same schema that comes with the book
After the portlet creation it is shown how to consume it

Chapter 7, Discussions, Wiki, and Blog Services

Presents the concepts about Forum, Wiki and Blog
Presents and shows how to use ADF TaskFlows to consume Discussion Services
Lists the ways to use Wiki and Blog. There are examples of two of then:
- Using an IFrame (included in a custom WebCenter application page)
- Using the Web Clipping portlet.

Chapter 8, Search Service

The chapter talks about the following concepts:
- Searching enterprise applications (in our case WebCenter applications)
- Organizing information with tagging and linking
- RSS and its importance
There is also examples using WebCenter applications to:

Integrate with WebCenter Tags Service (it uses ADF Security for Authentication/Authorization)
Integrate with the WebCenter Links Service
Integrate with the WebCenter RSS Service

Chapter 9, Personalization with Oracle Composer

Presents personalization for two different roles: developers and end users
As developers, we prepare a dynamic environment for end users:

Lists, explains and uses Composer’s components

As end user, we personalize the page

Resource Catalog Overview
MDS Overview

Chapter 10, Oracle WebCenter Spaces Overview

Presents Oracle WebCenter Personal Spaces: Practical guide to page creation
Presents Oracle WebCenter Group Spaces: Practical guide to group creation

Sources

I took lots of things from those sources, including whole stretches (they were perfect):
Getting Started with Oracle WebCenter 11g
Getting Started with Oracle WebCenter 11g – The Solution
WebCenter Services Capabilities
Web 2.0 Solutions with Oracle WebCenter 11g

Happy Holidays everyone!

2010-12-02T13:57:00.000-08:00

Creating a centered page layout using ADF Faces (only)

2010-12-01T12:07:00.000-08:00

by Eduardo Rodrigues

It’s been a long time since I last posted here about my old friend: ADF. But now that I’m working in ADF BC’s awesome development team, I was already feeling the self-pressure and urge to go back to this vast subject. Let’s start easy though.

As you all might have noticed, ADF has gone a long way since it’s early days. ADF Faces 11g is quite different from its 10g predecessor. One of the differences, besides all the rich components and built-in AJAX capabilities, is the fact that some of the components that were present in 10g have moved from ADF Faces library to Apache Trinidad (or MyFaces). One of those components was the “rowLayout”. That layout component certainly makes it very easy to center its content in relation to the entire page, both horizontally and vertically. However, using Trinidad components ad tag libraries alongside with ADF Faces, although possible, may not be always desired. So, the question is: how to achieve same layout using only ADF Faces’ layout manager?

If you’re somewhat used to the particularities of ADF Faces’ layout manager, you probably already know that setting up a container that not only stretches 100% of the page’s (or of its parent container’s) height but also centers its content vertically is not as easy and straightforward as one would think.

Thinking fast, the first approach most come up with (including myself) is this:

panelStretchLayout (only center facet enabled) > panelGroupLayout (Layout = horizontal ; Valign = middle) > panelGroupLayout (Layout = vertical ; Halign = center)

Pretty intuitive and simple, right? Yes! But the layout manager doesn’t seem to agree.

The problem with this layout structure is basically the fact that a horizontal panelGroupLayout does not support being stretched inside a panelStretchLayout facet. This is actually clearly documented: http://goo.gl/GYVud

After a lot of trial-and-fail, I eventually found a way (thanks to my good friends and ADF gurus Maiko Rocha and George Maggessy).

The layout structure is this:

panelStretchLayout (top, bottom and center facets enabled; TopHeight = 33% ; BottomHeight = 33%) > panelGroupLayout (Layout = vertical ; Halign = center)

Even if you don’t need any content going in the Top or Bottom facets, it’s important to add at least an empty panelGroupLayout to both of them just to ensure the specified 33% of height will be allocated.

To wrap up, here’s a complete source code example of a page with a page-centered login form:

<?xml version='1.0' encoding='UTF-8'?>
<jsp:root xmlns:jsp="http://java.sun.com/JSP/Page" version="2.1"
          xmlns:f="http://java.sun.com/jsf/core"
          xmlns:af="http://xmlns.oracle.com/adf/faces/rich">
  <jsp:directive.page contentType="text/html;charset=UTF-8"/>
  <f:view>
    <af:document id="d1" title="Login Page">
      <af:form id="f1" defaultCommand="#{'pgTplt:cbLogin'}">
            <af:panelStretchLayout id="psl1" startWidth="33%" endWidth="33%"
                                   topHeight="33%" bottomHeight="33%">
              <f:facet name="center">
                <af:panelGroupLayout id="pgl1" layout="vertical"
                                     halign="center">
                  <af:panelFormLayout id="pflLogin" rows="2" maxColumns="1">
                    <af:inputText id="itUsername" label="Username"
                                  value="#{Login.username}"/>
                    <af:inputText label="Password" id="itPassword" secret="true"
                                  value="#{Login.password}"/>
                    <af:spacer width="10" height="10" id="s1"/>
                    <af:commandButton text="Login" id="cbLogin"
                                      actionListener="#{Login.doLogin}"/>
                  </af:panelFormLayout>
                </af:panelGroupLayout>
              </f:facet>
              <f:facet name="bottom">
                <af:panelGroupLayout id="pgl2"/>
              </f:facet>
              <f:facet name="top">
                <af:panelGroupLayout id="pgl3"/>
              </f:facet>
            </af:panelStretchLayout>
      </af:form>
    </af:document>
  </f:view>
</jsp:root>

And this is how the page above should look like:

Our second official book review in on its way...

2010-11-24T16:19:00.001-08:00

by Fábio Souza

Hi everyone, are you doing well? I'm fine, thanks for asking.
Well, I was wondering these days : "It would be a good idea to change the topic for a little while and talk about a product". It seems that someone at Packt Publishing heard me, and, to incentive myself, they invited us to write a review of a new book .
The topic is hot: Webcenter 11g. As you all know (I hope :)) this is a strategic product that arose to compete in the Enterprise 2.0 market.

The book is:
Web 2.0 Solutions with Oracle WebCenter 11g

I'm very excited to read the book, but I can already tell you some good things about it:
- Vince Casarez is one of its reviewers
- It covers all the product's main points (installation, environment setup, custom applications, and so on)
- There is a Oracle WebCenter Spaces Overview

I hope to be posting the review soon!

Ahhh! I almost forgot (kidding, I was planning it) to say that our big friend and also architect master, George Maggessy, helped the writers with his WebCenter knowledge.

See you!

JDeveloper Overview and (book) Review

2010-07-17T14:55:00.000-07:00

Dear visitors,
I will pretend I forgot that I'm about a year without posting anything and I will go straight to the subject that I owe to you: The review of the book "Processing XML documents with Oracle JDeveloper 11g". Before start I would like to make an introduction to JDeveloper.

About JDeveloper

Introduction

JDeveloper is the Oracle's Strategic IDE. Besides being used to develop Oracle's products it is a tool that has the goal to support all Oracle's technological stack, and also the majority of Java EE patterns and derivables. For example: you can draw a BPEL flow that orchestrates several services available in Oracle Service Bus, develop WSRP portlets that will be consumed by your Oracle Portal, implement EJBs that can be easily published like Web Services, or even build an portal that follows the E2.0 trend with Webcenter ADF components.

Main features

• Integrated enterprise development environment
• Visual and declarative editors
• Complete development lifecycle coverage
• Built-in development framework
• Advanced features for Java, Database, Web services, SOA, and Web development

Being a developer, I cannot forget to mention the powerful Oracle ADF (Application Development Framework). I will not cover details but worth it is saying that part of the framework is considered a model JSF implementation -- by the way, it was donated to the Apache Foundation -- and it has a powerful abstraction of the model layer (JSR-227) that, with the JDev's drag-and-drop-declarative interface, makes the development of lowly coupled and highly cohesive applications much simpler and more productive.

JDeveloper and Netbeans

Both IDEs are very important and their development will not stop. While support for Oracle's stack will be implemented mostly in JDev, NetBeans will continue to provide a powerful open source IDE.

License:

JDeveloper is a free tool, you don't have to purchase it. Besides, there is a runtime license required for any production application if you choose to use Oracle TopLink or the Oracle ADF. This license is already included in all editions of the WebLogic Server. Nevertheless you can use JDev for development/evaluation. There is no special license for academic/open source use, but in many cases the "Free OTN License Agreement" (check it out) for JDeveloper will cover this type of usage.

This is the information that I've found in Oracle's website (see the links at the end of the article). Remember: "license things" can and do change. Always check them first before using the product.

Curiosities

For those who are willing to know something about the JDev’s origins.

Processing XML documents with Oracle JDeveloper 11g – The Book

Introduction

In my opinion, JDeveloper 11g shows that it has reached a very good maturity with regard to dealing with XML. The idea of this book is to show us several easy-to-use out-of-the-box JDev’s features while, at the same time, it exposes fundamental concepts about XML processing.

Some concepts about XML presented by the book:

DOM Parser
SAX Parser
Validation with XSD
XML navigation using XPath
XML transformation using XSLT

Some JDev's features that are also presented by the book:

XML Edition(by code and also by its own structure)
Creation of XSD using drag and drop
Wizard to generate XMLs using XSDs
Automatic XML validation (using XSD)

The book has 11 chapters and each of those has a well defined scope and they can be read independently (you don't need to follow any particular order). Generally each chapter introduces a different technology with the following order: an overview, discussion about the main features and the demonstration of a practical example. The examples are very simple and made with dozens of print-screens and source codes. Although it has about 370 pages, the big part of them are used to explain these examples, what turns the book more practical than theoretical.

Public

In my vision, the book targets to two types of public:
1 - Advanced Developers: the book can introduce technologies and may be used as a quick reference guide.
2 - Novice Developers: the book introduces fundamental concepts/tools, showing them in a practical way.

What caught my attention

The book talks about several recent technologies that are in evidence. There are chapters about DOM 3.0 LS, XSL-FO, XML conversions, and also Oracle Berkley DB. For each approach concerning XML processing, the author always uses the Oracle’s Implementation (Oracle XDK). This is very interesting because it shows us features that you don’t have when using the default interfaces.

What I missed the most

There are no details about performance or even about how to choose the right technology for the right problem. And... there is no mention of StAX.
Another point to mention is that the book doesn’t talk about the tight-coupling disadvantages that exist when the Oracle XDK implementation is used instead of the default interfaces.

Chapters overview

Chapter 1: Creating and Parsing an XML Document

Introduction to SAX and DOM

Chapter 2: Creating an XML Schema

Introduction to Schema (XSD)
How to create a schema using the IDE and its wizards
How to create XML based on a schema

Chapter 3: XML Schema Validation

How to programmatically validate a XML document using schema

Chapter 4: XPath

How to use the IDE to "query" the XML using XPath
How to programmatically "query" the XML using XPath

Chapter 5: Transforming XML with XSLT

Introduction to XSLT
How to programmatically use XSLT
How to use Oracle's XSLT extension function to call Java classes (very interesting, and seems very powerful as well)

Chapter 6: JSTL XML Tag Library

Introduction to the JSTL XML tag library
How to configure a web project that uses this tag library in JDev
Using JSTL to parse and transform a XML document

Chapter 7: Loading and Saving XML with DOM 3.0 LS

What is and how to use it
Exposes advantages of using it

Chapter 8: Validating an XML Document with DOM 3 Validation

Shows the "dynamic validation", a very relevant functionality that is part of DOM 3.0 LS. This is a very important stretch that I took from inside this chapter:

Oracle XDK 11g is the only XML API that implements the DOM Level 3 Validation specification (http://www.w3.org/TR/2003/PR-DOM-Level-3-Val-20031215/); however, some of the DOM 3 Validation features are not implemented by the Oracle XDK 11g API. A test was run by W3C to determine which of the DOM 3 Validation methods are supported by the XDK 11g implementation. You can refer to the results of the DOM3 Validation test (http://www.w3.org/2003/11/26-DOM3-Val-Oracle-Result.html) when developing a DOM 3 Validation application.

Chapter 9: JAXB 2.0

Introduction to JAXB 2.0
How to use JDev to generate the JAXB classes using a schema
Introduction to the marshalling/unmarshalling process and to the JAXB 2.0 annotations

Chapter 10: Comparing XML Documents

Overview about tools that can do XML comparison/diff
Introduces the Oracle XDK 11g. This tool provides an API to work with comparison/diff and to automate the XSL generation. The second feature is achieved based on the original XML and the XML which we want to transform the original one into. Example: XML1 has X differences comparing to XML2. Using these X differences the API knows how to create an XSL that transforms XML1 into XML2

Chapter 11: Converting XML to PDF

Overview of several tools that generate PDF files
Short explanation about XSL-FO
Demonstration on how to transform a XML into XSL-FO document using XSL
How to use the Apache FOP API to transform a XSL-FO document into a PDF file

Chapter 12: Converting XML to MS Excel

Overview about several tools to read/write XLS files
This chapter is not like the others that introduce some standard patterns and concepts. This one just shows how to use Apache POI-HSSF to work with XLS

Chapter 13: Storing XML in Oracle Berkeley DB XML
This is a very interesting chapter because BDB XML is not a subject that is commonly discussed. It's worth to know the basics because it is an efficient way (and also fast) to store and query XMLs. Main points:

Overview about Oracle Berkeley DB XML ("an embeddable XML database for storing and retrieving XML documents").
Using BDB XML from command line (basic operations)
Using BDB XML with a Java API (basic operations)

Chapter 14: Oracle XML Publisher

Overview about Oracle XML Publisher (now it is called Oracle BI Publisher) and its advantages over the Apache FOP/Apache POI (shown on chapters 11 and 12, respectively).
How to use Oracle XML Publisher + XSL-FO to generate PDF (and how it is possible to generate different formats as well)
How to create and merge XSL-FO docs
How to merge PDF docs
How to use the "Data Engine API" to create an XML document from database data

Sources:

Processing XML documents with Oracle JDeveloper 11g
JDeveloper
JDeveloper History
JDeveloper FAQ
Oracle Java Tools FAQ

How to upgrade your Dell’s BIOS directly from Ubuntu

2010-05-24T21:27:00.001-07:00

I know this post is totally off topic but I faced this same issue last week and I’m pretty sure this will be very handy for a lot of people out there. So why not share it, right?!

Many people worldwide are migrating from Microsoft Windows to Linux nowadays, specially Ubuntu, which is probably the most friendly and stable distribution currently. I’m sort of one of those people. I’ve always used Unix but mainly at my work environment. But I’ve recently decided to switch one of my desktop PCs, a Dell Optiplex 755, from a 32-bit Windows XP Professional to a brand new 64-bit Ubuntu 10.04. And so far I’m an extremely happy user, I must say (finally making full and intelligent use of my 4GB RAM and certainly much more efficient use of my Core 2 Duo CPU).

Problem was, as I was eager to get rid of my old Windows XP, I didn’t pay attention to details, such as the PC’s BIOS version. By the moment I realized it was still A14 while the most recent was A17, I had already installed Ubuntu without any dual boot option and spent hours installing cool apps and tweaking everything to my own personal taste. As you probably know, flashing the newest BIOS release from Dell without any DOS or Windows partition on the PC would now be quite a handful and would probably involve using some freeware stuff to create a DOS or Windows recovery bootable CD and then execute Dell’s flash utility from an USB drive or something like that.

As usual, I sought help from our good friend Google and found some promising blogs and forums on the subject. However, none gave me a full-circle solution. But after some deeper research, I was able to put all necessary pieces together and compile the easiest steps to flash virtually any Dell’s BIOS directly from Ubuntu’s terminal prompt and using the most reliable source: Dell’s upgrade utility itself.

So, these are the magical steps, compiled from several different forums and articles (they might work on other Linux distributions as well but I only tried on Ubuntu):

apt-get update (not always needed, but won’t hurt)
apt-get install libsmbios-bin
getSystemId (will display info about your Dell, including BIOS version, System ID and Service Tag)
Download the most recent BIOS upgrade for your system from http://support.dell.com as you would if using Windows
Execute the downloaded utility on a command line. For example: O755-A17.exe –writehdrfile
This step can be executed on any Windows machine or directly on Ubuntu using Wine (apt-get install wine).
(you can skip this step if Wine was used in step 5) FTP or copy the extracted .hdr file to your Ubuntu
modprobe dell_rbu (loads the dell_rbu driver)
trigger the BIOS upgrade using dellBiosUpdate –u –f <the .hdr file from step 5>
reboot the machine (the BIOS will be automatically upgraded during boot)

Voilá! You have safely upgraded your Dell’s BIOS directly from Ubuntu, without having to create any external boot CD or USB drive.

Hope this helps. Enjoy!

Micromanaging Memory Consumption

2010-05-12T13:09:00.000-07:00

by Eduardo Rodrigues

As we all know, specially since Java 5.0, the JVM guys have been doing good job and have significantly improved a lot of key aspects, specially performance and memory management, which basically translates into our good old friend, the Garbage Collector (a.k.a. GC).

In almost all articles I’ve read on the memory subject, including those from Sun itself, a particular comment was always present. That briefly is:

The JVM loves small-and-short-living objects.
Don’t “nullify” variables (myObject = null;) when you decide they aren’t needed anymore as a way of hinting the GC that the objects once referenced by those variables are OK to be disposed.

I guess, after reading this “message” so many times, I finally internalized it in the form of a programming style, if I may. It’s actually very simple and takes advantage of a very basic structure, which is extremely common and kind of taken for granted. I’m talking about the very well-known code block. Yes, I’m talking about those code snippets squeezed and indented between a pair of curly braces like { <my lines of code go here> }.

In general, most programmers use these structures just because they have to as they are mandatory in so many portions of the Java language’s syntax. You need them when declaring classes, methods, try-catch-finally blocks, multi-line for loops, multi-line if-then-else blocks, etc. But the detail many programmers seem to forget is that these code blocks may actually be defined anywhere in a method body, unattached to any particular keyword or command. Even more, code blocks can be nested as well.

Besides being syntactically mandatory, the use of code blocks demarcated by opening and closing curly braces also imply a very important feature of the language. Code blocks also define variables scopes! I’ll explain…

Any variable that happens to declared inside a curly-braces-pair-demarcated code block will “exist” only within the context of that particular code block (or scope). Such variables are said to be “local variables”. Actually, if we try to use a local variable outside of its scope, we’ll most certainly get a compilation error, because that variable literally doesn’t exist outside that code block (or scope) where it was declared. And right there lies the very code of this best-practice tip.

Specifying well-defined scopes for all your local variables is actually the best way of hinting the GC about what strong references are or not in use when it kicks in. Simple enough, any strong reference coming from a variable declared inside a scope that is currently not being executed, is clearly to be considered as not in use by the GC, thus increasing the chances of proper and prompt disposal of the referenced object (if no other strong references to it exist, of course).

So, in order to better illustrate my case, here is a simple example. First let’s consider this very innocent piece of code:

public class Foo
{
   public final static void main(final String args[])
   {
        try 
        {
            DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
            DocumentBuilder builder = builderFactory.newDocumentBuilder();
            URL cfgUrl = this.getClass().getClassLoader().getResource("config.xml");
            File cfgFile =  new File(cfgUrl.toURI());

            Document cfg = builderFactory.newDocumentBuilder().parse(cfgFile);
            XPath xpath = XPathFactory.newInstance().newXPath();
            Node cfgNode = (Node)xpath.evaluate("//*[local-name()='config']", cfg, XPathConstants.NODE);

            (...)
           
        } catch (Exception ex) {
           ex.printStackTrace();
        }
    }
}

If we consider that in the very first part, variables builderFactory, builder and cfgUrl are not really needed after cfgFile is instantiated, rewriting that part like this would be preferred:

public class Foo
{
   public final static void main(final String args[])
   {
        try 
        {
            Document cfg;

            {
                DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
                DocumentBuilder builder = builderFactory.newDocumentBuilder();
                URL cfgUrl = this.getClass().getClassLoader().getResource("config.xml");
                File cfgFile =  new File(cfgUrl.toURI());
                cfg = builder.parse(cfgFile);
            }

            XPath xpath = XPathFactory.newInstance().newXPath();
            Node cfgNode = (Node)xpath.evaluate("//*[local-name()='config']", cfg, XPathConstants.NODE);

            (...)
           
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

With that, when the execution is passed the red code block, all local variables declared only in that context will cease to exist for all practical means. This simple example is a mere illustration and certainly doesn’t represent any major benefit but, believe me, in a real life code, using this approach of well-defined scopes for local variables may have a significant and positive impact on your application’s memory consumption profile.

As you can see, this is indeed a very simple Java best-practice tip. It’s easy to adopt, has no collaterals whatsoever and can prove to be very powerful. So, why not use it?

Enjoy!

The X (Path) File

2010-05-07T17:42:00.000-07:00

by Eduardo Rodrigues

This week I came across one of those mysterious problems where I had some test cases that needed to verify the content of some DOM trees to guarantee that the test went fine. So, of course, best way to achieve this is using XPath queries. And because the DOM trees involved were all quite simple, I figured writing the XPath queries to verify them would be like a walk in the park. But it wasn’t.

I spent hours and hours trying to figure out what was I doing wrong, googling around but nothing seemed to make any sense at all. Then, just when I was almost giving up and throwing myself through the window, I finally realized that tiny little detail that explained everything and pointed me out to the right solution. The culprit was the default namespace specified in my root element!

Turns out, whenever a namespace URI is specified without any prefix (like xmlns=”http://foo.com/mynamespace”), this is considered to be the document’s default namespace and it usually don’t affect parsers. But, as I found out, it does affect XPath big time. XPath, by definition, will always consider namespaces, even the default one. The problem with that is, because a default namespace don’t have any specific prefix, we completely lose the ability of using the most simple and common path-like approach when writing queries to locate nodes in the DOM tree.

Here’s a very simple example that illustrates the issue very well. Consider the following well-formed XML:

<?xml version="1.0" encoding="iso-8859-1"?>
<HR Company="Foo Inc.">
    <Dept id="1" name=”Board”>
        <Emp id="1">
            <Name>James King</Name>
            <Salary>150000</Salary>
        </Emp>
        <Emp id="10">
            <Name>Jon Doe</Name>
            <Salary>100000</Salary>
            <ManagerId>1</ManagerId>
        </Emp>
        <Emp id="20">
            <Name>Jane Smith</Name>
            <Salary>100000</Salary>
            <ManagerId>1</ManagerId>
        </Emp>
    </Dept>
</HR>

If I want to check if there’s really an employee named “Jane Smith” earning a 100K salary in the “Board” department, a very simple XPath query such as “//Dept[@name='Board']/Emp[string(Name)='Jane Smith' and number(Salary)=100000]” would easily do the job.

Now just add an innocent default namespace to the root element:

<HR xmlns=”http://foo.com/HR” Company="Foo Inc.">

and try that very same XPath query that worked so well before. In fact, even the most simple of all queries – “/” – won’t work as expected anymore. That’s just because XPath considers the default namespace context and therefore requires it to be referenced in the query. We just don’t have any way of referring to that namespace in the query since it doesn’t have any prefix associated to it. My particular opinion on this issue is that it represents a huge design flaw in XPath specs., but that’s a completely different (and now pointless) discussion.

Unfortunately, there’s no magic in this case. To keep using XPath queries in this kind of situations, we need to use a more generic (and less compact) syntax where we can be more specific about when we do care about fully qualified (or expanded) names and take namespaces into consideration, or if we just care about local names but do not about namespaces. Bellow is the very same query, using this more generic syntax and these 2 different naming flavors, both providing the exact same outcome:

If you need (or want) to consider the namespace:

//*[namespace-uri()=’http://foo.com/HR’ and local-name()=’Dept’ and @name='Board']/*[namespace-uri()=’http://foo.com/HR’
and local-name()=’Emp’ and string(Name)='Jane Smith' and number(Salary)=100000]

If you just care about the elements’ names, then just remove the “namespace-uri” conditions:

//*[local-name()=’Dept’ and @name='Board']/*[local-name()=’Emp’ and string(Name)='Jane Smith' and number(Salary)=100000]

The reason why I prefer to use function local-name() instead of name() is simply because, together with namespace-uri(), this is the most generic way of selecting nodes since local-name() doesn’t include the prefix, even if there is one. In other words, even if you had a node such as <hr:Dept>, local-name() would return simply “Dept”, while name() would return “hr:Dept”. It’s much more likely that the prefix for a particular namespace will vary amongst different XML files than its actual URI. Therefore, using predicates that combine functions namespace-uri() and local-name() should work in any case, regardless of which prefixes are being used at the moment.

Enjoy!

References

The easy-small-simple-quick-step-by-step-how-to article on AspectJ you’ve been looking for is right here

2010-05-01T15:26:00.001-07:00

by Eduardo Rodrigues

That’s right. Have you ever spent hours of your precious time googling the Web trying to find an easy, small, simple, quick and step-by-step tutorial, article or sample on how to use AspectJ to solve that very simple use case where you only need to add some trace messages when certain methods of a certain library are called (and you don’t have access to its source code)? If your answer is “YES” then this post is exactly what you’ve been looking (maybe even praying) for.

For convenience, readers may download all files mentioned bellow at http://sites.google.com/site/errodrigues/Home/aspectj_sample.zip?attredirects=0&d=1

Step 0: get the latest stable version of AspectJ

AspectJ can be downloaded from http://www.eclipse.org/aspectj/downloads.php. I recommend using the latest stable release, of course. As of now, this would be release 1.6.8.

To install the package, just run “java -jar aspectj-1.6.8.jar”

Step 1: write the aspect code

Well, AspectJ doesn’t really have what I would call a very intuitive syntax so I’ll not try to explain it more than the strictly necessary for this post. A good way to start learning is reading the official documentation at http://www.eclipse.org/aspectj/docs.php.
In my case, I only needed to write 1 single and very simple aspect capable of capturing all calls to 1 particular method in 1 particular class and then log those calls when the method’s argument was equal to a particular value. In my opinion, this “trace calls” use case is the most simple, obvious and probably one of the most used for any AOP (Aspect Oriented Programming) solution. Therefore, this simple example will probably cover the requirements of most of the readers. That’s what I hope, at least. So here is my aspect FULLProjection.aj:

import java.util.logging.Logger;
import java.util.logging.Level;
import com.foo.Projection;

/**
 * Simple trace AspectJ class. In summary, what it does is:
 * Before executing any call to method setProjection() on any instance
 * of class com.foo.Projection, execute method logFullProjection() defined
 * in this aspect.
 *
 * Modifier "privileged" gives this aspect access to all private members
 * of the captured object.
 */
privileged aspect FULLProjection {

    private static final Logger logger = Logger.getLogger(Projection.class.getName());

    /**
     * AspectJ syntax: defining the execution points to be captured at runtime.
     * target(p) sets the target object to be the instance of
     * class Projection which method setProjection() is called on.
     */
     pointcut tracedCall(Projection p):
          call(void Projection.setProjection()) && target(p);

    /**
     * AspectJ syntax: defining what should run right before the
     * pointcut specified above is executed.
     * Argument p will contain the exact instance of class Projection
     * which is being executed at runtime.
     */
     before(Projection p): tracedCall(p) {
          // m_projectionName is a private member of object p.
          // that's why this aspect must be declared as privileged.
          if("FULL".equals(p.m_projectionName))
               logFullProjectionByZimbra(); // call our Java method, which does the trace
     }

    /**
     * Standard Java method to be executed.
     * Just logs the call if it came from any class under mypackage.*
     */
     private void logFullProjection()
     {
          try {
               StringBuilder c = new StringBuilder("FullProjectionCall: ");
               StackTraceElement st[] = Thread.currentThread().getStackTrace();

               for (StackTraceElement e : st)
               {
                    if(e.getClassName().startsWith("mypackage."))
                    {
                         c.append(e.getClassName())
                          .append(":")
                          .append(e.getMethodName())
                          .append(":")
                          .append(e.getLineNumber());
                         logger.log(Level.WARNING, c.toString());
                         break;
                    }
               }
          } catch(Throwable t) {
               return;
          }
     }
}

Most recent versions of AspectJ have 3 distinct modes of performing the necessary instrumentation on needed classes:

instrument the original source code directly (when available) at compile time;
weave the class files after before using or deploying them as a post-compilation step;
weave classes on demand at runtime (a specific agent must be configured when starting the JVM).

I won’t discuss the pros and cons. In my my case, option 1 was discarded because I didn’t have access to the source code and option 3 was not good because I didn’t want to mess with the container’s setup. So I chose option 2.

Step 2: instrument classes with AspectJ

Since I decided to use option 2 above, my compilation process didn’t need to change a bit. What I needed was to add a post-compile step in which the necessary AspectJ instrumentation would be performed on already compiled classes. The right tool to do that is AspectJ’s command-line compiler: ajc.

This was the hardest part for me because I could not find any tutorial or example on the Web that would show me, in a direct and simple way, how to use ajc. So, instead of making the same mistake and start trying to describe and explain the compiler tool and its options, I’ll simply paste the shell script I used in my case:

#!/bin/bash

# AspectJ's install dir
ASPECTJ_HOME=/scratch/aspectj
# Defining the classpath to be used by ajc
MYCLASSPATH=<include here all elements needed by the classes to be instrumented>

$ASPECTJ_HOME/bin/ajc -classpath $MYCLASSPATH -argfile my_aspects.cfg

The trick here was to use file my_aspects.cfg to define all other parameters to be passed to ajc. Here is its content:

-1.5          # Java 1.5 compatibility
-g            # add debug info
-sourceroots  # path containing the aspects to be compiled (FULLProjection.aj in my case)
/scratch/WORK/aspects
-inpath  # path containing the classes to be instrumented (com.foo.Projection inside platform.jar in my case)
/scratch/WORK/jlib/platform.jar
-outjar  # output JAR file (I preferred to have a separate JAR and keep the original)
/scratch/WORK/platform-instrumented.jar

This file must contain only 1 command-line option per line. When an option requires a subsequent value (like –sourceroots), the value must also be declared on a separate line, subsequent to the option line. A complete reference to ajc can be found at http://www.eclipse.org/aspectj/doc/released/devguide/ajc-ref.html.

Notice that, as long as you don’t choose the runtime instrumentation mode, there’s no need to deploy your aspects with the application. They can be completely separate from your source code as well.

Conclusion

AspectJ as well as any other AOP solution can be a very powerful and useful ally to any software developer. Even as an architectural element by itself. Its possible applications go far beyond the simple use case shown in this post. So, if you’re still undecided, give it a try. Don’t be afraid of using it when necessary and, if you have the opportunity, consider including it as an active component since the beginning of your application’s development cycle.

And remember… if possible, share your experiences with the community (in an objective and clear way, please).

Enjoy!

How to write a simple yet “bullet-proof” object cache

2010-04-29T03:20:00.000-07:00

…continued from a previous post, by Eduardo Rodrigues

As promised, in this post, I’ll explain how we solved the 2nd part of the heap memory exhaustion problem described in my previous post: the skin resources cache. This part was much trickier to fix because we couldn’t simply get rid of the custom cache as we did with the resources bundle. We needed to keep the skins cache for 2 reasons: the cached objects were not instances of java.util.ResourceBundle but a String containing a CSS and they had to be indexed by a specific combination of variables as cache keys, which include skin base name, client locale and user agent. As already mentioned before, what causes the heap exhaustion are the various different possible combinations of the components used to compose the cache keys so the challenge here was to find a safe and efficient way of dealing with any possible amount of entries in our skins cache and also ensuring its size won’t ever cause an OutOfMemoryException that would ultimately crash the JVM.

(I know this will be a rather complicated post but I’ll do my best to keep it as simple as possible.)

The very first and most obvious approach that came to mind was to implement our own “HashMap” using weak or soft references, available since JDK 1.2. In a tiny nutshell, objects of these classes wrap the access to a specified target object and they receive a special treatment from the garbage collector (GC). If the only “live” references – meaning references that are valid and currently in use, therefore are not orphans – to an object come from instances of either java.lang.ref.WeakReference or java.lang.ref.SoftReference then such objects, including the Weak/SoftReference objects referencing them, are eligible for discard even if the Weak/SoftReference objects themselves are “alive”. The main difference between a weak and a soft reference is that the former allows the GC to discard them as soon as possible while the latter causes the GC to keep them alive as long as heap space available is not critical yet.

The reason why we didn’t initially think of using class java.util.WeakHashMap is obviously due to the fact that it uses a weak reference as the map’s key and we needed it to be as the map’s value. Another detail is that we wanted to have 2 implementations in hand, one using weak and the other using soft references. That’s because we had some concerns regarding how each solution would affect the GC performance on a heavily loaded and concurrent productions environment. Actually, our concern was more with regard to the soft references. We feared that the leniency of the GC with soft references would potentially cause such objects to quickly accumulate in the heap until the critical point where the GC would have to discard them. Since the amount of soft references to be discarded tends increase fast, this could have a substantial impact on the overall cost for the GC to do its job.

So, we started implementing our first idea, which basically consisted in writing classes WeakCache and SoftCache, both implementing interface java.util.Map and using an internal java.util.HashMap<String, java.lang.ref.WeakReference or java.lang.ref.SoftReference> as backing storage. With that, we also needed to associate each new value reference with its own java.lang.ref.ReferenceQueue since this would apparently be the only way of keeping track of the target objects being discarded by the GC and, thus making it possible for us to remove the corresponding map entry as well. Hopefully, with that we’d ultimately achieve the same functionality offered by java.util.WeakHashMap but adding a soft reference flavor to it and moving the auto-discardable references from the map’s keys to its values.

Looks good. But not good enough.

There are two big issues with the approach described above:

A WeakReference can be too volatile.

In theory, even to the point where it can render our WeakCache almost useless. The main reason for this is the fact that, in our particular case, we were dealing with a web application where the view layer is practically 100% implemented as an AJAX client and the skin resources being cached are only used in the mid-tier (where the cache exists) during the user login flow, which is processed in one single request. Therefore, we can pretty much assume that once a login request is over, unless another one that works with the same just-cached skin resource is currently being processed on the same JVM, there won’t be any "hard” references left at all to that object.

If you are somewhat familiar with Java, you’ll probably agree with me that one of the most certain things about the Garbage Collector is actually its uncertainty. By definition, we can never guarantee when the GC will kick in, much less how it will behave or which objects will be actually discarded during its cycle, therefore, no code should ever rely on any particular assumption regarding any behavioral aspect of the GC. On top of that, if we look closely into the Javadoc for java.lang.ref.WeakReference we’ll also notice that there are no guarantees whatsoever on the exact moment when disposable weakly-referenced objects will in fact be discarded by the GC. In our case, in a worse case scenario, this unpredictability pretty much means that a new skin resource object that’s just been added to our WeakCache during a login request could be immediately discarded by the GC soon after the request was processed by the mid-tier. In other words, depending on the rate at which new login requests would reach the container (or JVM) where the cache lives as well as on the skin resources they would be working with, our WeakCache could tend to be empty most of the time. And that wouldn’t be a very useful cache :)

The way we solved this “minor set-back” was by adding a configurable structure to our WeakCache, which we called a “hard cache”. The idea is very simple: we just added an internal linked list and an optional argument called “hardCacheSize”. If not needed, then specifying zero as the hardCacheSize (or not specifying it at all) would basically turn the internal hard cache off. But if the hard cache is needed (and it was in our case), then a greater than zero hardCacheSize should be specified in which case, every time a skin resource is added to the cache as a weakly-referenced value in its internal HashMap or when a skin resource is found and fetched directly from the cache, a normal (“hard”) reference to the very same instance of the skin resource is also appended to the internal LinkedList until its size reaches the defined hardCacheSize at which point the first (oldest) node in the list will simply be removed to make room for the one being appended (newest), thus limiting the list’s maximum size so that it will never overflow the hardCacheSize defined. The intention here is to add a simple LRU (Least Recently Used) feature to our cache. The more an object is “hit” in the cache, the more hard references to it will exist in the LinkedList and, because its a linked list, the order in which each hard reference was added is maintained, thus ensuring that its first node will always point to the oldest reference and its last node to the newest. Having at least one hard reference to a cached object coming from the internal hard cache should be enough to prevent the high volatility issue described here.

If you’re asking why haven’t I mentioned the SoftCache here, it’s not because I forgot. Due to the nature of soft references, at least in theory, they are much less vulnerable to the issue just described.
Efficient monitoring of associated reference queues can be tricky and costly.

As I mentioned before, the Javadocs for package java.lang.ref you’ll see that the recommended way to keep track of which weakly or softly-referenced objects have been disposed by the GC is to associate a java.lang.ref.ReferenceQueue instance with the references. We have two distinct ways to work with this queue:

2.1. keep polling the queue in a controlled loop to check whether a new discarded reference is available;
2.2. “listen” to the queue by using blocking method remove([long timeout]).

Either way, if we want to keep our cache’s availability high, the smartest way of implementing a reference queue handler in our cache would probably involve the use of background threads or, since we’re talking about a J2EE application using Java 5.0, we should then make use of some executor service available in package java.util.concurrent.

Needless to say, this certainly represents an overhead both in terms of code writing and complexity.

So, while we were thinking hard to come up with the most efficient, generic and elegant way of finally implementing our weak and soft caches, Mr. Eric Chan, who is one of the main architects in Oracle Beehive team, had a very interesting breakthrough. In short terms, he thought of a very nice way of combining both WeakReference and SoftReference in our weak and soft caches so that they would provide exactly the same functionality without having to deal with those reference queues at all. Basically, instead of using a plain HashMap as our backing storage, we used a java.util.WeakHashMap in both our cache implementations. The hat trick was what and how to store things in it.

Let’s start with the easiest one…

The WeakCache

This one was simple. We just replaced the backing HashMap<Key,Value> with a WeakHashMap<Key,Value> and then we used that internal “hard cache” I mentioned above as an implicit way of controlling which entries in the internal map are still in use and which ones can be disposed. The trick here was to add hard references to the map entries’ keys in the “hard cache” instead of to the values. So, if there aren’t any hard references to a map’s key neither outside the WeakCache nor in the cache’s internal linked list, this naturally implies that the corresponding map entry in the WeakHashMap is available for disposal. To make it easier to understand, look at the diagram bellow, summarizing how the WeakCache works:

Now let’s go on to the trickier one…

The SoftCache

This one was the trickiest. We also replaced the internal HashMap with a WeakHashMap, however the mapped values are SoftReference objects. But instead of simply pointing them directly to the target object being added to the cache, they point to a wrapper class called ValueHolder. The trick here was to not only hold the map entries’ value objects but also their corresponding key objects. And what goes into the internal “hard cache” in this case are hard references to these ValueHolder instances. Therefore, because the internal “hard cache” is the only place where hard references to the softly referenced ValueHolder objects mapped in the internal WeakHashMap are being held and those ValueHolder objects hold hard references to their corresponding map keys, ultimately, we can assume that, provided no external hard references to a key K inside the map exist, the map entry corresponding to that key K will be kept in the map as long as there’s at least 1 instance of ValueHolder containing key K exists in the internal “hard cache”. Otherwise, the weakly referenced key K will be available for disposal and so will be its corresponding map entry.

Explaining the internal “hard cache”

This is merely an auxiliary structure. Its content does not represent the cache’s actual content, which will always be identical to that of the internal WeakHashMap. This, by the way, is a sine qua non condition. Other then that, the following rules apply:

when a new object is added to the cache a new reference to it is also immediately appended to the “hard cache”;
when an object which exists in the cache is requested, a new reference to this object is appended to the “hard cache”;
when the amount of elements in the “hard cache” reaches the maximum limit defined for it, then, immediately before any addition of a new reference to the “hard cache”, the oldest one (least recent) will be removed from it in order to open space to the new one (most recent);
the size limit that is imposed to the “hard cache” does not apply to the cache itself but uniquely and exclusively to the “hard cache”.

From that, we can extract this corollaries:

The “hard cache” contains only “hard” (or normal) references to objects that exist in the cache, therefore are contained in its internal WeakHashMap, in the natural and sequential order by which those references were added. Hence, the smaller the order of a reference in the “hard cache”, the least recent was its utilization.

There will never be a reference in the “hard cache” to an object that does not exist in the WeakHashMap. The contrary, however, is allowed. In other words: an object may exist in the WeakHashMap even if there aren’t any references to it in the “hard cache”. In this case, that particular object would be available for disposal if it’s not referenced at all from outside the cache.

The quantity Q of references in the “hard cache” to a single cached object may be defined as:

Q = [ 1 (object was simply added to the cache but never requested from it) , N ]
N = [ total # of requests to that object , S ]
S = maximum size defined for the “hard cache”

From the definition of Q above, in an extreme scenario, nothing prevents that during any given period of time the “hard cache” be completely filled with references to one single object A in the cache. For that, these 3 conditions must be met by a consumer application:

add object A to the cache;
make N requests for A to the cache where N = S (size limit defined for the “hard cache”) – 1;
the requests mentioned in the previous condition must be contiguous. They cannot intercalate with requests or additions of other objects.

Just to illustrate how both cache will work, consider this scenario:
Given objects A, B, C, D and E and the following sequence of operations on a cache with a maximum “hard cache” size limited to 10 elements:

   1: put ( “c” , C );

   2: put ( “a” , A );

   3: put ( “d” , D );

   4: get ( “a” );

   5: get ( “e” ); // returns null as “e” wasn’t added to the cache yet

   6: get ( “a” );

   7: get ( “d” );

   8: put ( “b” , B );

   9: put ( “e” , E );

  10: get ( “b” );

  11: get ( “e” );

  12: get ( “a” );

Then the internal state of the cache would be:

Notice that there are no references (arrows) to the map’s entry containing object C coming from the “hard cache”. Therefore, provided there aren’t any references to C from outside the cache either, if the GC was to execute now, there’s a fair chance the internal state of the cache would become like this:

Don’t forget to get the sample code

For convenience, I’ve also made sample implementations of both caches explained available for download:

Beware of the obvious fact that, being sample implementations, they’re only intended to better illustrate the concepts described here and are not at all to be used as-is in real world applications. Even though they’re very good initial templates. Some of the issues I see for a high-end real world application are:

the use of wrapper java.util.Collections.synchronizedCollections(…) as a solution for concurrent access to the internal WeakHashMap as opposed to a more fine grained and efficient use of java.util.concurrent.locks.ReentrantReadWriteLock to establish and control all needed critical regions.
the use of a java.util.LinkedList as the internal “hard cache” has the main advantage of guaranteeing constant performance for the most used operations add(…) and getFirst() but the remove(…) operation on the other hand doesn’t. Actually, this is the main issue with these samples because it tends to add significant overhead when explicitly removing any object from the cache. I don’t really have any quick and ready solution for this particular problem but it could involve using a more sophisticated data structure and/or some parallel processing to mitigate the overhead.

So, enjoy and…
Keep reading!

* credits go to Mr. Eric Chan for the conceptual design and to Mrs. Khushboo Bhatia for the implementation of both caches.

Oracle + Sun + Iron Man 2: Awesome!

2010-04-09T13:09:00.000-07:00

A cool Iron Man 2 teaser...

Don’t be smart. Never implement a resource bundle cache!

2010-03-13T17:07:00.004-08:00

by Eduardo Rodrigues

Well, first of all, I’d like to apologize for almost 1 year of complete silence. Since I’ve transferred from Oracle Consulting in Brazil to product development at the HQ in California, it’s been a little bit crazy here. It took a while for me to adjust and adapt to this completely new environment. But I certainly can’t complain, cause it’s been AWESOME! Besides, I'm of the opinion that, if there’s nothing really interesting to say, then it’s better to keep quiet :)

Having said that, today I want to share an interesting experience I had recently here at work. First I’ll try to summarize the story to provide some context.

The Problem

A few months ago, a huge transition happened in Oracle’s internal IT production environment when 100% of its employees (something around 80K users) were migrated from old OCS (Oracle Collaboration Suite) 10g to Oracle’s next generation enterprise collaboration solution: Oracle Beehive. Needless to say, the expectations were big and we were all naturally tense, waiting to see how the system would behave.

Within a week with the system up and running, some issues related to the component I work on (open source Zimbra x Oracle Beehive integration) started to pop up. Among those, the most serious was a mysterious memory leak, which had never been detected before during any stress test or even after more than a year of production, but was now causing containers in the mid-tier cluster to crash after a certain period.

After a couple days of heap dump and log files analysis, we discovered that the culprit were 2 different resource caches maintained by 2 different components in Zimbra’s servlet layer, both related to its internationalization capabilities. In summary, one was a skin cache and the other was a resource bundle cache.

Once we dove into Zimbra’s source code, we quickly realized we were not really facing a memory leak per se but an implementation which clearly underestimated the explosive growth in memory consumption that a worldwide deployment like ours has a huge potential to trigger.

Both caches were simply HashMap objects and, ironically, their keys were actually the key to our problem. The map keys were defined as a combination that included client’s locale, user agent and, in the case of the skins cache, the skin name was also included. Well… you can probably imagine how many different combinations of these elements are possible within a worldwide system deployment, right? Absolutely! In our case, each HashMap would quickly reach 200MB. Of course, consuming 400MB out of 1GB of configured heap space with only 2 objects is not very economic (to say the least).

So, OK. Great! We found our root cause (which is awesome enough in this kind of hard-to-analyze-production-only bugs). But now comes the harder part: how can we fix it?!

The Solution

First of all, it’s very important to keep this very important aspect in mind: we were dealing with a source code that wasn’t ours, therefore, keeping changes as minimal as possible was always crucial.

One thing we noticed right away was the fact that we were most likely creating multiple entries in both maps that ended up containing identical copies of a same skin or resource bundle content. That’s because our system only supported 15 distinct locales, which means, every unsupported client locale would fallback to one of the supported locales, ultimately, the default English locale. However, the map key would still be composed with the client’s locale, thus creating a new map entry, and even worse, mapping to a new copy of the fallback locale. Yes, locales and skins that had already been loaded and processed, were constantly being reloaded, reprocessed and added to the caches.

So, our first approach was to perform a small intervention with the only intention to prevent any unsupported client locale from originating a new entry in those maps. Ideally, we would want to change the maps’ key composition but we were not very comfortable with this idea, mainly because we were not sure we fully understood all the consequences of that, and fix the problem causing another was not an option.

Unfortunately, days after patching the system, our containers were crashing with OutOfMemory exceptions again. As we discovered – the hardest way – simply containing the variation of the locale component in the maps’ key composition was enough to slow down the heap consumption but not enough to avoid the OOM crashes.

Now it was time to put our “fears” aside and dig deeper. And we decided to dig in two simultaneous fronts: the skin cache and the resource bundle cache. In this post, I’ll only talk about the resource bundle front leaving the skin cache front to a next post.

When I say “resource bundle”, I’m actually referring to Java’s java.util.ResourceBundle, more specifically its subclass java.util.PropertyResourceBundle. With that in mind, 2 strange things caught my attention while looking carefully into the heap dumps:

Each ResourceBundle instance had a “parent” attribute pointing to its next fallback locale and so on, until the ultimate fallback, the default locale. This means that each loaded resource bundle could actually encapsulate other 2 bundles.
There were multiple ResourceBundle instances (each one with a different memory address) for 1 same locale.

So, number 1 made me realize that the memory consumption issue was even worse than I thought. But number 2 made no sense at all. Why have a cache that is only stocking objects but is not able to reuse existing ones? So I decided to take a look at the source code of class java.util.ResourceBundle in JDK 5.0. The Javadoc says:

“Implementations of getBundle may cache instantiated resource bundles and return the same resource bundle instance multiple times.”

Well, turns out Sun’s implementation (the one we use) DOES CACHE instantiated resource bundles. Even better, it uses a soft cache, which means all content is stored as soft references, granting the garbage collector the permission to discard one or more of its entries if it decides it needs to free up more heap space. Problem solved! – I thought. I just needed to completely remove the unnecessary resource bundle cache from Zimbra’s code ant let it take advantage of the JVM’s internal soft cache. And that’s exactly what I tried. But, of course, it wouldn’t be that easy…

Since at this point I already knew exactly how to simulate the root cause of our problem, I started debugging my modified code and I was amazed when I saw that the internal JVM’s cache was also stocking up multiple copies of bundles for identical locales. The good thing was that now I could understand what was causing #2 above. But why?! The only logical conclusion was, again, to blame the cache’s key composition.

The JVM’s resource bundle cache also uses a key, which is composed by the bundle’s name + the corresponding java.util.Locale instance + a weak reference to the class loader used to load the bundle. But then, how come a second attempt to load a resource bundle named “/zimbra/msg/ZmMsg_en_us.properties”, corresponding to en_us locale and using the very same class loader was not hitting the cache?

After a couple hours thinking I was loosing my mind, I finally noticed that, in fact, each time a new load attempt was made, the class loader instance, although of the same type, was never the same. And I also noticed that its type was actually an extended class loader implemented by inner-class com.zimbra.webClient.servlet.JspServlet$ResourceLoader. When I checked that code, I immediately realized that class com.zimbra.webClient.servlet.JspServlet, which itself is an extension of the real JspServlet being used in the container, was overriding method service() and creating a new private instance of custom class loader ResourceLoader and forcefully replacing the current thread’s context class loader with this custom one, which was then utilized to load the resource bundles.

My first attempt to solve this mess was to make the custom class loader also override methods hashCode() and equals(Object) so they would actually proxy the parent class loader (which was always the original one that was replaced in method service()). Since the web application’s class loader instance would always be the same during the application’s entire life cycle, both hashCode and equals for the custom loader would consistently return the same results at all times, thus causing the composed keys to match and cached bundles to be reused instead of reloaded and re-cached. And I was wrong once again.

Turns out, as strange as it may look at first sight, when the JVM’s resource bundle cache tries to match keys in its soft cache, instead of calling the traditional equals() to compare the class loader instances, it simply uses the “==” operator, which simply compares their memory addresses. Actually, if we think more about it, we start to understand why they implemented this way. Class loaders are never expected to be constantly instantiated, over and over again, during the life cycle of any application, so why make an overhead method call to equals()?

Finally, now I knew for sure what was the definitive solution. I just needed to transform the private instances of ResourceLoader into a singleton, keeping all the original logic. Bingo! Now I could see the internal bundle cache being hit as it should be. Problem solved, at last!

At the end, after having completely removed the custom resource bundle cache implemented in Zimbra’s servlet layer and performed the necessary changes to make Zimbra take full and proper advantage of the built-in bundle cache offered by the JVM, instead of wasting a lot of time and memory reloading and storing hundreds of instances of resource bundles, mostly multiple copies of identical bundles, I could now confirm that despite all different client locales coming in clients' requests, the JVM’s bundle cache was holding no more than those corresponding to the 15 supported locales. With that, we had finally fixed the memory burning issue for good.

Conclusion

As this article’s title suggests, don’t try to be smarter than the JVM itself without first checking whether it’s doing it’s job well enough or not. Always do carefully read the Javadocs and, if needed, check your JVM’s source code to be sure about its behavior.

And remember…. never implement a resource bundle cache in your application (at least if you’re using Sun’s JVM) and be very careful when implementing and using your own custom class loaders too.

That’s all for now and…
Keep reading!

Our first official book review in on its way...

2009-04-09T18:24:00.007-07:00

by Eduardo Rodrigues

Great news!

We have been contacted (surprisingly) by Packt Publishing a few days ago and asked if we would be interested in read one of their new books and then publish an independent review here in the blog. We were very honored and flattered with the offer, so we obviously accepted it.

The book is "Processing XML documents with Oracle JDeveloper 11g" by Deepak Vohra.

One good thing that immediately caught my attention as soon as the book arrived today was seeing the name "Frank Nimphius" in the list of reviewers. This is certainly a good sign :)

Anyway, Fabio and I already have the book and we'll both read it and then compile our comments in a 4-hand post to come. We hope it wont take too long (and also hope this will be just the first of many other book reviews to come).

Cheers and keep reading!

Java and the Timezones

2009-03-24T11:07:00.012-07:00

by Fábio Souza

Timezone, when using Java, can be something tricky. You can get surprised when you change your OS timezone (something common to countries with DST) and it doesn't take any effect on your Java programs. What does Sun have to say about that? It can be found at their FAQ:

Do my operating system's timezone patches fix the Java platform's timezone data?

No. The Java SE platform's timezone data is not read from the local or host operating system. The Java SE platform maintains a private repository of timezone data in locally installed files (.../jre/lib/zi) as part of the Java Runtime Environment (JRE) software. Applying whatever operating system timezone patches (for example Solaris OS, Linux, Windows) will have no effect on the accuracy of the Java SE platform's timezone data.

Sun recommends you to maintain all JVM up to date to avoid Timezone problems. To specific cases where a Timezone update is needed, they provide a tool called TZUpdater. TZUpdater is a JAR that contains timezone informations provided by Olson Timezone Database. To update the JVM's timezone you need to execute TZUpdater's most recent version like this:

java -jar tzupdater.jar -u (don't forget to check if you have read/write access to Java's directory structure)

We are ok about Sun's JVM... But the solution is the same to all JVMs? Not really =), but it is very similar to IBM JVM and Oracle JRockit. In all cases we either have to download a program and execute it to update the installed JVM or wait for the next JVM update release.

Here are some useful links:

Sun:

About Java and Timezone

Timezone Data Versions in the JRE Software

Sun's tzupdater download

Sun's tzupdater guide

Sun's FAQ

IBM (thanks to my friend and co-worker Frederico Hakamine):

Olson x JTZU Releases

Download JTZU

JTZU Guide

Oracle:

Oracle JRockit's tzupdater download (this is the JRockit's download page. You can find the tzupdater there)

Oracle JRockit's tzupdater guide

Web Component Developer Exam (SCWCD) for JEE 5

2009-02-10T11:38:00.006-08:00

by Wellington Carvalho

I hope my feedback can help those who want to prepare for The Web Component Developer exam (SCWCD) for JEE 5. I will use the same structure of an older post that Fabio Souza wrote to pass his own experience with SCJP 5 exams.

Main reading resource

Head First Servlets and JSP: Passing the Sun Certified Web Component Developer Exam

The first edition of the book deals with J2EE 1.4 spec, but the changes in JEE 5 were
most related to EJB 3, so I think you won’t have troubles with the exam! But if you want, the second edition of the Head First Servlets and JSP is prepared for JEE 5 and you can find it here: http://oreilly.com/catalog/9780596516680/?CMP=AFC-ak_book&ATT=Head+First+Servlets+and+JSP

Main resources for exercises

I used an application similar to Whizlabz named Enthuware. Enthuware is an exercise tool with many tests that covers all the objectives mentioned by Sun. The exercises of Enthuware are very similar to the exam, and you have good explanations, no matter the answer is right or wrong.

In order to get a better performance in your study you should take the standard tests of Enthuware sequentially and exclusively when you have read the entire SCWCD book.
To evaluate your chapter’s knowledgement you should take the objective-wise tests only after reading the chapter. To get the full version of Enthuware you will spend around US$29,00 which is cheaper than the other: Whizlabz (US$74,95). If you fail the SCWCD exam the company guarantee your money back (must get at least 70% in the tests).

My study schedule

It took me 35 days to read all the 14 chapters and to do all the tests
and exercises of Enthuware. I spent 2 hours and a half per day to read the chapter and 1 hour to do the exercises. In the week before the exam I just took the standard tests and reviewed the material.

I think this schedule is reasonable to people that is working with JEE, however if you aren’t using JEE in your daily work you should reserve a longer study time.

The exam

In my opinion the exam is relatively easy to people that have work experience in JEE development, however you need to pay attention to every detail because the exam has good tricks hidden in the code. But don’t get scared. Most of them are covered by Enthuware.

To take this exam you must know about all the objectives listed by Sun, but with a special care on these below:

Servlet Technology Model
Expression Language (EL)
Custom Tag Library - Classic Model
JSP Standard Actions

It is important to remember that you need at least 70% to pass, but trust me (and Enthuware), if you start scoring more than 70% in the standard tests, you will pass easily.

SCJP 6 Upgrade Exam (310-056)

2008-11-24T11:14:00.018-08:00

by Fábio Souza

Guys and girls,

I guess the best way to start this post is with my appologies. I really wanted to post it earlier, but you all know how things are =P.

It has been almost three months from when I took the SCJP 6 upgrade exam (310-056). The exam follows the same SCJP 5's (310-055) difficulty level where it's needed to pay attention to every detail. There is time enough to do everything calmly but the tricks are there like they were in SCJP 5. The "secret" is still the same: MAKE LOTS OF EXERCISES. In my case, my knowledge on 310-055 was pretty sharp and I just had to work out the new features of Java 6. Lets talk about the exam and how to go for it.

PS: To get your knowledge on 310-055 "pretty sharp", I recommend my previous post: SCJP5 and 6

Technical information:

Number of Questions: 48
Score needed: 66% (32 of 48 questions)
Time limit: 150 minutes

Resource material:

For those who don't have (or want to have) specific SCJP 6 material (the same of the previous post):

JDK 6 Documentation
What's New in Java SE 6
Java SE 6 Key Features
Java SE 6 Top Ten
Notable changes in the Java SE 6 Core Libraries
SCJP Sun Certified Programmer for Java 5 Study Guide (Exam 310-055), by Katherine Sierra (Author), Bert Bates (Author)

For those who want to buy new material:

SCJP Sun Certified Programmer for Java 6 Exam 310-065, by Katherine Sierra (Author), Bert Bates (Author)^*

* I've never read this book but if the quality is the same of the previous version, there is no doubt it should work.

I made an abstract to help me with the SCJP 6 content by comparing the requisites for both exams (310-055 and 310-065), and studying the Java 6 new features. Here it is:

- Ordered and Sorted comparison:

- NavigableSet and NavigableMap methods comparison:

- Other things:

Deque: in this interface's javadoc there is a full explanation about the different methods that can be used to add/remover an object of this queue. Be sure that you know the ones that returns an special value (generally null) or that throws an exception instead, when something goes wrong.

ArrayDeque: the best (fastest) way to make FIFO and LIFO queues. This is also the recommended implementation when we need a Stack.

LinkedList: choose this implementation when objects should be added/removed between the first and the last element.

java.util.Collections: there are two new methods in this class, newSetFromMap and asLifoQueue, learn them.

java.util.Arrays: avoid System.arraycopy, use the new methods Arrays.copyOf or Arrays.copyOfRange instead.

java.io.File: getFreeSpace, getTotalSpace, setExecutable, setWritable, setReadable. These are new methods to learn.

java.io.IOException: there are new constructors that enable exception chaining IOException(String,
Throwable) and IOException(Throwable).

java.io.Console: this is a new class and its instance should be got using System.console. I think all of its methods are importants, take a look at the javadoc. Pay attention to the Console.readPassword, it will fill an array with the provided password, it is a good think to know how to clean this array. This is a good way: java.util.Arrays.fill(src, '\0') and this is a valid and bizarre way: java.util.Arrays.fill(src, 0, src.length, (char)0).

SCJP 6 - To do or not to do

Because the difference between Java 5 and Java 6 is very little, in my opinion, people who already have the SCJP 5 don't need to worry about SCJP 6. For those who don't have SCJP 5 or any SCJP certificate at all: DON'T DO the Java 5 exam; GO STRAIGHT to Java 6.
In my exam (just to freeze: 310-056), the Java 6 content was kind of ridiculous. Only THREE specific questions. It means (this is not an advice) I could take the SCJP 6 exam without any knowledge about its specific content. I'll tell you how it was like:

java.io.Console: A question about its usage. It was necessary to know how to get the Console instance and how to get typed lines.

java.util.NavigableSet and the implementation java.util.TreeSet: The question was about how to use them focusing the method subset.

Chosing the best Collection implementation: I don't remember exactly what was asked, but keep this in mind:

Stack or queue needed: use ArrayDeque.

Removing/Adding elements that might not be at the top or at the bottom: LinkedList.

Sorted Collection (unique elements): TreeSet.

That is all folks! Please send your comments, corrections and share your own experiences.

SCJP 5 and 6

2008-08-08T04:41:00.007-07:00

by Fábio Souza

OK, I know that is a little late to talk about SCJP 5 but I will give my feedback about the exam. This will help people that feels unprepared to take SCJP 6 (that was my case) and the ones that are going to take this new version, what I know is that both exams (5 and 6) are very similar.

Resources:

Main resource material: SCJP Sun Certified Programmer for Java 5 Study Guide (Exam 310-055), by Katherine Sierra (Author), Bert Bates (Author). This book is just great! The authors are very careful with the explanations and they also made very good illustrations. Each chapter ends with the "Two-minute Drill" section and the exercises. The "Two-minute Drill" has an abstract about the chapter that makes very easy to review forgotten items. The exercises are very well written, and every answer is explained (both wrong and right).
Main resource material for exercises: MasterExam. My SCJP Sun Certified Programmer for Java 5 Study Guide (Exam 310-055) book came with a CD containing the MasterExam software. This software gives two ways to do its questions, like the real exam or "ad-hoc". The MasterExam's questions are like the book. A "permission" to get a bonus exam also come with the CD, you just need to register on their site and then download the program.
Other resources for exercises that I found:
- Whizlabs: There is a software like MasterExam made by Whizlabs. It seems to be a very good software. Its interface is great and you have the possibility to do "adaptive" tests. I downloaded a trial version from their website and I liked it pretty much.
- EPractize: Their software is like the MasterExam too but it doesn't have any strength point. I didn't like the software's interface. I also downloaded a trial version from their website.
- "TestKiller": This one is an exercise's book and seems to be made by Troytec. It's a very weird book, with a lot of writing mistakes on it. The strength point is that it promises to cover only questions from the real exam. When I took the exam I had the feeling of recognizing one or two questions (I took a look on 60 from 224 questions of this book).

My study schedule: That was the way I worked out to pass the exam.

8 months to read the SCJP Sun Certified Programmer for Java 5 Study Guide (Exam 310-055) and to do all exercises. It took me something like one hour per day (only workdays). It's a good way to start because you can understand how Java works without pushing yourself.
1 month to read again all "Two-minute Drill" sections and the most complicated chapters (chapters 7, 9, 8, 6, in descending difficulty order, in my opinion).
4 days (eight hours per day) to do A LOT of exercises. This is the main point. The exercises are full of tricks and the only way to become familiar with those tricks is doing a lot of exercises.

The exam: In my opinion, the exam is difficult. You have to be really confident before taking it. A good thing that I noticed was that the time is enough to do everything with attention and patience. In my opinion to take this exam you must know about everything, but with a special care with these points below:

Threads.
Generics.
Methods Override/Overload (and covariant return).
Box/Unbox.
Method calling with widening and autoboxing.
Access Control.
java and javac commands.
Differences between collections (Ordered, Sorted, Hash).
How equals and hashCode works with collections.
Inner Classes.

Differences between SCJP 310-055 and 310-065 (you can find it here):

Questions concerning System.gc() have been removed.
Coverage of the java.io.Console class has been added.
Coverage of navigable collections has been added.
Several of the previous objectives have been strengthened (so you can expect more questions and more detailed questions on them). These strengthened objectives include: exception handling, collection classes and collection interfaces, assertions, threads, and flow control.
Number of questions: 310-055 = 72; 310-065 = 72
Pass score: 310-055 = 59%; 310-065 = 65%
Time limit: 310-055 = 175 minutes; 310-065 = 210 minutes

Advice: If you aren't feeling confident to take the SCJP 6 exam just because you don't know what is new in this version, don't take the SCJP 5. I did it and I am a little regretful. There is no "SCJP 5 to 6" exam, there is only one "upgrade" exam to version 6 called "Exam 310-056". In this exam you have to prove your Java 5 knowledge again. Well, soon I will take the 310-056 and then I will post about it.

Please comment and share your experiences with SCJP (any version is welcome :)).

Links:

There's a cook in JDev's development team indeed

2008-07-08T11:25:00.003-07:00

Sometime ago I was surprised with a peculiar "tip of the day" which simply mentioned a traditional angel cake recipe.

Well, today I got the confirmation. There's certainly a cook amongst JDev's developers!

Look at the "tip" showed to me today:

Hmmmm... interesting... :)

Windows XP SP3 crashing JDeveloper 10.1.3.3

2008-05-07T17:35:00.004-07:00

by Eduardo Rodrigues

Attention!!!

Today, my Windows XP automatically upgraded to SP3 and, after rebooting, my JDev 10.1.3.3 began to crash during its initialization, generating a JVM core dump in directory <JDEV_HOME>\jdev\bin.

After a quick look into the dump file, I noticed some problem with VFS, so I decided to disable VFS commenting the following line in file <JDEV_HOME>\jdev\bin\jdev.conf:

#AddVMOption -DVFS_ENABLE=true

Then, thank God, JDev came back to life again. So, don't panic.

Cheers!

OracleAS Java Object Cache

2008-04-25T07:28:00.021-07:00

by Fábio Souza

Before I start, I would like to say that many texts and observations were took from the Oracle® Containers for J2EE Services Guide 10g (10.1.3.1.0), Chapter 7.

What is it?

The Java Object Cache (JOC) is a OC4J service that makes cache (in memory or disk) of any kind of Java object.

Use it:

To store frequently used objects.

To store objects that are costly to create/retrieve.

To share objects between applications.

Characteristics:

By default, cached objects are stored in memory but disk persistence can be configured.

By default, cached objects are local. The DISTRIBUTE mode can be configured.

By default, the "write lock" is disabled.

Cached objects doesn't have a "read lock".

Cached objects are represented by name spaces.

Cached objects are invalidated based on time or an explicit request.

Cached objects can be invalidated by group or individually.

Each cached Java object has a set of associated attributes that control how the object is loaded into the cache, where the object is stored, and how the object is invalidated.

When an object is invalidated or updated, the invalid version of the object remains in the cache as long as there are references to that particular version of the object.

The Java Object Cache organization:

Cache Environment. The cache environment includes cache regions, subregions, groups, and attributes. Cache regions, subregions, and groups associate objects and collections of objects. Attributes are associated with cache regions, subregions, groups, and individual objects. Attributes affect how the Java Object Cache manages objects.

Cache Object Types. The cache object types include memory objects, disk objects, pooled objects, and StreamAccess objects.

This table gives a little description about the cache environment and the object types.

JOC API:

Only "cache.jar" must be added to application's classpath to start with JOC API. This archive is located in $ORACLE_HOME\javacache\lib\cache.jar. A project in JDeveloper just needs to import the built-in "Java Cache" library.

Distributed Cache Characteristics:

The cache management is not centralized. Cache updates and invalidation are propagated to all application server nodes.

The OAS' JOC configuration of each OAS node is not propagated to other nodes.

The distributed objects must be in the same namespace in each node.

Distributed Cache Configurations:

There are two ways to configure JOC: programmatically (using the oracle.ias.cache.Cache class) and through the javacache.xml file. This post only covers javacache.xml.

Configuring the javacache.xml file:

The javacache.xml localization is specified in the server.xml OC4J configuration file (tag: "<javacache-config>"). To work with javacache.xml it is necessary to start the container with the -Doracle.ias.jcache=true property.
To run JOC in "distributed mode", <communication> tag must be configured in the javacache.xml file like the example below:

<communication>  <isdistributed>true</isdistributed>  <discoverer ip="192.168.0.2" port="7000"> <discoverer ip="192.168.0.3" port="7000"> </communication>

Configuring the distributed cached objects:

There are three important attributes that we can use when configuring objects:
DISTRIBUTE: used to mark objects that will be shared between applications.

SYNCHRONIZE: used to allow "write locking" on cached objects.

SYNCHRONIZE_DEFAULT: the same use as the SYNCHRONIZE attribute, but it can mark only a region or a group. When they are marked, all objects will be allowed to use the write lock.
The example below shows a distributed cache utilization:

public void defineRegions() { // Creating attributes Attributes remoteRegionAttr = new Attributes(); /* * Extracted from Attributes.setFlags() javadoc: * specifies which of the listed attributes should be set in the Attributes object. * The flags may be OR'ed together, i.e., Attributes.DISTRIBUTE | Attributes.SPOOL. * Any previous settings will be disregard. */ remoteRegionAttr.setFlags(Attributes.SYNCHRONIZE_DEFAULT | Attributes.DISTRIBUTE); try { // Creates the "RemoteRegion" with the specified attributes. CacheAccess.defineRegion("RemoteRegion", remoteRegionAttr); } catch (Exception e) { e.printStackTrace(); } } public void createCachedObject(Object object) { CacheAccess access = null; try { // Accessing the created region. access = CacheAccess.getAccess("RemoteRegion"); // Owning cache object's write lock (5 seconds to timeout) access.getOwnership("CachedObject", 5000); // Caching the object access.put("CachedObject", object); // Releasing cache object's write lock (5 seconds to timeout) access.releaseOwnership("CachedObject", 5000); } catch (Exception ex) { ex.printStackTrace(); } finally { if (access != null) { /* * Extracted from CacheAccess.close() javadoc: * releases the resource used by current CacheAccess object. * Application is required to make this call when it no longer need this CacheAccess instance. */ access.close(); } } } public Object retrieveCachedObject() { Object object = null; CacheAccess access = null; try { access = CacheAccess.getAccess("RemoteRegion"); // Accessing the created region. /* * Extracted from CacheAccess.get() javadoc: * returns a reference to the object associated with name. * If the object is not currently in the cache and a loader object has been * registered to the name of the object, then the object will be loaded into * the cache. If a loader object has not been defined for this name, then, for * DISTRIBUTE object, the default load method will do a netSearch for the object * to see if the object exist in any other cache in the system; for a non * DISTRIBUTE object, an ObjectNotFoundException will be thrown. The object * returned by get is always a reference to a shared object. Get will always * return the latest version of the object. A CacheAccess object will only * maintain a reference to one cached object at any given time. If get is called * multiple times, the object accessed previously will be released. */ object = access.get("CachedObject"); // It's not necessary to own the writing lock to read cached objects. } catch (Exception ex) { ex.printStackTrace(); } finally { if (access != null) { /* * Extracted from CacheAccess.close() javadoc: * releases the resource used by current CacheAccess object. * Application is required to make this call when it no longer need this CacheAccess instance. */ access.close(); } } return object; }

Concurrent access sensible applications may need read and write synchronism. Because JOC only offers write lock, I had to "force" read lock. This is the solution I found:

/** * This method reads and updates a cached object forcing synchronizing both to * reading and writing. * If it is being executed in multiple threads, each thread will only be able to * read the object when another thread release it's writing lock. The writing lock will * be released after a thread finishes it's reading and writing. */ public void synchronizedReadUpdateCachedObject() { CacheAccess access = null; try { access = CacheAccess.getAccess("RemoteRegion"); access.getOwnership("CachedObject", 5000); Object object = access.get("CachedObject"); System.out.println(object.toString()); Object newCachedObject = "New Cached Object"; // This is the method used to update cached objects. access.replace("CachedObject", newCachedObject); access.releaseOwnership("CachedObject", 5000); } catch (Exception ex) { ex.printStackTrace(); } finally { if (access != null) { access.close(); } } }

Declarative Cache:

The JOC offers a way to configure its regions, subregions, groups and objects using XMLs. To use this feature the "preload-file" tag must be added to the javacache.xml. This tag will point to the XML with the environment definitions. More information at: http://http//download.oracle.com/docs/cd/B32110_01/web.1013/b28958/joc.htm#i1085809

CacheWatchUtil:

By default, the Cache Service provides the CacheWatchUtil cache monitoring utility that can display current caches in the system, display a list of cached objects, display caches' attributes, reset cache logger severity, dump cache contents to the log, and so on. It depends on the "dms"library. To execute:

java -classpath $ORACLE_HOME\lib\dms.jar;$ORACLE_HOME\javacache\lib\cache.jar oracle.ias.cache.CacheWatchUtil -config=<path_to_javacache.xml>

More Oracle Application Server 10g caches (entirely took from OC4J documentation):

Oracle Application Server Web Cache. The Web Cache sits in front of the application servers (Web servers), caching their content and providing that content to Web browsers that request it. When browsers access the Web site, they send HTTP requests to the Web Cache. The Web Cache, in turn, acts as a virtual server to the application servers. If the requested content has changed, the Web Cache retrieves the new content from the application servers.

The Web Cache is an HTTP-level cache, maintained outside the application, providing fast cache operations. It is a pure, content-based cache, capable of caching static data (such as HTML, GIF, or JPEG files) or dynamic data (such as servlet or JSP results). Given that it exists as a flat content-based cache outside the application, it cannot cache objects (such as Java objects or XML DOM—Document Object Model—objects) in a structured format. In addition, it offers relatively limited postprocessing abilities on cached data.

Web Object Cache. The Web Object Cache is a Web-application-level caching facility. It is an application-level cache, embedded and maintained within a Java Web application. The Web Object Cache is a hybrid cache, both Web-based and object-based. Using the Web Object Cache, applications can cache programmatically, using application programming interface (API) calls (for servlets) or custom tag libraries (for JSPs). The Web Object Cache is generally used as a complement to the Web cache. By default, the Web Object Cache uses the Java Object Cache as its repository.

A custom tag library or API enables you to define page fragment boundaries and to capture, store, reuse, process, and manage the intermediate and partial execution results of JSP pages and servlets as cached objects. Each block can produce its own resulting cache object. The cached objects can be HTML or XML text fragments, XML DOM objects, or Java serializable objects. These objects can be cached conveniently in association with HTTP semantics. Alternatively, they can be reused outside HTTP, such as in outputting cached XML objects through Simple Mail Transfer Protocol (SMTP), Java Message Service (JMS), Advanced Queueing (AQ), or Simple Object Access Protocol (SOAP).

References:

Oracle Application Server Containers for J2EE Services Guide 10g Release 3 (10.1.3)
Oracle Application Server 10g (10.1.2) JOC Tutorial
Oracle Application Server 10g (10.1.2) JOC javadoc

Diagnostics beyond OracleAS Control Console

2008-04-23T07:04:00.005-07:00

by Eduardo Rodrigues

One of the many companies recently acquired by Oracle is a small one called "Auptyma", whose founder and former CEO, Mr. Virag Saksena, was previously Director of the CRM Performance Group at Oracle. It's main contribution for Oracle's fast growing product line was its "Java Application Monitor" which is now part of the "Oracle Fusion Middleware Management Packs" (that's where http://www.auptyma.com now takes us to) and was renamed to "Oracle Application Diagnostics for Java" or simply Oracle AD4J (although it's also referred as JADE as in Java Application Diagnostics Expert). Holy alphabet soup!

The "new" component integrates the Oracle Enterprise Manager 10g Grid Control suite but it also lives as a separate product, which is great, specially for us developers. It can be downloaded from http://www.oracle.com/technology/software/products/oem/htdocs/jade.html. The installer is very small and also very easy and intuitive.

AD4J may seem awkward at first, when you notice the fact that it also installs and uses the old Apache JServ and a PostgreSQL database. But it's worth it!

I've tried AD4J with my OC4J 10.1.3.3 standalone and I can say it works pretty well and is certainly a much more interesting and useful diagnostics tool than OracleAS 10g Control Console with JVM metrics enabled. My only extra work was to reinstall JDK 1.5.0_14 and reconfigure OC4J to use it (I was already update 15) because it's the most recent JDK 1.5.0 update currently supported by AD4J.

Its main features includes:

Production diagnostics with no application instrumentation, saving time in reproducing problems.

Visibility into Java activity including in-flight transactions, allowing administrators to proactively identify issues rather than diagnosing after-the-fact (application hangs, crashes, memory leaks, locks).

Tracing of transactions from Java to Database and vice-versa, enabling faster resolution of problems that span different tiers.

Differential heap analysis in production applications.

Possibility to setup alerts based on configured thresholds and forward them using SNMP (which means potential integration with other enterprise monitoring products like OpenView, for instance)

As you can see, it's a pretty interesting, yet small and simple, diagnostic and monitoring tool for Java applications. You can also check these resources for further info:

Cheers and... keep reading!

A comprehensive XML processing benchmark

2008-04-20T10:14:00.031-07:00

by Eduardo Rodrigues

Introduction

I think I've already mentioned it here but, anyway, I'm currently leading a very interesting and challenging project for a big telecom company here in Brazil. This project is basically a complete reconstruction of the current data loading system used to process, validate and load all cellphone statements, which are stored as XML files, into an Oracle CMSDK 9.0.4.2.2 repository. For those who aren't familiar, Oracle CMSDK is an old content management product, which has succeeded the older Oracle iFS (Internet File System). Because it's not an open repository, we are obligated to use its Java API if we want to programmatically load or retrieve data into or from the repository. That, obviously, prevents us from taking advantage of some of the newest tools available like Oracle's XML DB or even the recent Oracle Data Integrator.

Motivation

One of our biggest concerns in this project is with the performance the new system must deliver. The SLA is really aggressive. So, we decided to make some research to find out the newest XML processing technologies available, try and compare them in order to make sure which ones would really help us in the most efficient way. The only constraints are: we must not consider any non-industry-standard solution nor any non-production (or non-stable) releases.

Test Sceneries

That said, based on research and also on previous experience, these were the technologies I've chosen to test and compare:

JAXP SAX 2 compliant parsers:
Oracle XDK parsers shipped with JDeveloper 10.1.3.3
Apache Xerces 2.9.1

StAX 1 compliant pull parsers (Streaming API for XML):
StAX 1.0 (JSR-173) API
Codehaus Woodstox 3.2.4

XML binding:
Sun's JAXB 2.1.6 Reference Implementation
Apache Commons Digester 1.8

I've initially discarded DOM parsers based on the large average size of the XML files we'll be dealing with. We most certainly can't afford the excessive memory consumption involved. I've also discarded Oracle StAX Pull Parser, because it was still a preview release, and J2SE 5.0 built-in XML parsers, since I know they're a proprietary implementation of Apache Xerces based on a version certainly older than 2.9.1.

The test scenery designed was very simple and was intended only to measure and compare performance and memory consumption. The test job would be just to parse a real-world XML file containing 1 phone statement, retrieving and counting a predefined set of elements and attributes. In summary, rules were (for privacy's sake, real XML structure won't be revealed):

Parse all occurrences of "/root/child1/StatementPage" element
For each <StatementPage> do:

Store and print out value of attribute "/root/child1/StatementPage/PageInfo/@pageNumber"
Store and print out value of attribute "/root/child1/StatementPage/PageInfo/@customerCode"
Store any occurrence of element <ValueRecord>, along with all its attributes, within page's subtree
Print out the number of <ValueRecord> elements stored

Print out the total number of <StatementPage> elements parsed
Print out the total number of <ValueRecord> elements parsed

Also, every test should be performed for 2 different XML files: a small file (6.5MB), containing a total of 420 statement pages and 19,133 value records and a large one (143MB) with 7,104 pages and 464,357 value records.

Based on the rules above, I then tested and compared the following technology sets:

Apache Digester using Apache Xerces2 SAX2 parser
Apache Digester using Oracle SAX2 parser
Sun JAXB2 using Xerces2 SAX2 parser
Sun JAXB2 using Oracle SAX2 parser
Sun JAXB2 using Woodstox StAX1 parser
Pure Xerces2 SAX2 parser
Pure Oracle SAX2 parser
Pure Woodstox StAX1 parser

Based on this tutorial fragment from Sun: http://java.sun.com/webservices/docs/1.6/tutorial/doc/SJSXP3.html and considering that performance is our primary goal, I've chosen StAX's cursor API (XMLStreamReader) over iterator. Still aiming for performance, all tested parsers have been configured as non-validating.

In time; all tests were executed on a Dell Latitude D620 notebook, with an Intel Centrino DUO T2400 CPU @ 1.83GHz running on Windows XP Professional SP2 and Sun's Java VM 1.5.0_15 in client mode.

Results

These were the performance results obtained after parsing the small XML file (for obvious reasons, I decided to measure heap usage only when the large file was parsed):

As you can see, Apache Digester's performance was extremely and surprisingly poor despite all my efforts to improve it. So, I had no other choice than to discard it for next tests with the large XML file, from which the results are presented bellow:

Notice that the tendency toward a better performance when <!DOCTYPE> tag is removed from the XML document has been clearly confirmed here.

As for memory allocation comparison, I've once again narrowed the tests only to the worst case from performance tests above: large XML file including <!DOCTYPE> tag. The results obtained from JDev's memory profiler were:

Another interesting information we can extract from these tests is how much XML binding represents in terms of overhead when compared to a straight parser:

Conclusion

After a careful and thorough revision and confirmation of all results obtained from the tests described here, I tend to recommend a mixed solution. Considering its near 12MB/s throughput verified here, I'd certainly choose pure Woodstox StAX parser every time I'll have to deal with medium to large XML sources but, for convenience, I'd also choose JAXB 2 whenever there's a XML schema available to compile its classes from and the size of the source XML is not a concern.

As for complexity, I really can't say that any one of the tested technologies was found considerably more complex to implement than the others. In fact, I don't think this would be an issue for anybody with an average experience with XML processing.

Important Note

Just for curiosity, I've also tested Codehaus StaxMate 1.1 along with Woodstox StAX parser. It's a helper library built on top of StAX in order to create an easier to use abstraction layer for StAX cursor API. I can confirm the implementor's affirmation that StaxMate shouldn't represent any significant overhead for performance. In fact, performance results were identical when compared to pure Woodstox StAX parsing the large XML file. I can also say that it really made my job pretty easier. The only reason I won't consider StaxMate is that it depends on a StAX 1.0 API non-standard extension which is being called "StAX2" by guys at Codehaus.

That's all for now.

Enjoy and... keep reading!

Fixed blog's appearance on IE6

2008-04-19T13:53:00.003-07:00

Great news folks!

We've just fixed our blog's template so it'll be displayed 100% correctly on Internet Explorer 6 browser. Now, we expect this blog to render identically on both Internet Explorer and Firefox (hopefully on all other browsers too).

Please, let us know if there's still any visualization issues.

Cheers and... keep reading!

Is there a cook in JDev's team?

2008-04-16T05:58:00.004-07:00

This week I found something at least very curious when I launched my JDeveloper 10.1.3.3 as I do almost every morning. This was the "Tip of the Day" it showed me:

Well, I don't know what this means but, anyway, here is a full recipe, just in case: http://www.foodnetwork.com/food/recipes/recipe/0,1977,FOOD_9936_15602,00.html

:)