Kartik Shah

Kubernetes Setup

noreply@blogger.com (Kartik Shah) — Wed, 24 Aug 2016 02:19:00 +0000

This post is a walkthrough for getting Kubernetes environment up and running. (Truly speaking, it is more of a self note to get Kubernetes environment running.) We will use Vagrant as Kubernetes provider to configure a Kubernetes cluster of VirtualBox VMs.

Prerequisities
Get following componentes installed:

Kubernetes
Virtualbox
Vagrant
Docker

Step 1: Configure and start Kubernetes

Note: One change I had to make with Kubernetes 1.3.3 running on Mac using Vagrant as Kubernetes provider is to instruct Vagrant to not create ssh keys. I modified the Vagrantfile in kubernetes install by adding

config.ssh.insert_key = false

To start a Kubernetes cluster

export KUBERNETES_PROVIDER=vagrant 
export NUM_NODES=2 
cluster/kube-up.sh

This will create three VirtualBox images namely master, node-1 and node-2. At the end of the process you will see messages on console like this.

Kubernetes cluster is running.
The master is running at:
  https://10.245.1.2 
Administer and visualize its resources using Cockpit:
  https://10.245.1.2:9090 
For more information on Cockpit, visit http://cockpit-project.org 
The user name and password to use is located in /Users/kartik/.kube/config 

... calling validate-cluster 
Found 2 node(s).
NAME                STATUS    AGE 
kubernetes-node-1   Ready     4m 
kubernetes-node-2   Ready     44s 
Validate output:
NAME                 STATUS    MESSAGE              ERROR 
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
Cluster validation succeeded 
Done, listing cluster services:

Kubernetes master is running at https://10.245.1.2 
Heapster is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/heapster 
KubeDNS is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-dns 
kubernetes-dashboard is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard 
Grafana is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana 
InfluxDB is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb 

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Step 2: Run Docker image

Let’s start a simple Docker image. I will use the Spring Boot application Docker image that we create in the last blog entry - Setting up development environment with Docker, Maven and IntelliJ

To configure the authentication for Docker private repository

docker login [server]

This will create credentials under your $HOME/.docker/config.json

To start docker image that is available locally on Docker

kubectl run options --image=kartikshah/options-analyzer --replicas=2 --port=8080 
deployment "options" created

It will take a minute or so for the pods to get to Running status. If you catch them in act of starting up you will see ContrainerCreating Status.

$ kubectl get pods 
NAME                       READY     STATUS              RESTARTS   AGE 
options-2554117421-5dwws   0/1       ContainerCreating   0          11s 
options-2554117421-7ec17   0/1       ContainerCreating   0          11s 

$ kubectl get pods 
NAME                       READY     STATUS    RESTARTS   AGE 
options-2554117421-5dwws   1/1       Running   0          4m 
options-2554117421-7ec17   1/1       Running   0          4m

You can validate docker process is running by

$ vagrant ssh node-1 -c 'sudo docker ps'
CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS               NAMES 
1b56f4a3222a        kartikshah/options-analyzer                                            "java -Djava.security"   3 minutes ago       Up 3 minutes                            k8s_options.e44b7492_options-2554117421-5dwws_default_b7a648ff-5a58-11e6-9527-08002769954a_0da80b48

Step 3: Find IP address of the node

Describe all to find the IP address of the node.

$ kube describe all 
…
Name:        options-2554117421-7ec17 
Namespace:    default 
Node:        kubernetes-node-2/10.245.1.4 
Start Time:    Thu, 04 Aug 2016 10:32:56 -0500 
Labels:        pod-template-hash=2554117421 
        run=options 
Status:        Running 
IP:        10.246.21.3 
…

The IP address listed agains “IP” is the IP address this node is known inside the cluster. You can run a simple curl command from inside the node

$ vagrant ssh node-1 -c 'curl http://10.246.21.3:8080/'
Hello Options Trader, how are you? Connection to 127.0.0.1 closed.

Step 4: Expose Service Loadbalancer

Now let’s expose the pods running on both nodes using a LoadBalancer. Kubernetes load balancer are deployed as Replication Controller (or newer Replication Set). There are three types of Load Balancer options:
1. ClusterIP - Exposes an IP only available from within Kubernetes Cluster
2. NodePort - Exposes special port on a special node IP, but load balances across nodes.
3. LoadBalancer - Only provided by cloud providers e.g. Google, AWS, OpenShift

There is active development going on providing the LoadBalancer option on bare metal Kubernetes deployment. You can read more about it here at service-loadbalancer

We will use NodePort type of replication set to expose as service to outside world.

$ kubectl expose rs options-2554117421 --port=8080 --target-port=8080 --name=option-service --type=NodePort 
service "option-service" exposed

Describe the service to get node ip and port that is exposed to host machine.

$ kubectl describe service
… 
Name:            option-service 
Namespace:        default 
Labels:            pod-template-hash=2554117421 
            run=options 
Selector:        pod-template-hash=2554117421,run=options 
Type:            NodePort 
IP:            10.247.237.53 
Port:                8080/TCP 
NodePort:            30728/TCP 
Endpoints:        10.246.21.3:8080,10.246.33.5:8080 
Session Affinity:    None 
...

Now you can access the service from you host machine. In my case from the Mac which is running the VirtualBox VMs.

$ curl http://10.245.1.4:30728/
Hello Options Trader, how are you?

There you have it - a Kubernetes cluster running a Docker image across multiple VMs (nodes) with NodePort loadbalancing.

Step 5: Console
This step is optional. If you want to explore the Kubernetes dashboard UI, you have to setup a private certificate. One of the ways Kubernetes dashboard UI authenticates is via identity cert. You can create this identity cert as follows:

#Copy the certs from master node
vagrant ssh master-c 'sudo cp /srv/kubernetes/kubecfg.* /vagrant/ && sudo cp /srv/kubernetes/ca.crt /vagrant/'
#Move them to separate folder
mv kubecfg.* ../certs/ && mv ca.crt ../certs/
#Create private cert using open ssl
openssl pkcs12 -export -clcerts -inkey ../certs/kubecfg.key -in ../certs/kubecfg.crt -out kubecfg.p12 -name "ks-kubecfg”
#For Mac only; open the private cert to install it in Keychain Access
open kubecfg.12

Now you can visit the dashboard by visiting the URL provided at the startup message.
https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

Setting up development environment with Docker, Maven and IntelliJ

noreply@blogger.com (Kartik Shah) — Fri, 22 Jul 2016 22:40:00 +0000

This post walks through the setup of development environment for Docker with IntelliJ using Maven. We will use a Spring boot application and configure IntelliJ for iterative development.

Spring Boot Application

The demo application is a simple one page app which displays a chart of VIX index. We will wire Quandl’s web service to fetch the historical data about VIX index and use CanvasJS charting library to display the chart. You can learn more about Spring Boot application with this Spring Guide. Here are some highlights:

Application.java
@SpringBootApplication is the convenience annotation that marks the class as an entry point.

AppConfig.java
Setup Spring wiring using java in this class. It is any class that is marked with @Configuration annotation.

View - JSPs
Application’s view pages are under /webapp/WEB-INF/jsp. application.properties file provided configuration to wire Spring’s view resolver.

Static Content
Application’s static content under src/main/resources/static as below.

You can find entire source code here on github.

pom.xml
spring-boot-maven-plugin make executing lifecycle events easier. If you open IntelliJ’s Maven window, you will see all spring-boot-run, which will allow you to run the project. You can also create run configuration for easy access.

Setting up Docker

You can setup Docker environment by following instructions from docker for mac documentation.

Docker, Maven and IntelliJ

We will use docker-maven-plugin from com.spotify

Docker file

Create src/main/docker/Dockerfile.

pom.xml

Configuration for docker plugin

ServerId tag specifies the docker repository you want to push to. The credentials to the serverId needs to be provided in your maven settings.xml. You have option of mvn encrypting the password. Follow instructions here

Run Configuration

To get it running in IntelliJ, you can setup Run Configuration as follows:

1. Open Run -> Edit Configuration

2. Add Maven Configuration

3. Provide maven command line as clean package docker:build

4. Go to the Runner tab and provide environment variables

5. Similarly, you can also create docker:push run configuration, if you want to push docker image to docker registry.

Web Service implemented as JAX-WS and JAX-RS

noreply@blogger.com (Kartik Shah) — Tue, 14 Oct 2014 04:33:00 +0000

This post walks you through exposing a java web service implementation as both SOAP and REST service. This project uses Apache CXF as a framework to implement JAX-WS and JAX-RS based service on same implementation class.

https://github.com/kartikshah/sample-soap-rest

Web Service Interface

Here is the simple web service interface that has the annotations to expose it as both JAX-WS and JAX-RS endpoint.

Contract - WSDL and WADL

One of the crucial benefit of web service is that contract is either written or generated which serves as integration document. For that purpose, in my opinion, it is essential that either a WSDL or WADL is generated for methods exposed as web service. This applies specifically when using implementation first approach.

WADL

You can also generate WADL in JSON form by appending

?_wadl&_type=json

WADL generation

WadlGenerator configuration in spring application context is key to generating correctly linked representation.

Autolayout and Orientation

noreply@blogger.com (Kartik Shah) — Sat, 11 Jan 2014 21:41:00 +0000

This is a sample project which demonstrates following:

Use of Autolayout to setup proportional views
Use of Autolayout programmatically to move subviews to support different orientation

Consider following contrieved example where we have different layout for landscape and portrait for set of controls.

Steps and Explanation
Divide sections of the screen into subview
Group the controls into some sort of organization based on relationship among them. Use subviews to arrange them as in sample above. The subviews does not need to have visual characteristics as background color or borders. The colors used above is just for easier visualization.

Define the horizontal and vertical constraints using visual programming language To make the task of defining constraints easier inspect the expected result for both orientation. Come up constraints for the leftView and rightView in relation with containerView.

For Landscape,

Horizontal Constraints: [superView]-[leftView]-[rightView]-[superView] ==> H:|-[leftView]-[rightView]-|
Vertical Constraints: [superView]-[leftView]-[superView] ==> V:|-[leftView]-| and [superView]-[rightView]-[superView]V:|-[rightView]-|
leftView.width = factor x rightView.width (Factor for same width = 1.0)
leftView.height = rightView.height (Factor for same height = 1.0)

For Portrait,

Horizontal constraint: [superView]-[leftView]-[superView] ==> H:|-[LeftView]-| and [superView]-[rightView]-[superView] ==> H:|-[rightView]-|
Vertical constraint: [superView]-[leftView]-[rightView]-[superView] ==> V:|-[leftView]-[rightView]-|
leftView.height = factor x rightView.height (Factor for same height = 1.0)
leftView.width = rightView.width (Factor for same width = 1.0)

Code updateViewConstraints method
For Landscape:
For Portrait:
Complete source code at: https://github.com/kartikshah/AutoLayoutTemplate

Hidden Design Elements

noreply@blogger.com (Kartik Shah) — Sun, 30 Jun 2013 18:46:00 +0000

There are many design elements to consider while creating iOS apps. Some of these elements your users can notice visually. And some of these elements are hidden which users will notice while experiencing the app.

Elements that users see

There are aspects of design that your users can see. They exists on the screen. They can see them and they can interact with some of them. These aspects differ for different types of app.

Users will look at how components are laid out on the screen. They will notice the choice of UI component. Text on the app conveys information. Users will notice its readability. It needs to be easy to read for your demographics. Graphics styles your app. Creative assets needs to align with the purpose of the app. Most essentially it needs to connect with your user. Users will notice if it doesn’t.

Elements that users notice

There are aspects of design that user won’t see. But they will notice their absence. Here are some of these hidden design elements. (This is by no means an exhaustive list)

Consistency

Consistency reinforces designs’ purpose. It reinforces behavior. It makes it easy for the user to remember actions they want to perform. It makes interaction obvious.

Consistent use of UI component. Decide which UI components best serve the purpose for a specific action in UI. Stick with same component throughout the app for similar usage pattern. For example, if app displays user agreements, privacy policy in which user has to agree, use similar controls for accept/decline, use similar controls to display agreement text.

Consistent use of navigational flow. It is essential to determine patterns for navigational flow. Define navigational pattern used for sub flows within apps and stick to it consistently. If it is drill down navigation, provide consistent controls to go further or go back. Reinforce next/back actions by providing them in specific place on the screen. If popover window slides up, make sure similar transition is applied to screens following same action.

There could be exceptions to consistency. Define why exception is required and use it sparingly and judiciously.

Simplicity

Complexity within the app points to job half done. Take time to simplify.

Simplify complex UI. If a single screen is doing more than one primary function, simplify the screen. Limit the number of popovers from single screen. Avoid gestures that are not optimum for certain screen sizes. For example, three/four finger gestures aren’t well suited for smaller screen. Screen has a functional need to display too much data, find a way to display information in simple manner.

Simplify any complex navigation flows. Find simpler organization for screens where wizards have too many steps. Avoid deep drill downs where users can’t track their way backwards. Keep navigational flow linear, don’t branch and re-branch off into sub flows. If that is functional need abstract that complexity out from user.

Simplify configuration options. Provide a quick way to start using the app. Users rely on app designer to prepare the optimum default configuration for the app. If app requires too many steps to start using the app, users will balk in the setup process.

Responsiveness

Responsiveness of UI is of utmost importance. Make sure that UI actions are not starving for resources against other time consuming operations. Move those non-UI time consuming operations to background queues. Transitions are required to be smooth. Slides and swooshes of UI elements provides context to subconscious mind. If they get slow or waiting to load data, its effectiveness is reduced and purpose unachieved.

Visual Cues

Use visual cues effectively. Effective use visual cues to make user aware of hidden gestures. Gestures like swipe for split views and page controls are not easily obvious to certain users. Use visual cues like bounce to make user aware of such available interactions.

Use Transitions appropriately. Use transitions correctly to display additional screens. If popovers or modal windows are sliding up it makes sense if they slide down when dismissed. Avoid transitions from all edges the screen.

Information Presentation

Present information user wants to know at correct level. Present it at level which is most logical. Present it at too high level and user will have to navigate deeper to get the information they need. Present it too detail and users will get overwhelmed by information.

Finally remember this. Users will notice when hidden design elements are missing.

How to make Xcode use latest SVN Client

noreply@blogger.com (Kartik Shah) — Fri, 10 May 2013 03:18:00 +0000

Recently working on an iOS project faced an issue importing project to subversion. All svn commands worked fine from command line, but would not work from Organizer. The issue turned out to be incompatibility between svn client version to the version on subversion server. Xcode was using svn client bundled within Xcode app, while I had later version of the client installed elsewhere available on path.

Here is how you can point Xcode to use latest version of svn.

JAXB - Unmarshal non root element

noreply@blogger.com (Kartik Shah) — Wed, 01 Feb 2012 21:02:00 +0000

While generating JAX-WS web service client code using wsimport, it generates Object factory top level method input and output type. This restricts the use to marshal and unmarshal classes which are contained within the top level objects.

More often than not, top level Schema objects are non-domain specific objects like MethodInput/MethodOutput. You can unmarshal them using ObjectFactory

ObjectFactory.java

query.xml

MethodInput.java

JaxbUnmarshallerMethodInput.java

But if you want to unmarshal XML chunks of objects contained within those top level object, the same approach does not work, if those generated classes are not included as part of ObjectFactory or they do not have annotation @XmlRootElement on top of that.

account.xml

Option 1

So the obvious option is to add @XmlRootElement to any generated classes that you want to unmarshal directly, but when you are using wsdls are from an external source, idea of updating generated classes breaks the process.

Option 2

Another option is to pass the child element's Node object to unmarshal

SOA Composite deployment - Oracle SOA Suite

noreply@blogger.com (Kartik Shah) — Sat, 09 Jul 2011 18:15:00 +0000

This post is going to be about very specific subject - SOA Composite deployment on Oracle SOA Suite. I am going to capture few issues faced while deploying SOA composite.

Sample Composite

For purpose of this blog, consider a simple SOA composite that does the following:

Read entries from a database
Create XML from the entries
Store XML entries to another database
Read XML entries
Post each XML entry to a web service

There is good amount of workflow and business logic that makes it a perfect use case for doing it with BPEL, but that is beyond the purpose of this entry.

Composite Dependencies

Sample SOA composite had following dependencies:

Couple of Datasource (one to read source data and other to store XML Entries)
JMS Queue (Weblogic’s Uniform Distributed Queues)

Deployment Process

1. Datasources

Create Generic Datasources pointing to individual Oracle RAC node and a Multi Datasource using WLST scripts. (Most of the scripts are written for Weblogic 10.3.3, so haven’t explored newly introduced GridLinked Datasource in Weblogic 10.3.4.)

2. JMS Resources

Create JMS Server, JMS Module, Sub deployment, Connection Factory and JMS Queues using WLST scripts.

3. DBAdapter.rar

The composite uses DBAdapter connector to interact with databases, so add outbound connection pools the DBAdapter.rar pointing to actual Weblogic Datasources and redeploy application

4. Deploy Composite

Finally, deploy the composite using Enterprise Manager Console.

Lessons Learned

1. Server Start parameters

Using Enterprise Manager to deploy, the deployment window got stuck and never returned back. The issue, it turns out, is that the deployment was not able to communicate among instance servers. If SOA cluster is using multicast to communicate across weblogic instances, you will required following parameters on each server instance. The parameter adds well known address

-Dtangosol.coherence.wka1=soa01.mycompany.com -Dtangosol.coherence.wka2=soa02.mycompany.com -Dtangosol.coherence.localhost=soa01.mycompany.com -Xmx2048m

2. Failure updating DBAdapter.rar connector application

On attempt to create outbound connection pool pointing to weblogic datasource, activation fails complaining FileNotFoundException: Plan.xml. The file is present on the primary server instance, but it isn't replicated on other nodes. I am not sure why but for some reason the Plan.xml was required to present on all the nodes, even if deployment is being done from machine with AdminServer.

3. Failure on interaction with DB on some instances.

Even after the deployment was successful, any DB activities from nodes other than primary node failed giving following error:

BINDING.JCA-12511
JCA Binding Component connection issue.
JCA Binding Component is unable to create an outbound JCA (CCI) connection.
apptools:InsertLogDB [ InsertLogDB_ptt::insert(TestLogCollection,TestLogCollection) ] : The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue: BINDING.JCA-12510 JCA Resource Adapter location error.
Unable to locate the JCA Resource Adapter via .jca binding file element <connection-factory/> The JCA Binding Component is unable to startup the Resource Adapter specified in the <connection-factory/> element: location='eis/DB/application-ds'. The reason for this is most likely that either 1) the Resource Adapters RAR file has not been deployed successfully to the WebLogic Application server or 2) the '<jndi-name>' element in weblogic-ra.xml has not been set to eis/DB/application-ds. In the last case you will have to add a new WebLogic JCA connection factory (deploy a RAR). Please correct this and then restart the Application Server
Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections. Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections.

One noticeable thing on weblogic console was that under testing tab of DBAdapter.rar all outbound connection pools were not visible. The issue is that updated Plan.xml is not replicated to all other nodes. You have to manually copy the Plan.xml containing all outbound connection pool to all server instance nodes.

Here is what you need to do:

Copy the Plan.xml containing all outbound connection pool to all nodes
Update the connector application DBAdapter.rar
Restart DBAdapter.rar and Application server instance

4. BAM Cluster Multicast misconfiguration

A misconfiguration on the Multicast address of BAM cluster resulted in Access Forbidden 403. The issue was that BAM server was trying to communicate with another environment cluster. (i.e. prod bam cluster trying to communicate with test bam cluster).

Here is the error:

<BEA-000141> <TCP/IP socket failure occurred while fetching statedump over HTTP from 142374575950937656S:10.50.XX.XXX:[XXXXX,XXXXX,-1,-1,-1,-1,-1]:soa:soa_server1.
java.io.FileNotFoundException: Response: '403: Forbidden' for url: ' http://10.50.XXX.XXX:16101/bea_wls_cluster_internal/psquare/p2?senderNum=3&lastSeqNum=0&PeerInfo=10,3,4&ServerName=soa_server1'
        at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:487)
        at weblogic.cluster.HTTPExecuteRequest.connect(HTTPExecuteRequest.java:67)
        at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:83)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
   at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

So make sure that the Mutlicast address and port combination used for BAM cluster and SOA cluster is unique. There is a Multicast test utility that you can use to test communication between two instance. More information about the utility is available here. Also refer this to troubleshoot Multicast configuration.

export CLASSPATH=${CLASSPATH}:[bea_home]/server/lib/weblogic.jar

On Machine A:

java utils.MulticastTest -A [multicast address] -P [multicast port] -N TestServer1

On Machine B:
java utils.MulticastTest -A [multicast address] -P [multicast port] -N TestServer2

Hope this helps!.

OSB Project Structure

noreply@blogger.com (Kartik Shah) — Wed, 20 Apr 2011 18:13:00 +0000

I am doing some development with Oracle Service Bus (OSB). The primary goal is to mediate web services using OSB. Oracle does provide set of eclipse plugins to develop OSB configuration - OEPE suite. One of question that I wanted answer was what will be a good project structure to organize various components of OSB configuration.

Typically, OEPE plugins gives two eclipse project templates:

Oracle Service Bus Configuration Project
Oracle Service Bus Project

OSB Config project is the top level project which can include various OSB Project. A minimal OSB project includes components like business service, proxy service and WSDL. It is important to come up with a good project structure since the component shows up as is on sbconsole. You probably don't want to see all the components at same level.

After this exercise, I ended up with following structure of the project that I wanted to share. So here it is,

The top level project - OSB Config Project - represents a business domain. So OSB configuration of all services serving a particular domain goes under that umbrella. Individual OSB Project contains mediation component for a given physical service. I used folders to group business and proxy services to allow for multiple proxy services for a given business service. Using this structure it becomes easy to manage configuration using sbconsole.

I ended up using the ant build script and import/export WLST script by tweaking the ones given as part of this book - The definitive guide to SOA. The build script can export configuration from local server or workspace. Import deploys it to the target server. The import/export script also takes care of variables that changes from environment to environment. OSB has a features to create customization XMLs. You can create customization XML per environment and run it post configuration package deployment.

Using Clojure to work with RSS feeds

noreply@blogger.com (Kartik Shah) — Mon, 24 Jan 2011 05:18:00 +0000

As I mentioned in my last blog entry, I am trying to learn Clojure. I think it is an interesting challenge to internalize new thinking paradigm. So following entry/code is an attempt to learn the paradigms of the language. Drop a comment, if you have any suggestion.

I found that, while trying to learn new language, one of the challenge is to find interesting practice problem to solve. The practice problem which helps you understand the strength of the language. I think one such problem space Clojure's application would be interesting is parsing and data crunching.

Example Problem
So the problem I picked is a simple one. I heavily use google reader and for that matter most of my web reading happens as part subscribed RSS feeds. I find myself manually filtering the feed list to suite my interest at the time. This exercise creates parser for RSS feed. This parser creates Clojure records based data structure. Once you have all the feed represented in records structure you can write variety of functions to filter the feed or even apply intelligent web algorithms.

Organization of Code
I am looking for a good way to organize code in clojure. Just like Java packages, clojure has namespaces. But I have still not found a good way to organize clojure functions into files/namespaces. For now, I chose to create namespace per domain.

rss.data.clj - Define records data structure and parser
rss.feed.clj - Defines functions to get feeds
rss.stat.clj - Defines functions to filter feeds (and may be some statistical algorithms)

data.clj


(ns rss.data
  (:require [clojure.xml :as xml])
  (:import  [java.io ByteArrayInputStream]))

(defrecord rss-channel [title description link lastbuilddate pubdate items])
(defrecord rss-item [title description link guid pubdate])

(defn create-rss-channel [map items]
  (let [title (:title map)
        pubDate (:pubDate map)
        description (:description map)
        link (:link map)
        lastBuildDate (:lastBuildDate map)]
    (rss-channel. title description link lastBuildDate pubDate items)))

(defn create-item [item-data]
  (let [description (:description item-data)
        pubDate (:pubDate item-data)
        guid (:guid item-data)
        link (:link item-data)
        title (:title item-data)
        ]
    (rss-item. title description link guid pubDate)))

(defn parse-item [item-data]
  (create-item (zipmap (map :tag item-data) (map :content item-data))))

(defn parse-items [items-data]
  (map parse-item items-data))

(defn parse-channel [channel-content]
  (loop [x (:content channel-content)]
    (create-rss-channel
      (into {} (filter #(not (= :item (key %))) (zipmap (map :tag x) (map :content x))))
      (parse-items (map :content (into [] (filter #(= :item (:tag %)) x)))))
      ))

(defn parse-rss [rss]
  (let [rss-data (ByteArrayInputStream. (.getBytes rss "UTF-8"))]
  (for [x (xml-seq (xml/parse rss-data))
        :when (= :rss (:tag x))]
    (map parse-channel (:content x)))))

feed.clj

(ns rss.feed
  (:require [clj-http.client :as client]))

(defn get-feed [url]
  (:body (client/get url)))

(defn get-feeds [urllist]
  (pmap get-feed urllist))

stat.clj

(ns rss.stat)

(def keywords ["Clojure" "Groovy" "Python"])

(defn find-items [rss-item-list keyword]
  (filter #(.contains (first (:title %)) keyword) (flatten rss-item-list)))

(defn find-items-keywords [rss-item-list keyword-list]
  (flatten (map #(find-items rss-item-list %) keyword-list)))

Trying Delicious RSS
Let's use RSS feed published by delicious.com. All RSS feed can found here. Let's use the feed popular which got has tag "programming". (http://feeds.delicious.com/v2/rss/popular/programming)


user=> (rss.stat/find-items-keywords (map :items (flatten (rss.data/parse-rss (rss.feed/get-feed "http://feeds.delicious.com/v2/rss/popular/programming")))) ["Clojure" "Python" "Groovy"])
(#:rss.data.rss-item{:title ["Hidden features of Python - Stack Overflow"], :description nil, :link ["http://stackoverflow.com/questions/101268/hidden-features-of-python/102062"], :guid ["http://www.delicious.com/url/16bbee33bbf4f6a03448058c5fd9f461#thaiyoshi"], :pubdate ["Sun, 23 Jan 2011 16:53:58 +0000"]} 
#:rss.data.rss-item{:title ["Writing Forwards Compatible Python Code | Armin Ronacher's Thoughts and Writings"], :description nil, :link ["http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/"], :guid ["http://www.delicious.com/url/c6fb2679bca6c193a41d92513f7edca1#papaeye"], :pubdate ["Sat, 22 Jan 2011 16:09:10 +0000"]})

This one only had few links. Let's try one which has lots of items. (http://feeds.delicious.com/v2/rss/tag/programming?count=100)


user=> (rss.stat/find-items-keywords (map :items (flatten (rss.data/parse-rss (rss.feed/get-feed "http://feeds.delicious.com/v2/rss/tag/programming?count=100")))) ["Clojure" "Python" "Groovy"])

(#:rss.data.rss-item{:title ["The Evolution of a Python Programmer : Aleks' Domain"], :description nil, :link ["http://metaleks.net/programming/the-evolution-of-a-python-programmer"], :guid ["http://www.delicious.com/url/15311aafa2207da0ffc770903091ccea#tsung"], :pubdate ["Mon, 24 Jan 2011 02:44:19 +0000"]} 
#:rss.data.rss-item{:title ["Hidden features of Python - Stack Overflow"], :description nil, :link ["http://stackoverflow.com/questions/101268/hidden-features-of-python/102062"], :guid ["http://www.delicious.com/url/16bbee33bbf4f6a03448058c5fd9f461#adharmad"], :pubdate ["Mon, 24 Jan 2011 02:43:07 +0000"]} 
#:rss.data.rss-item{:title ["PySCeS: the Python Simulator for Cellular Systems"], :description nil, :link ["http://pysces.sourceforge.net/index.html"], :guid ["http://www.delicious.com/url/40e012b894d62cd8cb9ba6b95855b868#tiguco"], :pubdate ["Mon, 24 Jan 2011 02:39:24 +0000"]} 
#:rss.data.rss-item{:title ["Hidden features of Python - Stack Overflow"], :description nil, :link ["http://stackoverflow.com/questions/101268/hidden-features-of-python/102062"], :guid ["http://www.delicious.com/url/16bbee33bbf4f6a03448058c5fd9f461#lonstile"], :pubdate ["Mon, 24 Jan 2011 02:24:25 +0000"]}

Source Code available at GitHub (https://github.com/kartikshah/rss-feeder)

Blogged with the Flock Browser

How-To - IntelliJ Idea for Clojure/Compojure on Google App Engine

noreply@blogger.com (Kartik Shah) — Sun, 12 Dec 2010 04:26:00 +0000

StrangeLoop conference really spiked my interest in learning Clojure. I wanted to find a way to do web development using Clojure. So this entry is primarily a how-to on setup of IntelliJ Idea to do Clojure using Compojure web framework, Leiningen build tool and Google App Engine deployment environment.

Step 1: Download Leiningen
Download Leiningen script file and follow instruction to put it in PATH.

Step 2: Install "La Clojure" and "Leiningen" plugin for IntelliJ Idea.
To do this go to Preferences -> Plugins. Go to "Available" Tab

Step 3: Setup Leiningen plugin properties
To setup Leiningen plugin properties in IntelliJ click on Preferences -> Leiningen and give the path to the executable.

Step 4: Create project template
Now with the installation complete, we are ready to create a new project. For now the project creation is done outside IntelliJ Idea using Lein on command line. This is the only step done outside the IntelliJ Idea. Choose the folder structure where your IntelliJ workspace is.


     $ lein new gae-cmpjr
     Created new project in: gae-cmpjr

You can see that Lein created project folder structure. In the project root it created a project.clj file used by Leiningen. Under src folder, it also created default namespace by the name of the project and created a core.clj file under it.

Step 5: Create project IntelliJ
Let's create IntelliJ Project, even though project structure exists pick the same folder path that you selected in step 4.

File -> Project New... and select project from scratch
Select both GAE and Clojure. Provide the path to the appengine sdk

Now basic setup for the project is done. You can browse the project files using IntelliJ. (Biggest benefit of La Clojure is it auto-inserts and matches brackets for very "brackety language" :-))

Step 6: Setup Leiningen to run from IntelliJ
To setup Leiningen from IntelliJ, click on Leiningen tool window. (Window -> Tool Windows -> Leiningen)

Step 7: Setup web development libraries
Setup the project to include Compojure and Ring libraries for development.
project.clj


(defproject gae-cmpjr "1.0.0-SNAPSHOT"
  :description "Example for Compojure on GAE using IntelliJ"
  :namespaces [gae-cmpjr.core] 
  :dependencies [[compojure "0.4.0-SNAPSHOT"]
                 [ring/ring-servlet "0.2.1"]]
  :dev-dependencies [[leiningen/lein-swank "1.2.0-SNAPSHOT"]]
  :compile-path "web/WEB-INF/classes"
  :library-path "web/WEB-INF/lib")

core.clj


(ns gae-cmpjr.core
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.core
        ring.util.servlet)
  (:require [compojure.route :as route]))

(defroutes hello
  (GET "/" [] "<h1>Hello World Wide Web!</h1>")
  (route/not-found "Page not found"))

(defservice hello)

Step 8: GAE setup

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
            http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
           version="2.5">
    <display-name>Compojure on GAE using IntelliJ</display-name>

    <servlet>
        <servlet-name>hello</servlet-name>
        <servlet-class>gae_cmpjr.core</servlet-class>
    </servlet>

    <servlet-mapping>
        <servlet-name>hello</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>
</web-app>

appengine-web.xml


<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>compgae</application>
  <version>v0-2</version>
</appengine-web-app>

Step 9: Compile and Build war

From the Leiningen Tool window in Idea run following commands.

lein clean
lein deps 
lein compile

Step 10: Run locally
To locally run click on already create "Run Configuration"

Step 11: To upload application to appengine
Tools -> Upload Appengine application

Blogged with the Flock Browser

Revisiting JMX DSL with Groovy GEP3

noreply@blogger.com (Kartik Shah) — Sun, 26 Sep 2010 04:11:00 +0000

In one of the previous blog post, I experimented with a sample DSL for JMX reporting using Jfree chart. Groovy 1.8.2 (Beta) has introduced command expression language. This GEP-3 essentially allows you to create command expression language which can further simplify language of DSL. The primary use of this is to aid in writing impromptu scripts to visually demo various MBean values. In this example we will see bar charts for module processing time for all web modules.

Command expression language allows you to alternate method name and parameter for even number of argument
e.g.
foo a1 a2 a3 a4 a5. Foo(a1).a2(a3).a4(a5)
You can find more example and explanation on GEP-3 page.

Revisiting JMX example so that I can simplify the language as


server.url "service:jmx:rmi://localhost/jndi/rmi://localhost:1090/jmxconnector" query "jboss.web:*" filter "j2eeType=WebModule"
def filteredModules = server.filteredModules

chart.type "Bar" modules filteredModules title "Module Processing Time" width 1200 height 700 refresh 500 attributes params labels graphLabels
chart.show()

Let's look at the supporting code.


import org.jfree.chart.ChartFactory
import groovy.swing.SwingBuilder

import org.jfree.data.category.DefaultCategoryDataset
import org.jfree.chart.plot.PlotOrientation as Orientation
import javax.swing.WindowConstants as WC
import javax.management.ObjectName
import javax.management.remote.JMXConnectorFactory
import javax.management.remote.JMXServiceURL as JmxUrl
import javax.naming.Context

class Chart {
    def chartModules
      def chartType
    def chartAttributes = {m -> [m.processingTime, m.path]}
    def chartLabels =["Time per Webapp", "Webapp", "Time"]
    def chartOptions = [false, true, true]
    def windowTitle
    def w
    def h
    def refreshRate
    def orientation = "VERTICAL"
    def dataset

    def modules(m) {
          chartModules = m
          this
      }

      def type(type) {
        chartType = type
         this
      }

      def attributes(attr) {
        chartAttributes = attr
        this
      }

      def labels (lbls) {
        chartLabels = lbls
        this
      }

      def options (opts) {
        chartOptions = opts
        this
      }

      def title (title) {
        windowTitle = title
        this
      }

      def width(width) {
        w = width
        this
      }

      def height(height) {
        h = height
        this
      }

      def refresh(r) {
        refreshRate = r
        this
      }

    void show(){
        switch(chartType){
            case "Bar": drawBarChart(); break;
            default: break;
        }
    }

    void drawBarChart(){
        calculateData()
        def chart = ChartFactory.createBarChart(*chartLabels, dataset, Orientation."${orientation}", *chartOptions)
        def swing = new SwingBuilder()
        def frame = swing.frame(title:windowTitle, defaultCloseOperation:WC.EXIT_ON_CLOSE){
            panel(id:'canvas') {rigidArea(width:w, height:h)}
        }
        while(true){
            calculateData()
            chart.fireChartChanged()
            frame.pack()
            frame.show()
            chart.draw(swing.canvas.graphics, swing.canvas.bounds)
            sleep(refreshRate)
        }
    }
    void calculateData(){
        def newDataset = new DefaultCategoryDataset()
        chartModules.each{ m ->
            def dsCall = chartAttributes.call(m)
            newDataset.addValue dsCall[0], 0, dsCall[1]
        }
        this.dataset = newDataset
    }
}


class Server {
    def server
    def queryObjects
    def moduleFilter
    def filteredModules

    def url(serverName){
    server = JMXConnectorFactory.connect(new JmxUrl(serverName)).MBeanServerConnection
    this
    }

    def query (queryString) {
        queryObjects = new ObjectName(queryString)
       this
    }

    def filter (filterString) {
         String[] allNames = server.queryNames(queryObjects, null)
        filteredModules = allNames.findAll { name -> name.contains(filterString)}.collect { new GroovyMBean(server, it)}
    }
}

server = new Server()
chart = new Chart()
params = {m -> [m.processingTime, m.path]}
graphLabels = ["Time per Webapp", "Webapp", "Time"]

server.url "service:jmx:rmi://localhost/jndi/rmi://localhost:1090/jmxconnector" query "jboss.web:*" filter "j2eeType=WebModule"
def filteredModules = server.filteredModules

chart.type "Bar" modules filteredModules title "Module Processing Time" width 1200 height 700 refresh 500 attributes params labels graphLabels
chart.show()

Problems faced
For some reason with the current beta build I couldn't do arrays or closures inline. so I had to put them as separate variables. I do not think it is by design and as 1.8.X comes close to release these wrinkles will be worked out.

Blogged with the Flock Browser

Data as Service - Thoughts

noreply@blogger.com (Kartik Shah) — Thu, 01 Apr 2010 04:46:00 +0000

I am exploring multiple ways to expose data as a service. Let's take a brief look at problem description. You have a central data repository which acts as source of truth for multiple supporting application. These supporting applications usually reads data and manipulate/transform it and do what it needs to do with it.

This is not very atypical scenario. In most of the organization you will have a central application - usually of the shelf product with specific customization. e.g. ERP systems, Billing systems, CRM systems etc. The data store supporting this application is not flexible to customization. You will also find supporting applications for custom needs of organization to use the central data store. In usual scenarios this application will be handling this via direct read from central data store - potentially giving way to duplication of effort to manage this objects.

These problems have been solved in multiple ways.

Allow each individual supporting application to access data directly or indirectly through data warehouses
Expose the catalog of finely-tuned queries expose by the supporting application as web services.
Define organization wide data directory containing well defined business entities/objects and expose web services to retrieve defined entities.

Types of Services
We can classify this data services in multiple categories

Data-Read Services - These are plain and simple data read or data lookup services. This services provide only minor transformation like column renames, etc
Transformation Services - These services provide some transformation of data. It operates on the column performs operation like truncation, concatenation, etc.
Filtering Services - These services allows opportunity to filter data based on input conditions. e.g provide active accounts, provide prospect accounts, etc.
Aggregation Services - These services provides performs certain aggregation function. e.g. sales by region, accounts by regions, etc. Though this logic could have served better purpose if kept in application, there are some performance benefits to do this in the data store layer.

Define Business Entities data dictionary
First and foremost identify organization wide data dictionary for the domain. This data dictionary is the definition of entities that applications needs to use. So there is not duplication. It also reduces confusion. For example, an entity named BillingAccount will mean the billing account for any application using it. This solves a larger problem, in the scenario where the supporting applications use the data directory it defines these central entities in multiple ways. So over time the definition of the same thing becomes convoluted. e.g. account number for application A is 10 digits and same for application b is first 8 digits of that. This results in duplication of data that can not inter operate. Defining a central data dictionary and allowing exchange of that in that contract helps avoid this problem.

Logical Representation
In most scenario the data store that comes bundled with the product is really generic so that product can be customized to different business. eg. ACCTPF having fields like ACCTPFNO, ACCTPFHONO, etc. One side effect can be seen on naming convention of entities and its properties. They are wierdly named and does not make any sense without actually looking at product documentation. My take is that some naming transformation is required. This can be done by "select as" queries or actually overlaying a logical data model on top of the existing one. Of course after performance consideration.

Expose as Web Service
Once you have the logical representation any of following two options can be followed

Write Select queries and expose them as either SOAP/REST web services
If select queries does not provide necessary transformation/aggregation, write db procedures and expose them as a service.

One of the sub variation of this scenario is to use DB specific data structures. (In case of Oracle, use oracle's user defined types). Performance consideration is essential because remember these services may very well be the most atomic activity overarching application will perform and it is essential that each data read is trivial and inexpensive.

Just some thoughts...

Blogged with the Flock Browser

Using Groovy Scriptom to report on Outlook

noreply@blogger.com (Kartik Shah) — Wed, 17 Mar 2010 04:05:00 +0000

As with most work places, I use MS Outlook Calendar at work for meetings, calendars, etc. With Outlook 2007 you have capability to categorize each meeting/time block that you have on your calendar. So you can classify each meeting as either design, architecture, learning, recurring, etc. Using Groovy Scriptom module you can do what you would usually do using VBScript. Scriptom allows you to use ActiveX windows components from Groovy. Needless to say this will only work on windows platform.

I used the scriptom to report on how I am spending my time. (ah.. have to fill those time sheet) Using Swingbuilder and JFreeChart, I threw in this simple script. You will need to setup dependency exactly as described on the help page here.

import org.codehaus.groovy.scriptom.ActiveXObject
import org.jfree.chart.ChartFactory 
import javax.swing.WindowConstants as WC

def outlook = new ActiveXObject("Outlook.Application")
def namespace = outlook.GetNamespace("MAPI")
def calFolder = namespace.GetDefaultFolder(9)
def myItems = calFolder.Items

def today = new Date()
startWeek = today - 7 
endWeek = today + 7
def categoryMap = [:]

for (i in 1..myItems.Count.value) {
     def currentMeeting = myItems.Item(i)
     if (currentMeeting.Start >= startWeek && currentMeeting.Start <= endWeek) {
           println "Subject: " + currentMeeting.Subject.value
           println "Category: " + currentMeeting.Categories
           println "Duration: " + currentMeeting.Duration.value

           category = currentMeeting.Categories
           durationValue = currentMeeting.Duration.value

           def value = categoryMap.get(category)
           value = value?value:0
           def newValue = value + durationValue
           categoryMap.put(category,newValue);
     }
}

def swing = new groovy.swing.SwingBuilder()
def titleString = 'Time Outlook: ' + String.format('%tm/%<td/%<tY', startWeek) + "-" + String.format('%tm/%<td/%<tY', endWeek)
def frame = swing.frame(title:titleString,
                        defaultCloseOperation:WC.EXIT_ON_CLOSE,
                        size:[800,600], locationRelativeTo: null) {
     borderLayout()
     panel(id:'canvas') { rigidArea(width:700, height:500) }
 }

def bounds = new java.awt.Rectangle(0,0, 700,500).bounds
def piedata = new org.jfree.data.general.DefaultPieDataset()
for(entry in categoryMap)
{
  piedata.setValue entry.key + " = " + entry.value, entry.value
}
def options = [false, true, false]
def chart = ChartFactory.createPieChart(titleString, piedata, *options)
chart.backgroundPaint = java.awt.Color.white
frame.pack()
frame.show()
chart.draw(swing.canvas.graphics, bounds)

Simple enough script... It gets the ActiveXObject for Outlook application. Then from namespace it gets calendarFolder (constant 9). You can find out more about constant here. calFolder contains all the meeting appointmentItems. Once you get to the appointmentItem you can ask for the properties using groovy dot-notation.

Here is the snapshot of the output

Blogged with the Flock Browser

Revisit Jmx Groovy DSL - Using AstTransformation

noreply@blogger.com (Kartik Shah) — Mon, 01 Mar 2010 05:59:00 +0000

In the previous blog entry, we created Jmx Charting DSL. Let visit some aspects of creating DSL using groovy.

In the previous blog example we created DSL using ExpandoMetaClass. Users will define the script starting with node jmx { }. With the use of ExpandoMetaClass we added the dynamic method which in turn delegated to class JmxClosureDelegate. Here is the snippet of the code.


static void runEngine(File dsl){
    Script dslScript = new GroovyShell().parse(dsl.text)
    dslScript.metaClass = createExpandoMetaClass(dslScript.class, {
      ExpandoMetaClass emc ->
        emc.jmx = {
          Closure cl ->
            cl.delegate = new JmxClosureDelegate()
            cl.resolveStrategy = Closure.DELEGATE_FIRST
            cl()
        }
    })
    dslScript.run()
   }

What we essentially did here:

Using GroovyShell it parses the script file passed as input.
Defines the ExpandoMetaClass and adds method by name "jmx" having closure as parameter.

So the script file you wrote gets a dynamic method injected into it using ExpandoMetaClass. This happens at runtime.

Getting rid of the Commas
The problem with creating methods with two arguments is that you have to specify commas between them. For example,


     server "nameofserver", {
          ...
     }

Undoubtedly commas clutter the langauge grammer. To get rid of the commas we used trick provided on Groovy users list.


//To avoid using "," between String and Closure argument  
  def methodMissing(String name, args) {  
    return [name, args[0]]  
  }

Using AstTransformation
Now lets explore a different possibility. You can achieve similar result using AstTransformation at compile time. The goal remains the same and that is to add method with name "jmx" with closure parameter.

First we will define the annotation.


//import statements skipped for brevity
@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.METHOD])
@GroovyASTTransformationClass(["info.kartikshah.jmx.ast.JmxDslTransformation"])
public @interface UseJmxDsl {}

Transformation Class

This transformation class needs to perform two activities:

Add method jmx(Closure cl) method
Invoke the script method being defined

We need to generate AST statements for following snippet of code.


     jmx = {  
          Closure cl ->  
            cl.delegate = new JmxClosureDelegate()  
            cl.resolveStrategy = Closure.DELEGATE_FIRST  
            cl()  
        }

We will use AstBuilder's buildFromSpec option to generate it. (AstBuilders added with Groovy 1.7 definitely makes generating statement structure relatively easy and clutter free. Not to mention it is also an example of DSL added to the groovy language :-) )


@GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS)
class JmxDslTransformation implements ASTTransformation {
  static int PUBLIC = 1
  static int STATIC = 8

  void visit(ASTNode[] astNodes, SourceUnit sourceUnit) {
    //Add method jmx(Closure cl)
    ClassNode declaringClass = astNodes[1].declaringClass
    MethodNode jmxMethod = makeMethod()
    declaringClass.addMethod(jmxMethod)

    //Insert method call inside run method of the script class
    MethodNode annotatedMethod = astNodes[1]
    List<MethodNode> allMethods = sourceUnit.AST?.classes*.methods.flatten()
    MethodNode runMethod = allMethods.find{ MethodNode method ->
      method.name == "run"
    }
    List existingStatements = runMethod.getCode().getStatements()
    existingStatements.add(0, createMethodCall(annotatedMethod))
  }

  Statement createMethodCall(MethodNode methodNode){
    def statementAst = new AstBuilder().buildFromSpec {
      expression{
         methodCall {
           variable "this"
           constant methodNode.name
           argumentList {}
         }
      }
    }
    Statement stmt = statementAst[0]
    stmt
  }

  MethodNode makeMethod() {
    def ast = new AstBuilder().buildFromSpec {
      method('jmx', PUBLIC | STATIC, Void.TYPE) {
        parameters {
          parameter 'cl': Closure.class
        }
        exceptions {}
        block {
          expression {
            binary {
              property {
                variable "cl"
                constant "delegate"
              }
              token "="
              constructorCall(JmxClosureDelegate.class) {
                argumentList()
              }
            }
          }
          expression {
            binary {
              property {
                variable "cl"
                constant "resolveStrategy"
              }
              token "="
              property {
                classExpression Closure
                constant "DELEGATE_FIRST"
              }
            }
          }
          expression {
            methodCall {
              variable "cl"
              constant "call"
              argumentList {}
            }
          }
        }
      }
    }
    MethodNode jmxMethod = ast[0]
    jmxMethod
  }
}

Why use AstTransformation?
The question is why one would want to use AstTransformation when you can add method during runtime. For the given scenario, it is correct that you want to stick with adding method runtime. But consider scenario where you want to "redefine" meaning of Groovy's syntax. For example like following imaginary script using Statement Labels to add more readability to your DSL syntax.


@info.kartikshah.jmx.ast.UseJmxDsl
runDsl () {
  jmx {
    setup:
      server "service:jmx:rmi://localhost/jndi/rmi://localhost:1090/jmxconnector"
      query "jboss.web:*"
      findAll "j2eeType=Servlet"

    draw:
    chart {
        chartType="Bar"
        attributes={m-> [m.loadTime, m.objectName.find("name=([^,]*)"){it[1]}]}
        labels=["Load Time per Servlet", "Servlet", "Time"]
        options=[false, true, true]
        windowTitle="JBoss Servlet Processing Time"
        width=1200
        height=700
        orientation="HORIZONTAL"
        refreshRate=5000
        show()
      }
  }
}

Spock Framework does similar twist by redefining meaning of existing construct.

With this type of language structure you will end up defining your own set of keywords, supporting parser and few AstTransformation to change the meaning of existing Groovy Syntax.

Blogged with the Flock Browser

A Groovy DSL - JMX Reporting

noreply@blogger.com (Kartik Shah) — Wed, 17 Feb 2010 04:30:00 +0000

In previous blog example, using JFreeChart, JMX and SwingBuilder we came up with dashboard type utility. With this post we will explore how to write small DSL which will allow to report on JMX using reporting charts. The DSL will using Groovy's SwingBuilder to draw JFree Chart to report on various MBeans. This type DSL can be used to write small scripts to monitor behavioral aspect of application server (or any JMX based application).

Simple DSL
First we will chart out how we want our domain language to look like. DSL syntax can be done multiple ways. It is necessary that you play with the syntax to come up with one that works for the scenario. For our example, here is the first cut that we will use.

Essentially, the script reports processing time of all web modules defined on (in this example - JBoss) application server using Bar Chart.


jmx {
    server "service:jmx:rmi://localhost/jndi/rmi://localhost:1090/jmxconnector" {
      query "jboss.web:*" {
        findAll "j2eeType=WebModule" {
          chart{
            chartType="Bar"
            attributes={m-> [m.processingTime, m.path]}
            labels=["Time per Webapp", "Webapp", "Time"]
            options=[false, true, true]
            windowTitle="JBoss Module Processing Time"
            width=1200
            height=700
            refreshRate=5000
            show()
          }
        }
      }
    }
  }

Other Use cases
Target users for this DSL are system administrators who can write simple scripts to monitor and/or report on various aspects of app server instances graphically.

Some other use cases for this type of DSL

Compare Processing Time of Web Application
Compare Load Time of Servlets
Compare Response Time
Compare memory usage
Compare total requests

DSL Engine
With Groovy there are multiple ways to write domain specific language as described in groovy documentation here

We will use nested closure approach. More information about nested closure here.

Here we use ExpandoMetaClass and series of Delegate classes to handle each closure. In the sample script above read each node (eg. jmx, server, query) as a dynamic method call with one or two arguments. For example jmx has a string argument followed by closure argument. String argument is we just store in instance variable. For closure argument we delegate the handling to separate Delegate Class. We follow the same pattern for the rest of the nested nodes.

First up Engine
This is the main class for the DSL. It gets the file passed as command args and pass it to GroovyShell to create Script object. It uses ExpandoMetaClass to dynamically add methods/closure. It add adds jmx closure and sets properties for handling closure.


class JmxReportingDslEngine {
  
  static main(String[] args){
    if(args.length != 1)
    {
      println("Usage: JmxReportingDslEngine <ScriptFileName>")
    }
    runEngine(new File(args[0]))
  }
  
  static void runEngine(File dsl){
    Script dslScript = new GroovyShell().parse(dsl.text)
    dslScript.metaClass = createExpandoMetaClass(dslScript.class, {
      ExpandoMetaClass emc ->
        emc.jmx = {
          Closure cl ->
            cl.delegate = new JmxClosureDelegate()
            cl.resolveStrategy = Closure.DELEGATE_FIRST
            cl()
        }
    })
    dslScript.run()
   }

  static ExpandoMetaClass createExpandoMetaClass(Class clazz, Closure cl){
    ExpandoMetaClass emc = new ExpandoMetaClass(clazz, false)
    cl(emc)
    emc.initialize()
    return emc
  }
}

Delegates

Next up we write series of Delegate Class responsible for handling each node of the language.

JMXClosureDelegate
This handles closure passed to jmxtag. It instantiates MBeanServerConnection and passes the reference down to the delegate chain.


class JmxClosureDelegate {
  //To avoid using "," between String and Closure argument
  def methodMissing(String name, args) {
    return [name, args[0]]
  }
  void server(param){
    def (serverUrl, cl) = param
    def server = JMXConnectorFactory.connect(new JmxUrl(serverUrl),env).MBeanServerConnection
    cl.delegate = new JmxServerClosureDelegate(server)
    cl.resolveStrategy = Closure.DELEGATE_FIRST
    cl()
  }

JMXServerClosureDelegate
This delegate handles closure passed to server tag.


class JmxServerClosureDelegate {
  def server
  JmxServerClosureDelegate(server){
    this.server = server
  }
  def methodMissing(String name, args) {
   return [name, args[0]]
  }
  void query(param){
    def (objectName, cl) = param
    def query = new ObjectName(objectName)
    String[] allNames = server.queryNames(query, null)
    cl.delegate = new JmxQueryClosureDelegate(allNames, server)
    cl.resolveStrategy = Closure.DELEGATE_FIRST
    cl()
  }
}

JmxQueryClosureDelegate
and so on...


class JmxQueryClosureDelegate {
  def allNames
  def server
  JmxQueryClosureDelegate(allNames, server){
    this.allNames = allNames
    this.server = server
  }
  def methodMissing(String name, args) {
    return [name, args[0]]
  }
  void findAll(param){
    def (filter, cl) = param
    def modules = allNames.findAll{ name ->
          name.contains(filter)
      }.collect{ new GroovyMBean(server, it) }
    cl.delegate = new JmxFindAllClosureDelegate(modules)
    cl.resolveStrategy = Closure.DELEGATE_FIRST
    cl()
  }
}

JmxFindAllClosureDelegate


class JmxFindAllClosureDelegate {
  def modules
  JmxFindAllClosureDelegate(modules){
    this.modules = modules
  }
  void chart(Closure cl){
    cl.delegate = new ChartDelegate(modules)
    cl.resolveStrategy = Closure.DELEGATE_FIRST
    cl()
  }
}

ChartDelegate
This class

Creates the dataset from the MBean values
Creates the chart with the dataset values
Creates the external frame using SwingBuilder


class ChartDelegate {

  def modules
  def chartType
  def attributes
  def labels
  def options
  def windowTitle
  def width
  def height
  def refreshRate
  def orientation = "VERTICAL"
  def dataset
  ChartDelegate(modules){
    this.modules = modules
  }
  void show(){
    switch(chartType){
      case "Bar": drawBarChart(); break;
     //TODO:Add more chart types
      default: break;
    }
  }
  void drawBarChart(){
    calculateData()
    def chart = ChartFactory.createBarChart(*labels, dataset, Orientation."${orientation}", *options)
    def swing = new SwingBuilder()
    def frame = swing.frame(title:windowTitle, defaultCloseOperation:WC.EXIT_ON_CLOSE){
      panel(id:'canvas') {rigidArea(width:width, height:height)}
    }
    while(true){
      calculateData()
      chart.fireChartChanged()
      frame.pack()
      frame.show()
      chart.draw(swing.canvas.graphics, swing.canvas.bounds)
      sleep(refreshRate)
    }
  }
  void calculateData(){
    def newDataset = new DefaultCategoryDataset()
    modules.each{ m ->
      def dsCall = attributes.call(m)
      newDataset.addValue dsCall[0], 0, dsCall[1]
    }
    this.dataset = newDataset
  }
}

Output
Let's run and look at sample output...

Click here for bigger images

Here is another script for Servlet Load Time

This one uses some intermediate groovy knowledge with attributes param taking a closure. It allows user to perform transformation on the MBean parameters. Below in the script it performs operates on long objectName attribute to get servlet name parameter.


jmx {
    server "service:jmx:rmi://localhost/jndi/rmi://localhost:1090/jmxconnector" {
      query "jboss.web:*" {
        findAll "j2eeType=Servlet" {
            chart{
              chartType="Bar"
              attributes={m-> [m.loadTime, m.objectName.find("name=([^,]*)"){it[1]}]}
              labels=["Load Time per Servlet", "Servlet", "Time"]
              options=[false, true, true]
              windowTitle="JBoss Servlet Processing Time"
              width=1200
              height=700
              orientation="HORIZONTAL"
              refreshRate=5000
              show()
            }
        }
      }
    }
}

and here is the output...

Click here for bigger image

Further you can,

Add different type of chart support to the DSL - Bar, XY, trending, etc
Use/Extend to work with different application server and/or application

Complete source code can be found at http://github.com/kartikshah/jmx-dsl

Blogged with the Flock Browser

Exploring Google Collections - Part 2

noreply@blogger.com (Kartik Shah) — Sat, 06 Feb 2010 03:35:00 +0000

In Part 1, we focused on few classes of google collection. Here in part 2, we will expand on the simple scenario and work on DOM like tree structure scenario.

Consider a scenario of an external API provided by vendor. The API had tree like data structure, described by diagram below.

Now the problem was that the main data structure part of external API was POJO. Though it was a tree structure it did not provide any operations for traversal or find operations. Here is the watered down version of the component.

public class NodeComponent {
    private String name;
    private List<NodeComponent> children;
    private Map<String, String> attributes;

    public NodeComponent(String name)
    {
        this.name = name;
    }

    // Getters and Setters omitted for brevity
}

Let's see how google collection can be used to provide nice searcher methods. Consider following use cases:

Find child nodes based on node name
Find child nodes based on attribute name and value
Find child nodes on composite criteria of node name and attribute name and value

Lets implement cases using google collections.

Case 1: Find child nodes based on node name
First we will define NodeNameCriteria implementing Predicate interface


public class NodeNameCriteria implements Predicate<NodeComponent>
{
    private String nameCriteria;

    public NodeNameCriteria(String nameCriteria)
    {
        this.nameCriteria = nameCriteria;
    }

    public boolean apply(NodeComponent node) 
    {
        return node.getName().equals(nameCriteria);
    }
}

Further, define NodeSearcher class which will represent current node and will provide all these helper methods


public class NodeSearcher
{
    private NodeComponent currentNode;

    public NodeSearcher(NodeComponent currentNode)
    {
        this.currentNode = currentNode;
    }

    public Collection<NodeComponent> findChildrenByNodeName(String name)
    {
        return Collections2.filter(currentNode.getChildren(),new NodeNameCriteria(name));
    }
}

Case 2: Find child nodes based on attribute name and value
Similar to first approach, this requires creating a Predicate implementation, let's name it AttributeNameValueCriteria


public class AttributeNameValueCriteria implements Predicate<NodeComponent>
{
    private String nameCriteria;
    private String valueCriteria;

    public AttributeNameValueCriteria(String nameCriteria, String valueCriteria)
    {
        this.nameCriteria = nameCriteria;
        this.valueCriteria = valueCriteria;
    }

    public boolean apply(NodeComponent component)
    {
        Set<Map.Entry<String, String>> entrySet = component.getAttributes().entrySet();

        Set<Map.Entry<String, String>> matchedAttrSet = Sets.filter(entrySet, new Predicate<Map.Entry<String, String>>(){
            public boolean apply(Map.Entry<String, String> entry) {
                return nameCriteria.equals(entry.getKey()) &&
                    valueCriteria.equals(entry.getValue());
            }
        });
        return matchedAttrSet != null && !matchedAttrSet.isEmpty();
    }
}

Case 3: Find child nodes on composite criteria of node name and attribute name and value
So far so good but the real benefit of defining the Predicate comes with the third case. In this case what is needed is to define a criteria which is composition of NodeNameCriteria and AttributeNameValueCriteria. We don't need to define third criteria implementation. Instead we will use Predicates compose method. Add following implmentation of findChildrenByNodeNameAndAttributeNameValue to NodeSearcher

public Collection<NodeComponent> findChildrenByNodeNameAndAttributeNameValue(String nodeName, String attrName, String attrValue)
{
     Predicate<NodeComponent> compositePredicate = Predicates.and(new NodeNameCriteria(nodeName),
                                                        new AttributeNameValueCriteria(attrName, attrValue));
     return Collections2.filter(currentNode.getChildren(), compositePredicate);
}

In simpler approach, one would have defined filter criteria as condition of if loop. But with implementing them as Predicate implementation, you can reuse criteria implementation and mix and match with help of following Predicates function and and or methods.

Blogged with the Flock Browser

Exploring google collections - Part 1

noreply@blogger.com (Kartik Shah) — Mon, 01 Feb 2010 04:49:00 +0000

Google Collection 1.0 released
Google collections finally hit version 1.0. API provides rich data structures building on JDKs collection API. It also provides utility classes for working with collections. With this post I want to focus on few google collections classes and demonstrate how they help in working with collections. In part 1 of the post, we will deal with very simple example. Later in part 2 of the post, we will work with more involved real life example.

We will focus on these classes:

Scenarios
While working with collections you may have noticed that code gets cluttered with for loops, nested for loops, or with for loops with embedded if loops. In most of these scenario what you are trying to do is to iterate through a given collection, either to search for specific elements or to operate on specific elements.

Lets break down the typical cases:
1. Iterating through a given collection and finding a specific element(s) based on criteria


for(Element element: elementsCollection){
     if(element.equals(something){ //Or any other test criteria based on state of element object
          //either add to collection of elements
          //or return this element
     }
}

2. Iterating though a given collection and if certain criteria matches perform operation on the collection


for(Element element: elementCollection){
     if(element.equals(something){
          //or operate on this element
     }
}

As an outcome of this, code ends up with

Code duplication of your iteration logic
Code duplication of your condition (matching criteria)
Code duplication of operation on the specific elements (This however can be avoided, if operation is re-factored into a method)

Google Collections Approach
Lets start with very simple example, here is how google collection can help:
1. Define you matching criteria using Predicate like


            Predicate<String> validEmailAddress = new Predicate<String>(){
                public boolean apply(String emailAddress){
                    return emailAddress.contains("@");
                }
            };

This can be used as


List<String> emailList = Lists.newArrayList("abc@gmail.com", "xyz@gmail.com", "aaagmail.com");
Collection<String> validEmailList = Collections2.filter(emailList, validEmailAddress);

2. Define your operation on element using Function like


        Function<String, String> extractUserName = new Function<String, String>(){
            public String apply(String emailAddress)
            {
                String userName = emailAddress.substring(0, emailAddress.indexOf('@'));
                return userName;
            }
        };

This can be used as


        List<String> emailList = Lists.newArrayList("abc@gmail.com", "xyz@gmail.com", "aaa@gmail.com");
        List<String> userNameList = Lists.transform(emailList,extractUserName);

We will take on more complex DOM-like tree based data structure in part 2 of the post...

Blogged with the Flock Browser

Helpful tips - Working with AST Transformation

noreply@blogger.com (Kartik Shah) — Sat, 23 Jan 2010 16:19:00 +0000

I have been playing with Groovy AST transformation for some time now. You can walk through some of the earlier posts here. With this entry I want to capture some of the lessons learnt, tips, etc.

Using AstViewer Tool (Inspect Ast)
With Groovy 1.6 AST transformation, one of the difficulty was to come up with tree structure for the code you want to inject or change. It involves calling complex set of Statement and Expression APIs. It is ridden with intricate details of language grammar. With groovy 1.7 AstBuilder, you can use buildFromString and buildFromCode which is lot more easier. But still if you have to use buildFromSpec you will need to have some understanding of Statement/Expression API.

One of the tool I have found useful is Groovy Console's AST Browser or Eclipse's ASTViewer. Write code you want to generate language structure for and try to replicate using API or using AstBuilder's buildFromSpec. The tool has gotten better from 1.6 to 1.7.
Open Groovy Console and type in following code


            def advice = new String("Hello AST")
            println advice

Go to Script --> Inspect Ast (or press Ctrl+T) and you will see the AST structure needed.

Educated Guess in Script
With AstBuilder now you can just start building AST in main method or just as script. This has proven to be useful if you just want to compare what you are generating with what is displayed in AstBuilder.


def nodes = new AstBuilder().buildFromXXX{
}

Working with Variables
If you are using variables that gets replaced during runtime you may face problem generating AST. One of the method that I found useful was to replace the variable with static value and see if it works. If it does than the problem is possibly due to the type of the variable.
For example, I wanted to generate


          def advice = new <<variableClassName>>()

where variableClassName will be passed in from annotation value. One option is to try generating it with static value like java.lang.String (or just String), if it works than there is some problem with the type of the variable you are passing.

Compile file individually (if using IDE)
Sometimes the tools you are working with just does not seem to give different output even when you are changing things around in your code. Sometimes stale classes just gets called and you never see the output from the updated classes. You may want to compile both transformation class and the class where the transformation gets used.

ByteCode Viewer
Also using Byte Code Viewer like this helps in looking at the classes that are getting generated. You can also write your class how it will look after AST transformation. Use byte code viewer to see the class and compare it with the class that is being generated through AST Transformation. See which statements are properly generated and which are not.

Testcases
Another extremely resourceful source for AstBuilder is test cases found here. It contains rich example set of how to use each node in the builder specifically for buildFromSpec option. AstAssert is another excellent utility to compare and test expected and actually generated structure. Looks like AstAssert class is not packaged with groovy-1.7, so you may need to download and drop it in your project.

Divide and Conquer
Treat each section of code for which AST is being generated as separate section. Comment out blocks of code and gradually uncomment them to see which section is actually failing and/or not giving desired output.

Hope this helps! Drop in a line to share your tips or problems you face with Groovy's Ast transformation.

Blogged with the Flock Browser

Revisiting Groovy AST Transformation with AstBuilder

noreply@blogger.com (Kartik Shah) — Mon, 18 Jan 2010 06:37:00 +0000

Groovy AST is powerful tool to inject changes into classes at various compile phases. It allows one to operate on AST (Abstract Syntax Tree) to manipulate class structure.

With Groovy 1.7 AstBuilder there are easier ways to write AST transformation. One can build AST nodes using one of following three options

Build From String
Build From Spec
Build from Code

Previously on this blog, we experimented with Groovy 1.6 AST Transformation Example and also AOP style usage at Groovy AST Transformation - AOP Style. Lets revisit this transformation from earlier post.

Lets see how we can generate the AST tree using these new methods from AstBuilder. Recap this previous post before proceeding ahead.

Looks like we need to rewrite initAdviceCall and createMethodCall to experiment with new methods of AstBuilder.

Build From String
Using buildFromString the task would become very easy. With this options, it is not required to build complicated statement and expression structure. All you need to do is embed code within string and pass it in to buildFromString method. If code you generate is not completely static in nature, you may have to parameterize the value. Here in this example, the declaration of "advice" instantiation depends on the Class value defined in the annotation.


  def initAdviceCall(classNodeType) {
    def codeString = "def advice = new " + classNodeType.type.typeClass.canonicalName + "()"
    def buildNodes = new AstBuilder().buildFromString(CompilePhase.SEMANTIC_ANALYSIS, true, codeString)
    Statement stmt = buildNodes[0].statements[0]
    stmt
  }

  def createMethodCall(method, Parameter[] parameters) {
    def codeString = "advice.before (" + "'$method.name'" +", [" + parameters.collect {it.name}.join(',') + "])"
    def buildNodes = new AstBuilder().buildFromString(CompilePhase.SEMANTIC_ANALYSIS, true, codeString)
    Statement stmt = buildNodes[0].statements[0]
    stmt
  }

Build From Spec
buildFromSpec option is more verbose. It also requires you to understand internal structure that is required to be build. With use of GroovyConsole's Inspect Ast option, it is not difficult to come up with the AST structure.


  def initAdviceCall (classNodeType) {
    def buildNodes = new AstBuilder().buildFromSpec {
      expression {
        declaration {
          variable "advice"
          token "="
          constructorCall(classNodeType.getType().getTypeClass()){
            argumentList()
          }
        }
      }
    }
    Statement stmt = buildNodes[0]
    stmt
  }

  def createMethodCall(method, Parameter[] parameters) {
    def buildNodes = new AstBuilder().buildFromSpec {
      expression {
        methodCall {
          variable "advice"
          constant "before"
          argumentList {
            constant method.name
            list {
              parameters.each {variable it.name}
            }
          }
        }
      }
    }
    Statement stmt = buildNodes[0]
    stmt
  }

There are some differences between using API directly and using spec builder option. One thing specifically that threw me off was that ConstructorCallExpression takes in ClassNode as parameter, however the constructorCall in builder takes Class. It wouldn't have been much of an annoyance, if I had read GEP-2 which states that any expression taking ClassNode, for builder you just need to pass class instance. Well in this example, value provided by annotation was ClassNode, so I had to get Class and pass it on to constructorCall.

Build From Code
Unlike above two options buildFromCode option did not work for me in above scenario. The problem I faced was I couldn't do def advice = <VariableClassName>() without using reflection. Method would have looked something like this


def initAdviceCall(classNodeType){
new AstBuilder().buildFromCode {
     def advice = classNodeType.getType().getTypeClass().newInstance()
}

Similarly, for createMethodCall, I would have to use reflection for passing in parameters.

More later...

Blogged with the Flock Browser

Groovy Truth

noreply@blogger.com (Kartik Shah) — Tue, 05 Jan 2010 13:48:00 +0000

Groovy had its truth. Groovy expanded on what Java allows to use in various control structures. While Java allows only boolean expression in if and while control structures, Groovy relaxed the requirement. In Groovy you can use objects in if and while expressions. Only a non-null and non-empty string will evaluate to true. A non-empty collection will evaluate to true. Non zero numbers will evaluate to true.

With Groovy 1.7, Groovy Truth got better. You can now customize the Truth based on your requirement. This can be done by implementing boolean asBoolean() method. Implement the method on any object and logic defined in the method will be used to evaluate. Look at the following example


class Account {
     ...
     String status
     boolean asBoolean(){
          if(status != 'active') 
               false
          else 
               true
     }
}
assert new Account(status:'active')
assert !new Account(status:'prospect')

You can also expand (or change) existing classes like String, Integer, etc using metaClass. Let's expand the groovy truth to return null if the string value is 'null'


     String.metaClass.asBoolean = {
                                    if(delegate  == null || delegate == '' || delegate == 'null') false
                                    else true
                                  }

     String nullString = null
     String nullLiteralString = "null"
     String validString = "JohnDoe" 
     String emptyString = ''
     
     assert !nullString
     assert !nullLiteralString 
     assert validString
     assert !emptyString

Asserts are one place where the overridden logic will be applied. It works with if,while, and ternary operator as well.


//continued from previous script
println "null"?"FALSE":"TRUE"

str1 = "null"
if(str1) println "Valid String" 
else println 'Null, empty or "null" literal string'

If you do not want to override the way an object evaluates to boolean globally. You can use categories.


class StringNullTruthCategory{
    boolean asBoolean(String str){
        if(str || str == 'null')  false
        else true
    }
}

use(StringNullTruthCategory){
   String parameterValue = "null"
   if(parameterValue)   {
       println 'Should not be printed'
   } else {
       println 'Should print'
   }
}

def httpRequestParmValue = "null"
print 'Valid Value: '
println httpRequestParmValue?'YES':'NO'

Similarly, you can expand Integer's truth value if value is either zero or negative value. you can expand truth value of Map based on specific key-value. You have to be careful in doing so though, completely overriding the evaluation criteria can lead to confusion. You can apply on customize builder object. Let's say you have customized builder parsing order.xml into OrderEntity who has multiple OrderItems. OrderEntity truth can be derived from all of its OrderItem.


class OrderEntity{ 
     ...
     OrderItem[] orderItems
     String status
     ...
     boolean asBoolean(){
          //Define based on status or status of orderItems or presense/absence of orderItems
     }
}

if(currentOrder) { 
     //do time consuming process
}

You can define asBoolean objects in your Java class and use that definition to determine boolean value. However, you have to be careful of legacy Java Objects which has method asBoolean already defined for different purpose.

Blogged with the Flock Browser

Mule Notes

noreply@blogger.com (Kartik Shah) — Thu, 31 Dec 2009 03:43:00 +0000

Mule Mind Map

Click Here for full size image

Blogged with the Flock Browser

Grails: belongsTo

noreply@blogger.com (Kartik Shah) — Fri, 20 Nov 2009 05:24:00 +0000

In Grails hasMany and belongsTo are used to define relationship between objects. There are various other uses of belongsTo. Trying to capture them here in this short post.

Let's take an example, say you have domain class TaskList which had many Task

Direction of Access
In class Task if you define belongsTo as
belongsTo = TaskList
it means that Task object can not access its associated instance of TaskList
If it is defined as
belongsTo = [taskList: TaskList]
it means that Task object can access its associated instance of TaskList by doing task.taskList

Cascading Deletes
Only if belongsTo is defined cascading delete will happen. If TaskList is deleted all Tasks part of that TaskList will be deleted.

addTo* method access
For many-to-many relationship, defining belongsTo enables dynamic addTo* method
Extending previous example, say each Task can have multiple Tag and a Tag can be given to multiple Tasks. Defining belongsTo enables addTo* dynamic methods. e.g. task.addToTags(tag)

Blogged with the Flock Browser

Groovy AST Transformation - AOP Style

noreply@blogger.com (Kartik Shah) — Fri, 10 Jul 2009 05:25:00 +0000

In my last blog about AST Transformation I followed a sample example and created AssertParamsNotNull local transformation. Further, I wanted to create AST Transformation which is generic enough and performs any check before method is executed. In process, goal is to learn about AST Transformation.

BeforeAdvisor AST Transformation
Use case for BeforeAdvisor AST involves:

Authorization Checking - Security by checking role from context
Print Parameter values with which the method is called
Asserts Parameters are not null
Check various entry-conditions/Pre-Conditions of the method

To make transform generic enough, idea is to inject a method call to before method of advice for each method annotated with @BeforeAdvisor(MyPreConditionAdvice). Advice method can choose to implement any checks/conditions to be performed before actual methods gets executed.

Few of the subtle characteristics of this type of solution are

It does not allow changing the method parameters.
It does not allow to stop execution of method. However, you can throw runtime exception.
Advice needs no arg constructor and must implement method before

Usage and Client
Starting with class that will use this transform. Following defines script-level method with annotation having advice information.


package com.learn.sts.groovy.ast

@com.learn.sts.groovy.ast.BeforeAdvisor(value= com.learn.sts.groovy.MyAdvice)
def sayHello(name, name2)
{
    println "Hello " + name + name2
}

sayHello("World", "Groovy")

Sample Advice
The advice that we will try to invoke will be something like this. This advice just prints parameter value that method is being invoked with. But you can implement any of the use cases described above.


package com.learn.sts.groovy

public class MyAdvice
{
    def before(String methodName, List listArg)
    {
        println 'Entering Method ' + methodName + ' with params ' + listArg
    }
}

Annotation
Now lets create the Annotation


package com.learn.sts.groovy.ast;

import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target
import java.lang.annotation.ElementType
import org.codehaus.groovy.transform.GroovyASTTransformationClass
@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.METHOD])
@GroovyASTTransformationClass(["com.learn.sts.groovy.ast.BeforeAdvisorASTTransformation"])
public @interface BeforeAdvisor {
    Class value ();
}

One difference to this annotation is that it declares a method value(). This allows us to get value being passed during annotation declaration on method.

AST Transformation


package com.learn.sts.groovy.ast

//Imports section skipped for brevity

@GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS)
public class BeforeAdvisorASTTransformation implements ASTTransformation 
{
    public void visit(ASTNode[] nodes, SourceUnit source)
    {
        AnnotationNode node = (AnnotationNode) nodes[0];
        final Expression classNode = node.getMember("value")
        
        List<MethodNode> allMethods = source.AST?.classes*.methods.flatten()
        List annotatedMethods = allMethods.findAll{ MethodNode method ->
                 method.getAnnotations(new ClassNode(BeforeAdvisor))
                }
        annotatedMethods.each{MethodNode method -> 
             List existingStatements = method.getCode().getStatements()
            Parameter[] parameters = method.getParameters()
            int i = 0;
            existingStatements.add(i++, initAdviceCall(classNode))
            existingStatements.add(i++, createMethodCall(method, parameters))
        }
      }
  
    public Statement initAdviceCall(classNode)
    {
      return new ExpressionStatement(
              new DeclarationExpression(
                      new VariableExpression("advice"),
                      new Token(Types.ASSIGNMENT_OPERATOR, "=", -1, -1),
                      new ConstructorCallExpression(classNode.getType(), new ArgumentListExpression())
                      )
              )
    }
 
  public Statement createMethodCall(method, Parameter[] parameters){
      List parameterExpressionList = new ArrayList()
      parameters.each{ parameter -> parameterExpressionList.add(new VariableExpression(parameter))}
      return new ExpressionStatement(
        new MethodCallExpression(
            new VariableExpression("advice"),
            "before",
            new ArgumentListExpression(new ConstantExpression(method.getName()),
                    new ListExpression(parameterExpressionList)
            )
        )
      )
  }
}

Couple of things to notice here:

First, creation of the AST tree involves

Creating instance of Advice Class - This is done through method initAdviceCall
Invocation of the method before with methodName and Parameter List - This is done through method createMethodCall

Creating AST structure is not the easiest task. One approach that has worked for me is the write the sample code that you are trying to generate AST for and then use Groovy AST viewer in eclipse.

Second, inspecting the value on ASTNode and getting Advice class value. First node contains information about the annotation and second node is the annotated node.

Lessons Learned

Compilation
It was required to compile Transformations files first and then compile the classes that use AST Transformation. If I compiled all three of them together the AST Transformation did not kick in. I used Eclipse IDE and it forced me to use Compile Groovy File explicitly each time I made change. "Build Automatically" or "Clean" option did not work.

Script Level Methods vs all Class Methods
In AssertParamsNotNull AST Transformation in previous blog, I used source.getAST()?.getMethods() what it did is that it only found top level (Script Level) methods. So AST Transformation did not apply to Class Methods, it only got applied to Script Level Method. For this one I changed to source.ast?.classes*.methods.flatten() (Line 12 in BeforeAdvisorASTTransformation). This also become apparent once you see AST structure for a given class using Groovy AST View in eclipse.

Retrive value from annotations
ASTNode being passed to visit method carries information about the annotation. First element (node[0]) contains information about annotation. You can get the values passed in to annotation during the visit method and use it during the transformation. In case above we get class passed in as value and it gets instantiated and before method gets invoked.

Groovy Compilation
When groovy compiler is invoked, any sourcefile.groovy goes under series of transformations.
From Source --> ANTLR Tokens --> ANTLR AST --> Groovy AST --> Bytecode

Using AST Transformation, we manipulate the way groovy AST gets generated. It allows to insert additonal statements.

There are various CompilerPhase that goes along with this process. I couldn't find much documentation on the process. Best information is found on Jochen Theodorou's blog post

Blogged with the Flock Browser

JMX, Groovy, JFreeChart and Swing

noreply@blogger.com (Kartik Shah) — Sun, 21 Jun 2009 04:35:00 +0000

Recently, I worked on creating JMX based dashboard application showing Weblogic instances' information. During the exercise I came across JMX and Groovy examples

Using JMX, Groovy, Swing and JFreeChart, I came up with a dashboard application which looks like this.

It show memory usage on two instances. It automatically updates every 10 seconds and redraws chart.

Surprisingly (or not so surprisingly), code required to be written to perform this activity is minimal. Specifically thanks to Groovy's SwingBuilder and JMX libraries. Groovy makes the task of using 4 different technologies really easy and fun.


import org.jfree.chart.ChartFactory
import javax.swing.WindowConstants as WC
import javax.management.remote.*
import javax.naming.Context;

/**
 * WeblogicJMXReporter monitors Memory value for given node
 * @author kartik.shah
 */
public class WeblogicJMXReporter
{

    static void main(String[] args)
    {
    	def env = [:]
    	env["java.naming.factory.initial"] = "weblogic.jndi.WLInitialContextFactory"
    	env[Context.SECURITY_PRINCIPAL] = "system"
    	env[Context.SECURITY_CREDENTIALS] = "*******" //Password is always stars
    	env[JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES] = "weblogic.management.remote"
    	def jmxServiceURL = new JMXServiceURL("http", "10.50.120.110", 19400,"/jndi/weblogic.management.mbeanservers.domainruntime")
    	def server = JMXConnectorFactory.connect(jmxServiceURL, env).MBeanServerConnection

    	def chart1Map = getChart("node1_svr", server);
    	def piedata1 = chart1Map."pieData"
    	def chart1 = chart1Map."Chart"
    	def jvmInfo1 = chart1Map."MBean"
    	
    	def chart2Map = getChart("node2_svr", server);
    	def piedata2 = chart2Map."pieData"
    	def chart2 = chart2Map."Chart"
    	def jvmInfo2 = chart2Map."MBean"
    	
    	def swing = new groovy.swing.SwingBuilder()

    	def frame = swing.frame(title:'Weblogic INT Memory Usage', defaultCloseOperation:WC.EXIT_ON_CLOSE, 
    	                        size:[800,600], locationRelativeTo: null) {
    	    borderLayout()
    	    panel(id:'canvas') { rigidArea(width:700, height:250) }
    	}

    	def bounds1 = new java.awt.Rectangle(0,0, 350,250).bounds
    	def bounds2 = new java.awt.Rectangle(351,0, 350, 250).bounds
    	while(true)
    	{
    	    recalculatePieData(piedata1, jvmInfo1)
    	    recalculatePieData(piedata2, jvmInfo2)
        
        	chart1.fireChartChanged()
        	chart2.fireChartChanged()
        	
        	frame.pack()
        	frame.show()
        	chart1.draw(swing.canvas.graphics, bounds1)
        	chart2.draw(swing.canvas.graphics, bounds2)
        	sleep(10000)
    	}
    }
    
    static getChart(nodeName, server)
    {
    	def jvmInfo = new GroovyMBean(server, 'com.bea:Location='+nodeName+',Name='+nodeName+',ServerRuntime='+nodeName+',Type=JVMRuntime')
    	def piedata = new org.jfree.data.general.DefaultPieDataset()
    	piedata.setValue "Free", jvmInfo.HeapFreeCurrent
    	piedata.setValue "Used", jvmInfo.HeapSizeMax - jvmInfo.HeapFreeCurrent
    	def options = [true, true, true]
    	def chart = ChartFactory.createPieChart(nodeName, piedata, *options)
    	chart.backgroundPaint = java.awt.Color.white
        
        ["MBean":jvmInfo, "pieData":piedata, "Chart":chart]
    }
    
    static recalculatePieData(piedata, jvmInfo)
    {
    	piedata.setValue "Free", jvmInfo.HeapFreeCurrent
    	piedata.setValue "Used", jvmInfo.HeapSizeMax - jvmInfo.HeapFreeCurrent
    	println piedata.getValue("Free")
    	println piedata.getValue("Used")

    }
}

For purpose of this blog and keeping it simple, I kept it all in one groovy class.

Blogged with the Flock Browser