Scratching my programming itch

Book Review: Programming Concurrency on the JVM

2012-01-30T21:02:00.002-06:00

Overview
Programming concurrency is a tough task to get right, just ask Venkat, he'll tell you so! I attended a presentation by Dr. Venkat Subramaniam on this very subject, Concurrency on the JVM, at my local Java user group a couple months ago and he really caught my attention. The presentation was a condensed version of the book, and needless to say, I bought the book to help fill in all the details.

Not only is Venkat a 'subject matter expert', but his presentation kept everyone engaged and laughing!

Contents:
The book is broken down in 5 parts, with 10 chapters total, at just over 250 pages. The main sections of the book are:

Strategies for Concurrency - this discusses the impact of multi-core machines and the three design approaches that are discussed throughout the rest of the book, Shared Mutability, Isolated Mutability and Pure Immutability.

Modern Java/JDK Concurrency - covers the Java 1.5+ concurrency features and talks about how to use the newer concurrency classes, like ExecutorService, CountDownLatch, Locks and Fork/Join to solve concurrency problems.

Software Transactional Memory - covers Software Transactional Memory (STM), popularized by Clojure, which places access to memory within transactions. STM is one option for getting away from the 'Synchronize and Suffer' model as Venkat call it. The intention is to make threaded code more deterministic. Using the JDK concurrency tools, there is no way to be sure you code is correct because it doesn't always fail when/where you need it to. By putting the memory access inside transactions, the transaction manager helps resolve the conflicts without explicit locking.

Actor-based Concurrency - covers actor design approaches as another option for concurrency design without using synchronization. This section exercises actors from the Akka library from Scala, Actors from the Groovy library GPars, and mixing Actors and STM as a possible solution.

Epilogue - is a short recap of the topics presented previously and points out the scenarios when each of the solutions would be appropriate.

The book lays out a couple example problems and then develops solutions for these problems across a range of JVM languages (Java, Groovy, Scala, Clojure and JRuby). The examples are short and concise, enough to help you to get started prototyping against your own concurrency issues.

Summary
This is a good book to get you started on some of these newer concurrency concepts and libraries. It was an easy read, and I completed the reading fairly quickly, as opposed to the "Java Concurrency in Practice" book, which I started and stopped twice and still have not completed yet!

I would recommend this book to anyone looking to improve their programming skills in the concurrency area. The book provides you with some alternatives to the old standby of ' Synchronize and Suffer' model.
My last recommendation, if you get the opportunity, attend one of Venkat's presentations, you won't be sorry!

Passing parameters into Groovy script using Binding class

2012-01-19T22:03:00.001-06:00

I recently saw a question posted on the Groovy/Grails group on LinkedIn asking about ways to pass in parameters to a Groovy script. There were several responses pointing to the CliBuilder class which is certainly one way to handle the problem. I had just finished reading an article by Ken Kousen in the November issue of GroovyMag where Ken mentioned another option: using the Binding class.

The example below shows a couple ways of setting variables in the binding, getting the variable values from the binding and how to capture standard output from the script. The one tricky part is the question "When is something in the Binding and when not?" The answer is: when it's not defined, it is in the binding! In the example below, variable c is placed in the binding, but since it is def'd in the script, the value for that variable comes from the local variable rather than the binding.

// setup binding
def binding = new Binding()
binding.a = 1
binding.setVariable('b', 2)
binding.c = 3
println binding.variables

// setup to capture standard out
def content = new StringWriter()
binding.out = new PrintWriter(content)

// evaluate the script
def ret = new GroovyShell(binding).evaluate('''
def c = 9
println 'a='+a
println 'b='+b
println 'c='+c 
retVal = a+b+c
a=3
b=2
c=1
''')

// validate the values
assert binding.a == 3
assert binding.getVariable('b') == 2
assert binding.c == 3 // binding does NOT apply to def'd variable
assert binding.retVal == 12 // local def of c applied NOT the binding!

println 'retVal='+binding.retVal
println binding.variables
println content.toString()

Output

[a:1, b:2, c:3]
retVal=12
[a:3, b:2, c:3, out:java.io.PrintWriter@1e0799a, retVal:12]
a=1
b=2
c=9

Hope this helps!

Book Review: SQL Antipatterns Avoiding the Pitfalls of Database Programming

2012-01-17T22:30:00.001-06:00

Overview
This is book for software developers that are either new to database programming (SQL) or those that may not have had any formal database education and may have learned 'on-the-job'.

The book is bit longer, approximately 300 pages, than some of the other titles I've read from The Pragmatic Programmers shelf, but it is a very easy read. The chapters all follow a similar format: Object, Antipattern, How to Recognize the Antipattern, Legitimate Uses of the Antipattern and Solution. Each chapter the author leads you through a problem that needs correcting within the scope of the database. Next the antipattern is laid out for you and the author shows the disadvantages of using the antipattern. Tips are provided to help identify the antipattern and he also suggests some possible legitimate uses of the antipattern. Finally, the author describes better solutions to the problem presented; ones without the disadvantages discussed previously and solutions that are considered 'best practices' and that will provide better database performance.

Contents
The book is broken down into 4 main sections:

Logical Database Design Antipatterns - planning database tables, columns and relationships
Physical Database Design Antipatterns - defining tables, indexes and choosing datatypes
Query Antipatterns - SQL command usage, for example, SELECT, UPDATE and DELETE
Application Development Antipatterns - correct usage within the scope of a language

Some of that antipatterns described may make you laugh and wonder, "Who the heck would do something like that?" while others you may have run into previously and some might even hit very close to home from your work or personal projects.

There is also an appendix that covers the Rules of Normalization. I had heard of terms like Normalization, First Normal Form, and Second Normal Form before, but never actually read any explanations. The appendix includes discussions on normalization and first normal form through fifth normal form with examples to help clarify the topics.

Summary
This is a very good book for software developers just starting to 'cut their teeth' with databases and SQL. It also serves as a good refresher for those with a bit more database experience. In my case, I fall into the later scenario. You can view it as a book of short stories, showing what not to do in these cases.

Jars signed with mutliple code signing certificates?

2011-10-26T13:27:00.001-05:00

For anyone that has built and maintained Web Start applications, you have probably been through this issue before. Your QA group or worse yet, a customer, calls to tell you that they cannot start your application because the download failed because "jar resources in jnlp are not signed by the same certificate".

If want your Web Start application to have full access permissions, then you need to sign all the jars that get downloaded. This generally presents the opportunity to encounter one of the following errors:

None of the jars are signed. Most likely a build issue and generally only happens once, (I hope!)
Single jar is not signed. Again, most likely either a build or process error.
Not all jars signed with the same certificate.

The first two are easy to resolve. The last is a bit of a pain because Web Start doesn't bother to tell you exactly which jars where signed with different code signing certificates. Now, it's time to make some educated guesses as to what changed recently and look at the most likely/usual suspects. Another option is to get the list of download jars from the JNLP document and figure out which certificates(s) each jar has been signed with.

We have been through this at work once or twice and so I finally decided to make it easier for the next time this happens. The approach I took was to read all the jars in a deployment folder, open them and look for the signature files inside the jar file. For the results, I created a map keyed by the jar name and the value being a list of the signature file names (*.RSA, where the * represents our code signing certificate alias). Keeping a list in the map allows for cases where a jar may have multiple signature files, which was encountered when we switched over code signing certificates. Writing this in Java would be possible, but why write so much code? Why not write a script in Groovy, it would be some much shorter and concise.

Groovy Script

def dir = new File("C:\\MyDeployment\\jboss\\myserver\\deploy\\myserver.ear")
def jars = dir.list( { d, f-> f ==~ /.*.jar/ } as FilenameFilter)
def map = [:]
jars.each() {
   def zipFile = new java.util.zip.ZipFile(new File(dir, it)) 
   zipFile.entries().each { zipEntry ->
      if (zipEntry.name.endsWith('.RSA')) {
          if (map.containsKey(it)) {
             def list = map[it]
             list << zipEntry.name
          } else {
             map[it] = [zipEntry.name]
          }
      }
   }  
}
map.each{ println it}

Hope this helps!

Lessons learned from recent PA/DSS audit

2011-09-29T16:11:00.000-05:00

The following are some tips/hints and lessons learned during our recent PA/DSS audit. Everyone's mileage will vary.

Our application runs on both Windows and Linux. We support JBoss and Weblogic for the application server. We support Oracle, SQL Server and MySQL for the database. We have three client applications which are all WebStart clients, so this is NOT a web application. Our EJBs are EJB 2.1 Session Beans.

Hints

Determine exact software stacks needed including all software levels, build levels and whatever else you need. Collect all of this in one safe place.
Get the hardware ready and any scripts that may be needed to use those devices. Our application is a POS, so we had a collection of pin pads that we were required to test with during the audit.
Get any utility software needed

Ghost for Win* images (or Win 7 provides a Backup/Restore feature)

We ran into some problems with Windows Server 2008 and had to use one of the Acronis tools to create the backup image.

PartImage for Linux images
Darik's Boot and Nuke
anything else you might need

Wipe the hard drives before loading any software, including the OS, using Darik's Boot and Nuke to be absolutely sure that the drive does contain anything that might match a credit/debit card number.
Install the OS and any 'base' software, which isn't the software to be verified by the audit

Keep the partition sizes as small as possible without causing disk output problems. The larger the disk, the longer the forensic scans will take to run.
On Windows - you can have a page file, just don't allow it to grow and shrink. Set the min and max file size to the same value.

Problem here is that if card holder data is written to the page file then later the page file size is shrunk, some of that card holder data MAY be inside freespace, which is NOT cleaned up by the OS. This can cause some instances of card holder data to be found during a forensic scan and NO ONE wants card holder data found during the scans.

On Window - you should also set the Windows Update setting to be Notification because you don't want Windows Updates being applied to a machine during the audit.
Our auditor also recommended we turn off the Windows Restore points. Again, you don't want Windows doing any extra work that could impact the audit results.

Take images of all boxes. These may be needed to reset a machine back to a 'clean' state for ad additional set of tests.
Talk to each of your authorization providers

let them know when your audit is scheduled and ensure you will have connectivity to their test servers
Verify that each provider can supply 'magic authorization values' that will help your trigger the following responses:

approvals
declines
referrals
timeouts
void
split tenders
partial auths

If any of the authorization providers encrypt their communications, then you may need to ensure you have whatever is needed for that. We needed to download and install the JCE .
If you support multiple authorization providers, you will have to run tests against each provider so be ready to do whatever it takes to switch your application from one provider to the next.

Practice taking images and restoring images on all boxes

The Linux tools are not as user friendly as the Windows tools, so you need to be ready and able to make and restore your images in a timely manner.

Now you can install your software and you should be ready for the audit.
If you do any testing before the auditor arrives, be ready to restore the machines back to a 'clean' state because you don't want to take a chance that your early testing wrote any card holder data somewhere on the hard drive.
Our auditor previously used WinHex to do the forensic scans. We used the evaluation version to try and double-check ourselves. The problem with that version is that it is slow and can only scan for one card number at a time. This time the auditor had a new set of software for the scans and unfortunately I didn't get the name of the package, although I know it was a purchased product.

Good luck - hope this helps!

Book Review: Building and Testing with Gradle

2011-07-20T12:56:00.002-05:00

Overview

Gradle is one of the new "cool kids on the block". It is quickly gaining momentum as projects like Hibernate and Spring migrate their build processes over to Gradle. Gradle can be considered the 3rd generation of Java build tools, with Ant and Maven representing the earlier generations.

Gradle is built on top of Groovy, which provides the ability to define a Domain Specific Language (DSL). In this case, the Gradle DSL provides a language tailored to the task of building code. If the DSL does not provide exactly what you need, then you can extend the DSL using plug-ins. It is possible to learn Gradle without knowing Groovy, but some initial knowledge of Groovy is beneficial.

This book is short (89 pages) . It brings back memories of the O'Reilly "A Developer's Notebook" series.

Contents

Chapter 1, Hello Gradle This chapter leads the reader thru the installation and configuration of Gradle and demonstrates the ubiquitous Hello World (build file) example.

Chapter 2, Gradle Tasks This is the best chapter in the book. It drives home the concepts associated with a 'task' and how the tasks can be configured and customized. The most helpful suggestion in the book is found in this chapter. The recommendation is that you read the Gradle DSL documentation. I had previously read the Gradle Users Guide but was still a bit fuzzy on what the properties and methods were and how they were used. . Reading the DSL documentation provides the full picture. It also helps drive home that you are working with a programming language/DSL and not a set of XML tags like Ant and Maven.

Chapter 3, Ant and Gradle This chapter compares and contrasts Ant and Gradle. It also demonstrates that Gradle can run Ant tasks by importing an Ant build.xml or by using the Ant Builder that is part of the Groovy distribution.

Chapter 4, Maven and Gradle This is the longest chapter in the book comprising about 25% of the book. Like all of the Gradle documentation, this book re-assures the reader that any previous investment in Maven or Ivy will not be lost in a move to Gradle. Gradle provides support for Ivy, Maven's Central repository and a local repository.

Chapter 5, Testing with Gradle This chapter is a good overview of using JUnit , TestNG, Spock, Geb and EasyB. Coverage of testing is a bit light given the title of this book.

Chapter 6, Multiproject Builds This chapter does a nice job of taking a sample project and showing three different approaches to creating the project build structure. The approaches shown are one master build file, project-specific build files and a hybrid approach. While the sample projects are very small, they provide the reader a starting point to converting their own multiproject builds.

Summary

This is an excellent book for getting started with Gradle. There are plenty of working examples provided in the source code download package. The authors do a very good job of explaining the concepts and presenting alternative ways of expressing the same options within the DSL.

One minor issue is that matching up the downloaded source examples to the examples in the printed chapters is a bit of an exercise because they are not organized in a folder structure by chapter like many other books.

Initially the slim size of the book worried me as to it's ability to provide a depth of coverage of Gradle, but the authors stated in the Preface that "Future volumes will cover the Gradle plug-in ecosystem, how to extend Gradle with your own business logic and even more advanced topics." Sounds good to me, bring it on Tim and Matthew!

Gradle dependencies with jars that have no version number

2011-07-13T23:44:00.000-05:00

I have seen presentations and read blogs and articles on Gradle. It sounds pretty good. Gradle is gathering momentum as some of the major players in the open source community (Hibernate and Spring Security to name a few) have switched their builds over to using Gradle. I figured it is time to give it a try.

The project I work on is fairly old, pre-Maven I suspect. The project is a multi-layer, multi-component Java project built using Ant. The project does not use Maven or Ivy. The project includes a folder that contains all of our dependent jars. These jars are committed to SVN and included as part of the project when it is checked out. This way we have all the dependencies we need to build the project. We have quite a few dependent jars that do not have version numbers in the jar name. Examples include a copy of junit.jar and log4j.jar. This all existed before I started working on the project, so I have no idea how it got that way - it just is. And since it's not really broken, we haven't attempted to fix it!

Not having version numbers in the jars is a minor inconvienence. If you read the Gradle User Guide chapter on Dependency Management, they seems to push dependency management for jars with version numbers. Most of the examples shown in the documentation and elsewhere declare the dependencies as seen in Example 34.1 in Dependency Management chapter. Great, but what do I do if I don't have version numbered jars in my dependencies? I did not believe the Gradle guys would leave me hanging... And they didn't!

I really wanted to figure out how to setup my classpath to include these jars without version numbers. I figured out how to do this and started writing this blog. Then as I re-read the Dependency Management chapter, I saw a clue that I must have blown right by the first time I read it. The clue is in Section 34.3.5 File dependencies. You can use the FileTree to help define the classpath to include these non-versioned jars.

I created a small project called GradleTest that did not use the standard Java plugin conventions for the source and resource files and has dependencies in the "libs" folder, none of which have version numbers in the jar names.

I created two different options, which both seem to work. The first is shown below. This option handles the dependencies more specifically, assigning just the jars that are needed for either compile or runtime.

apply plugin: 'java'

sourceCompatibility = 1.6
version = '1.0'

sourceSets {
    main {
        java {
            srcDir 'src'
        }
        resources {
            srcDir 'resources'
        }
        compileClasspath += fileTree(dir: './libs', includes: ['**/log4j.jar'])
        runtimeClasspath = classes + sourceSets.main.classes + fileTree(dir: './libs', includes: ['**/log4j.jar'])
        
    }
    test {
        java {
            srcDir 'test'
        }
        resources {
            srcDir 'testresources'
        }
        compileClasspath += fileTree(dir: './libs', includes: ['**/junit.jar'])
        runtimeClasspath += classes + fileTree(dir: './libs', includes: ['**/junit.jar', '**/log4j.jar']) 
        
    }
}

sourceSets.main.classesDir = new File("./build/classes")
sourceSets.test.classesDir = new File("./build/classes")

The second option looks very similiar to the first option, but uses the dependencies definition and it cheats by assigning all the jars in the "libs" folder to both the compile and runtime classpath, rather than just exactly what is needed.

apply plugin: 'java'

sourceCompatibility = 1.6
version = '1.0'

sourceSets {
    main {
        java {
            srcDir 'src'
        }
        resources {
            srcDir 'resources'
        }
        
    }
    test {
        java {
            srcDir 'test'
        }
        resources {
            srcDir 'testresources'
        }
        
    }
}

sourceSets.main.classesDir = new File("./build/classes")
sourceSets.test.classesDir = new File("./build/classes")

configurations {
    compile {
        description = 'compile classpath'
        transitive = true
    }
    runtime {
        extendsFrom compile
    }
}

dependencies {
    compile fileTree(dir: 'libs', include: '*.jar')
    runtime fileTree(dir: 'libs', include: '*.jar')

}

I am sure there are other, possibly better or cleaner alternatives. Please feel free to share improvements or other alternatives. Hope this helps....

Groovy vs Java difference provides a WTF moment

2011-05-19T22:06:00.001-05:00

We all (should) know that not ALL Java is valid Groovy, but if you don't spend you day working in Groovy, then sometimes you forget these differences. These 'gotchas' are documented here but who reads this stuff, right? :-)

I wanted to get DatabaseMetadata for a database, specifically a list of table names, so I started with a copy of Java code that already did the same thing.

Class.forName(jdbcDriver);
Connection con = DriverManager.getConnection(jdbcURL, "root", "root");
DatabaseMetaData dbmd = con.getMetaData();
String[] tableTypes = { "TABLE" };
ResultSet allTables = dbmd.getTables(null, null, null, tableTypes);
System.out.println("processing tables...");
while (allTables.next()) {
    String table_name = allTables.getString("TABLE_NAME");
    System.out.println("processing table"+table_name);
}

Should be simple, right - drop the semi-colons and look for other opportunities to shorten the code using Groovy.

def db = Sql.newInstance('jdbc:mysql://127.0.0.1:3306/mydb', 'root', 'root', 'com.mysql.jdbc.Driver')
def metaData = db.getConnection().getMetaData()
println("Driver Name: " + metaData.getDriverName())
println("Database Product: " + metaData.getDatabaseProductName())
println("Driver Name: "+ metaData.getDriverName())
println("Driver Version: "+ metaData.getDriverVersion())
String[] tableTypes = { "TABLE" }
ResultSet allTables = metaData.getTables(null, null, null, tableTypes)
while (allTables.next()) {
    String table_name = allTables.getString("TABLE_NAME")
    println "processing table "+table_name
}

And it almost worked! The thing that threw me was that not only does the code compile, but it also returns correct values for some of the methods in the DatabaseMetadata, but when it came time to iterate across the table names, there were none. Come on, I just ran almost the exact code in Java and it spit out all the table names in my database. What the heck did I mess up this time?

My error was the 3rd bullet on the list of gotchas from above - how NOT to initialize an array in Groovy...

String[] tableTypesJava = { "TABLE" }
String[] tableTypesGroovy = [ "TABLE" ]
println tableTypesJava.class
println tableTypesGroovy.class
assertEquals(tableTypesGroovy, tableTypesJava)  // this will fail!

Both variables say they are String[] but tableTypesJava is really closure when compiled in Groovy! The fix to the Groovy code is to use the brackets when initializing the array and not the braces - basically the tableTypesGroovy from above. Guess it's time to RTFM...

Hope this helps!

Grails AJAX Examples

2011-01-31T21:02:00.001-06:00

Early on, newcomers to Grails encounter a lack of working samples accompanying the framework. Good places for examples and documentation include the Grails documentation and bloggers.

Ajax is one of the features with built-in support by Grails. Once people get their "head around" the Grails basics, they generally want to move onto more interesting features like Ajax. This usually requires scouring the web, looking for examples and/or reading the documentation fairly closely. I went through this process and figured that others might benefit from a set of short working ajax examples in Grails. I have created a small application that demonstrates some of the Ajax features and provide the application for download to anyone interested. (see bottom for download information)

The example application uses each of the following:

FormRemote
RemoteLink
SubmitToRemote
RemoteFunction
RemoteField

General Steps

Pick your javascript library - prototype used in my example. Also shown is some Ajax event registration.



     function showSpinner(visible) {
         $('spinner').style.display = visible ? "inline" : "none";
     }
     Ajax.Responders.register({
     onLoading: function() {
           showSpinner(true);
     },
     onComplete: function() {
     if(!Ajax.activeRequestCount) showSpinner(false);
     }
   });

Update your controller with any code needed - below is my entire BookController

package ajax

class BookController {

     def scaffold = true
  
  def showEditAjax = {
   render (view:'editAjax', model:[bookInstance:Book.get(params.id)])
  }
  
  def showListSelect = {
   render (view:'listSelect', model:[bookInstanceList:Book.list()])
  }
  
  def showTitleSearch = {
   render (view:'titleSearchAjax')
  }
  
  def showRemoteLink = {
   render (view:'remoteLink', model:[bookInstanceList:Book.list()])
  }
  
  def showDetails = {
   render(template:'bookDetails', model:[bookInstance:Book.get(params.id)])
  }
  
  def showSearch = {
   render (view:'searchAjax')
  }
  
  def String wrapSearchParm(value) {
   '%'+value+'%'
  }
  
  def searchTitle = {
   def list = Book.findAllByTitleIlike(wrapSearchParm(params.searchvalue))
   render(template:'searchResults', model:[searchresults:list])
  }
  
  def search = {
   def list
   if (params.publisher && params.title)
    list = Book.findAllByTitleIlikeAndPublisherIlike(wrapSearchParm(params.title), wrapSearchParm(params.publisher))
   else if (params.publisher)
    list = Book.findAllByPublisherIlike(wrapSearchParm(params.publisher))
      else if (params.title)
       list = Book.findAllByTitleIlike(wrapSearchParm(params.title))
   
   render(template:'searchResults', model:[searchresults:list])
  }
  
  def listByPublisher = {
   def list
   if (params.filter.equals("All"))
    list = Book.list()
    else
      list = Book.findAllByPublisher(params.filter)
   
   render(template:'searchResults', model:[searchresults:list])
  }
  
 def update = {
  def bookInstance = Book.get( params.id )
  if(bookInstance) {
   bookInstance.properties = params
   if(!bookInstance.hasErrors() && bookInstance.save()) {
    render "
Book ${params.title} updated with Ajax using FormRemote"
   }
   else {
    render(view:'edit',model:[bookInstance:bookInstance])
   }
  }
  else {
   flash.message = "Book not found with id ${params.id}"
   redirect(action:edit,id:params.id)
  }
   
  }
  
}

Create/update view(s) - See screen shots below and/or download the application

Most of the examples are fairly straight forward and should not need a lot of explanation. They may not be great 'real world' examples but the intention is to provide a working example for someone to start from.

FormRemote
The formRemote taglib creates a form tag that uses a remote uri to execute an ajax call. In this case, I have slightly modified the default generated edit.gsp to change the form tag to a formRemote tag and added a div to show the edit results. The screen shot below shows the result of first clicking the Form Remote navigation menu button, changing the number of pages in the 'Grails in Action' book and pressing the Update button at the bottom of the screen.

RemoteLink
The RemoteLink tag creates a link that calls a remote function when clicked. In the example application, I display a list of book titles and the title is a link. Clicking on the title shows the book details at the bottom of the screen.


    ${fieldValue(bean: bookInstance, field: "title")}

SubmitToRemote
The submitToRemote tag creates a button that submits the surrounding form as a remote Ajax call serializing the fields into parameters. In the example application, I created search fields for Title and Publisher. The search button was created using the SubmitToRemote taglib to submit the search parameters using Ajax.


             Title:                        
             Publisher:

RemoteFunction
The example application uses the RemoteFunction taglib to create a remote javascript function that is assigned to the onChange DOM event. When the user selects or changes the value in the Publisher dropdown list, the remote function is called and the bottom half of the screen displays books published by the selected Publisher.

RemoteField
The example application uses the RemoteField taglib to create a dynamic search for Book titles, similar to Google Instant. Search results are returned the screen as you type values in the Title Search field. The controller code in this case, wraps the search parameter with the wild card character (%) to search the database and returns all matches to be displayed.

Title Search:

Problems Encountered

I ran into problems trying to get the spinner to show correctly until I found this entry on StackOverflow that showed how to register the Ajax events. When running locally on the same box, this happens very fast so you really need to keep and eye on the upper right corner of the screen if you want to see the spinner.
I also ran into problems with the RemoteFunction due to syntax problems. Being used to the taglibs, I was setting the update parameter using update="mydiv" when the syntax should have been update:"mydiv". This did not flag as an error. The controller was called, it just never updated the target div!

Downloads

Download a zip from here
Get a copy from Github - this is my first crack at creating a repo at Github, so if I did it wrong, please let me know.

It was fun putting this together. Hope it helps some of you!

Shutterfly Holiday Collection

2010-11-20T22:18:00.000-06:00

We have used Shutterfly to create family albums and mouse pads in the past. I have found the quality of the albums from Shutterfly is great. The album pictures are clear and sharp. Based on our past experiences and quality of products that we’ve received, we’re excited about using Shutterfly for our Christmas cards this year.

I have found the Shutterfly Christmas card selection is tremendous. In particular, my family was pleased at the number of religious photo cards that are available. We particularly like the following photo cards: here and here . The cards in the links above allow us to include four to five pictures of our family so that we don’t have to agonize of picking the one best picture! Our plan is to use one of these photo cards for the dual purpose of spreading the message of the season, as we believe it, and to share pictures of our family with loved ones and friends.

Enjoy the selection from Shutterfly and Happy Holidays!!!

Be careful using GStrings for SQL

2010-10-20T21:52:00.000-05:00

If you are like me, you love the ease with which you can create Groovy scripts to manipulate your databases. Gone are the days of writing utility classes to make these changes, thanks to Groovy and the SQL package. But if you aren't careful, you can run into a couple of strange errors if you use the GStrings.

For the below example, I have a table named LOCATION in my 'test' database that has two columns: location_id and location_name. The example below works fine and you notice that I am not using any variables in the SQL statement

import groovy.sql.Sql

db = Sql.newInstance('jdbc:mysql://127.0.0.1:3306/test', 'root', 'root', 'com.mysql.jdbc.Driver')

def sqlStatement = """select * from location where location_id > 3"""

db.eachRow (sqlStatement){ 
   println it
}

println "Done!"

Next, try making the SQL a bit more dynamic by specifying the table name, column name and key value as variables within the GString and watch what happens.

import groovy.sql.Sql

db = Sql.newInstance('jdbc:mysql://127.0.0.1:3306/test', 'root', 'root', 'com.mysql.jdbc.Driver')

def tableName = 'location'
def columnName = 'location_id'
def key = 3
def sqlStatement = """select * from $tableName where $columnName > $key"""

db.eachRow (sqlStatement){ 
   println it
}

println "Done!"

WARNING: Failed to execute: select * from ? where ? > ? because: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''location' where 'location_id' > 3' at line 1

The quick fix I found was to call the toString() method on the GString and this resolves the problem. I found a line in "Groovy In Action" which helps explain the problem:

The special use of GStrings as SQL statements limits the use of placeholders to places where a question mark would otherwise be allowed in a prepared statement.

import groovy.sql.Sql

db = Sql.newInstance('jdbc:mysql://127.0.0.1:3306/test', 'root', 'root', 'com.mysql.jdbc.Driver')

def tableName = 'location'
def columnName = 'location_id'
def key = 3
def sqlStatement = """select * from $tableName where $columnName > $key""".toString()

db.eachRow (sqlStatement){ 
   println it
}

println "Done!"

Keep this in mind the next time you use the Groovy SQL class!

Get value from SQL into an Ant property

2010-07-30T16:13:00.000-05:00

Recently I had to adjust some of our JMS configuration and needed make sure we set the ServerPeerID configuration value to a unique value among our servers. We have a unique server number in our database so I figured that I would just use that value. In order to do this, I needed to get the database value into an Ant property, which is then used to generate the configuration file, which includes the ServerPeerID. Below is the Ant task and steps I used to do this.

Steps:

Define where the SQL results will be written
Make sure the file does not already exist
SQL task to get the result
Load the file back into a property

Warning: The syntax highlighter appears to be reformatting my XML and not allowing the self closing tags. Probably my problem but you should be able to get the idea from the


   
   
   
      
      SELECT PREF_VALUE FROM PREFERENCE WHERE PREFERENCE_ID = 71;

Groovy Sockets example

2010-04-28T23:31:00.002-05:00

I found several sites and blogs with Groovy socket examples but none that showed a matched pair of the server and the client sending data back and forth. Collecting bits from different sources, I put together the following server and client scripts using Groovy.

The sample works by sending a line of data as a request; thus the server calls readLine() and the client sends a new line character at the end of the input. The server echos the request back to the client along with a timestamp. For those that skip reading the documentation, ServerSocket's accept method "Accepts a connection and passes the resulting Socket to the closure which runs in a new Thread." so this server creates a new thread for each request.

Groovy Socket server

import java.net.ServerSocket
def server = new ServerSocket(4444)

while(true) {
    server.accept { socket ->
        println "processing new connection..." 
        socket.withStreams { input, output ->
            def reader = input.newReader()
            def buffer = reader.readLine()
            println "server received: $buffer"
            now = new Date()
            output << "echo-response($now): " + buffer + "\n"
        }
        println "processing/thread complete."
    }
}

Groovy Socket client

s = new Socket("localhost", 4444);
s.withStreams { input, output ->
  output << "echo testing ...\n"
  buffer = input.newReader().readLine()
  println "response = $buffer"
}

Give it try! Hope this helps...

Trouble connecting to Web Start client?

2010-04-24T14:44:00.000-05:00

Do you have problems getting either the debugger or JConsole attached or connected to your Web Start client? We worked around this for way too long until we finally found the trick: environment variable JAVAWS_VM_ARGS.

JNLP allows for vm args to be passed to the Java runtime but there is a limited set of arguments that are allowed to be passed on. Anything not in that list of secure properties is ignored. You can find the list of secure properties in the Developers Guide.

As I said ealier, if you want to attach a debugger or JConsole to you Web Start application, then you need to use the JAVAWS_VM_ARGS parameter. For example, if I want to attached to my web start client for debugging purposes then I can do the following:

set JAVAWS_VM_ARGS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

in my environment, start the client and then attach to the process using port 5005. The same goes for attaching to JConsole, set the properties you need in JAVAWS_VM_ARGS, so com.sun.management.jmxremote.* should all be set inside JAVAWS_VM_ARGS.

Hope this help!!!

Ant task ported to Groovy - Part 3

2010-01-03T22:28:00.000-06:00

Welcome to the final posting of this series where I have been porting one of my Ant tasks from Java to Groovy. If you haven't read part 1 or part 2, then you might want to take a quick pass through those before reading this posting. In this posting, I will show the following:

FileValidator class ported to Groovy
Using HTML Builder to generate the HTML report
Full listings of the scripts & classes created

FileValidator ported to Groovy
One of the last tasks left to do is to port the FileValidator class from Java over to Groovy. The FileValidator class is created in the process() method of the Validator class and called for every file to be validated. This is another example of where the Groovy code is a lot shorter and more concise than the Java code. The FileValidator class handles the iteration thru a file, line by line, checking each regular expression checking for a potential error condition.

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.regex.Matcher;

/**
 * File validator runs a list of regular expressions against the file
 * attempting to validate the file.  Errors are kept in a list and
 * then the caller can get the list of errors and log them - right
 * now the ant task will log them. 
 *
 */
public class FileValidator {

    private List regExDescriptors;
    private File file;
    private List errors;
    
    /**
     * Default constructor
     * @param list List of RegExDescriptors used to validate a file
     * @param file File to be validated
     */
    public FileValidator(List list, File file) {
        regExDescriptors = new ArrayList(list);
        this.file = file;
        errors = new ArrayList();
    }
    
    public List getErrors() {
        return errors;
    }
    
    public void validate() {
      // see if we should exclude this file  
      processExcludes();
      
      // Run the regEx(s) against the file to check for problems
      BufferedReader in = null;
      try {
          in = new BufferedReader(new FileReader(file));
          String str;
          int lineNumber = 1;
          while ((str = in.readLine()) != null) {
              for (Iterator iter = regExDescriptors.iterator(); iter.hasNext();) {
                  RegExDescriptor validator = (RegExDescriptor) iter.next();
                  Matcher matcher = validator.getMatcher(str);
                  if (matcher.find()) {
                   errors.add( validator.getDescription() + " on line " + lineNumber);
                  }
              }
              lineNumber++;
          }
      } catch (IOException e) {
          e.printStackTrace();
      } finally {
          if (in != null)
              try {
                  in.close();
              } catch (IOException e) {
              }
      }  
    }

    private void processExcludes()
    {
        String fullName;
        try
        {
            fullName = file.getCanonicalPath();
        }
        catch (IOException e)
        {
            fullName = file.getName();
            e.printStackTrace();
        }
        for (Iterator iter = regExDescriptors.iterator(); iter.hasNext();)
        {
            RegExDescriptor descriptor = (RegExDescriptor) iter.next();
            if (descriptor.excludeFile(fullName))
            {
                iter.remove();
            }
        }
    }    
}

Groovy code for file validation

//**************************************
// process each file, line by line
//**************************************
files.each {fileName ->
  def errors = []
  def lines = new File(fileName).readLines()
  def lineNumber = 1
  lines.each {line ->
    // for each line - iterate thru descriptors looking for matches if file not excluded
    descriptorList.each {
      if (!(fileName =~ /$it.exclude/)) {
        if ((line =~ /$it.regex/).find()) {
          errors.add(it.description + " on line " + lineNumber)
          errCnt++
        }
      }
    }
    lineNumber++
  }
  if (errors)
    results.put((fileName), errors)
}

Generating the results report
The last part of the port was to create an html output report similar to a JUnit report. The Java code could have been better but it was simply a large set of write statements. I decided to use the HTML markup builder to generate the output report.

//*******************************************************
// generate the report
//*******************************************************
def writer = new FileWriter("$opt.r")
def html = new groovy.xml.MarkupBuilder(writer)
def rptDate = new Date()

html.html {
  head{
    title 'Validator Report'
  }
  body {
    h1 "File Validation - as of $rptDate"
    h2 "Validation Descriptors for file extension $opt.e"
    table ('border':0, 'cellpadding':5, 'cellspacing':2, 'width':'80%'){
      tr ('bgcolor':'#a6caf0') {
        th ("Description")
        th ("Validating Regular Expression")
      }
      //iterate thru descriptors - listing the regular expressions used in validation
      descriptorList.each { descriptor ->
        tr ('bgcolor':'#eeeee0') {
          td (descriptor.description)
          td (descriptor.regex)
        }
      }
    }
    // Summary table - just a file & error count
    h2 'Validation Summary'
    table ('border':0, 'cellpadding':5, 'cellspacing':2, 'width':'80%'){
      tr ('bgcolor':'#a6caf0') {
        th ("Files Processed")
        th ("Error Count")
      }
      tr ('bgcolor':'#eeeee0')  {
        td ("$files.size")
        td ("$errCnt")
      }
    }
    if (errCnt)
    {
      // Error details table
      h2 "Validation Details for extension $opt.e "
      table ('border':0, 'cellpadding':5, 'cellspacing':2, 'width':'100%'){
        tr ('bgcolor':'#a6caf0') {
          th ("File")
          th ("Errors")
          th ("Details")
        }
        //iterate thru results map key is filename and value is list of error text)
        results.each { k, v ->
            tr ('bgcolor':'#eeeee0')  {
               td (k)
               td (v.size())
               td {
                 ul {
                   //iterate thru the list of errors
                   v.each {
                     li (it)
                   }
                 }
               }
            }
        }
      }
    }
  }
}
writer.toString()

Porting Results
Initial Java Ant task: 3 classes and 326 lines of code
Groovy port: 1 script of 102 lines of code and 1 class with 9 lines of code. There are about 40 lines of processing code and the rest of the code/lines are from the html markup builder for the report.

Shorter and more concise, you decide!

Groovy script and class source
Here's the full Validator.groovy script

/**
 *  Validator script ported from Java Ant task
 */

//********************************************
// handle command line parms - all required
//********************************************
def cli = new CliBuilder(usage: 'groovy Validator -e extension -p propertyFile -d directory -r reportFile')

cli.h(longOpt: 'help', 'usage information')
cli.e(longOpt: 'extension', args: 1, required: true, 'file extension to be validated')
cli.p(longOpt: 'prop', args: 1, required: true, 'property file containing regular expresssion')
cli.d(longOpt: 'directory', args: 1, required: true, 'base directory to start file search')
cli.r(longOpt: 'reportFile', args: 1, required: true, 'output file for validation report')

def opt = cli.parse(args)
if (!opt) return
if (opt.h) cli.usage()

println "Processing $opt.e files \n\tusing $opt.p \n\tfrom directory $opt.d \n\tand generate report output to $opt.r"
println 'extracting files...'

//*******************************************
// get all the files for provided extension
//*******************************************
def files = new FileNameFinder().getFileNames(opt.d, "**/*.$opt.e")
println "processing $files.size files..."

//**************************************************************
// load properties into list of ValidatorDescriptor objects
//**************************************************************
Properties properties = new Properties();
try {
  properties.load(new FileInputStream(opt.p));
} catch (IOException e) {}

def config = new ConfigSlurper().parse(properties)
def descriptorList = []
config."$opt.e".each {
  descriptorList << new ValidatorDescriptor(it.value)
}

def errCnt = 0
def results = [:]

//**************************************
// process each file, line by line
//**************************************
files.each {fileName ->
  def errors = []
  def lines = new File(fileName).readLines()
  def lineNumber = 1
  lines.each {line ->
    // for each line - iterate thru descriptors looking for matches if file not excluded
    descriptorList.each {
      if (!(fileName =~ /$it.exclude/)) {
        if ((line =~ /$it.regex/).find()) {
          errors.add(it.description + " on line " + lineNumber)
          errCnt++
        }
      }
    }
    lineNumber++
  }
  if (errors)
    results.put((fileName), errors)
}

//*******************************************************
// generate the report
//*******************************************************
def writer = new FileWriter("$opt.r")
def html = new groovy.xml.MarkupBuilder(writer)
def rptDate = new Date()

html.html {
  head{
    title 'Validator Report'
  }
  body {
    h1 "File Validation - as of $rptDate"
    h2 "Validation Descriptors for file extension $opt.e"
    table ('border':0, 'cellpadding':5, 'cellspacing':2, 'width':'80%'){
      tr ('bgcolor':'#a6caf0') {
        th ("Description")
        th ("Validating Regular Expression")
      }
      //iterate thru descriptors - listing the regular expressions used in validation
      descriptorList.each { descriptor ->
        tr ('bgcolor':'#eeeee0') {
          td (descriptor.description)
          td (descriptor.regex)
        }
      }
    }
    // Summary table - just a file & error count
    h2 'Validation Summary'
    table ('border':0, 'cellpadding':5, 'cellspacing':2, 'width':'80%'){
      tr ('bgcolor':'#a6caf0') {
        th ("Files Processed")
        th ("Error Count")
      }
      tr ('bgcolor':'#eeeee0')  {
        td ("$files.size")
        td ("$errCnt")
      }
    }
    if (errCnt)
    {
      // Error details table
      h2 "Validation Details for extension $opt.e "
      table ('border':0, 'cellpadding':5, 'cellspacing':2, 'width':'100%'){
        tr ('bgcolor':'#a6caf0') {
          th ("File")
          th ("Errors")
          th ("Details")
        }
        //iterate thru results map key is filename and value is list of error text)
        results.each { k, v ->
            tr ('bgcolor':'#eeeee0')  {
               td (k)
               td (v.size())
               td {
                 ul {
                   //iterate thru the list of errors
                   v.each {
                     li (it)
                   }
                 }
               }
            }
        }
      }
    }
  }
}
writer.toString()

println "validation complete - possible error count = $errCnt"

//********************************
// for testing launch browser
//********************************
//"c:/program files/Internet Explorer/iexplore.exe $opt.r".execute()

Here's the one short class - ValidatorDescriptor.groovy

class ValidatorDescriptor {
    String description
    String exclude
    String regex

    String toString() {
       "regEx="+regex + " description=" + description + (exclude ? " exclude("+exclude+")" : "")
    }
}

And finally, my set of properties used for validation:

#################################################################
#
# SQL file regular expressions
#
#################################################################

#
# Find instances of trailing spaces after the semicolon.
# This causes MySQL problems.
#
#
sql.1.regex=;\\s+$
sql.1.description=Trailing spaces after semicolon
sql.1.exclude=SQLServer|Oracle

#
# Instances of lines starting with GO
#
sql.2.regex=^GO
sql.2.description=GO statement


#
# Find spaces before ending semicolon (but not DELIMITER ; which is used for MySQL Stored Procedures)
#
sql.4.regex=^(?!DELIMITER)\\s+;$
sql.4.description=Spaces before ending semicolon
sql.4.exclude=SQLServer|Oracle


#
# Oracle scripts using nvarchar
#
#
sql.5.regex=nvarchar
sql.5.description=Oracle scripts using nvarchar
sql.5.exclude=SQLServer|MySQL|Store Update

#
# MySQL scripts using ntext
#
#
sql.6.regex=\\sntext
sql.6.description=MySQL scripts using ntext data type
sql.6.exclude=SQLServer|Oracle

Summary
There you go - another itch successfully scratched! Maybe this task/script will be of help to others. The design was to make this a generic file validation utility and the SQL validation is the only use we have found so far. Enjoy!

Ant task ported to Groovy - Part 2

2009-12-21T23:44:00.004-06:00

Welcome back if you read part 1 of this posting. If you haven't read part 1, you may want to take a quick look at that before reading this part of the posting. This posting (part 2) will show the following changes or ports from Java to Groovy for the validator ant task.

CliBuilder for command line interface
ValidatorDescriptor class
Consuming the properties file

Switch from Ant to a Groovy task
As I mentioned in part 1, this port will not be an exact port because I am creating a groovy script rather than an Ant/Gant task - partly because I haven't played with Gant yet! Some of the existing Ant task code is shown below. For those familiar with creating Ant tasks, you will notice the class extends Task which is an abstract class. The code overrides the execute() method. The validator task checks the input and output parameters and then calls the validate() method to validate the files.

import java.io.File;
import java.io.FileFilter;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.regex.PatternSyntaxException;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

public class Validator extends Task {

    private static final String DEFAULT_EXCLUDE = "";
    
    // constants used to create the property file keys 
    // --> extension..regex or extension..description 
    private static final String REGEX       = ".regex";
    private static final String DESCRIPTION = ".description";
    private static final String EXCLUDE     = ".exclude";
    
    private String extension;
    private String directory;
    private String propertyFile;
    private String reportFile;
    
    private int fileCnt;
    private int errorCnt;
    private List descriptors;
    private Properties props;
    private Map errorMap;

    /**
     * Default constructor
     */
    public Validator() {
        props = new Properties();
        descriptors = new ArrayList();
        errorMap = new HashMap();
        errorCnt = 0;
        fileCnt = 0;
    }

    /**
     * Sets the extension for the task
     * @param extension extension of the files to be validated
     */
    public void setExtension(String extension) {
        this.extension = extension;
    }
    
    /**
     * Sets the root directory to be processed.  This directory
     * and all of its subdirectories will be processed.
     * @param directory directory to be processed.  
     */
    public void setDirectory(String directory) {
        this.directory = directory;
    }
    
    /**
     * Sets the root directory to be processed.  This directory
     * and all of its subdirectories will be processed.
     * @param directory directory to be processed.  
     */
    public void setPropertyFile(String fileName) {
        this.propertyFile = fileName;
    }

    /**
     * Set the report file to be created as output
     * @param reportFile name of the html file to be
     *   generated to show any errors
     */    
    public void setReportFile(String reportFile) {
  this.reportFile = reportFile;
 }
    
 /**
     * Creates an xml file from the bugs file
     */
    public void execute() throws BuildException {
        checkParameters();
        try {
            validate();
        } catch (BuildException e) {
            throw e;
        }
    }

    /**
     * Check that all required attributes have been set
     * @throws BuildException if any of the required attributes
     *      have not been set.
     */
    private void checkParameters() {
           if (extension == null || extension.length() == 0  
                ||   directory == null || directory.length() == 0 
                ||   reportFile == null || reportFile.length() == 0 
                || propertyFile == null || propertyFile.length() == 0) 
               throw new BuildException("Extension, directory, reportFile and property file attributes "
                      + "must be defined for task <" + getTaskName() + "/>" );
    }

    private void validate() {
        File dir = new File(directory);
        log("Processing file extension : "+extension+"\n");
        
        try {
            props.load(new FileInputStream(propertyFile));
                      
            setDescriptors();
            visitAllDirsAndFiles(dir);
            
            writeReport();
            
            log("Validator processed "+fileCnt+ " files and found "+errorCnt+" possible errors!");
            
        } catch (Exception e) {
            log(e.toString());
        }
    }
   private void writeReport() {
     try {
  FileWriter writer = new FileWriter(reportFile);
  writeHeaders(writer);
  writeContents(writer);
  writeTrailer(writer);
  writer.close();
 } catch (IOException e) {
  throw new BuildException(e);
 }
   }

   // some code omitted here - but shown later!!!
}

Groovy replacement using CliBuilder
While not a whole lot shorter than the Ant equivalent, it is short, concise and a good example of CliBuilder usage.

def cli = new CliBuilder (usage: 'groovy Validator -e extension -p propertyFile -d directory -r reportFile')

cli.h(longOpt: 'help', 'usage information')
cli.e(longOpt:'extension',  args:1, required:true, 'file extension to be validated')
cli.p(longOpt:'prop',       args:1, required:true, 'property file containing regular expresssion')
cli.d(longOpt:'directory',  args:1, required:true, 'base directory to start file search')
cli.r(longOpt:'reportFile', args:1, required:true, 'output file for validation report')

def opt = cli.parse(args)
if (!opt) return
if (opt.h) cli.usage()

println "start processing $opt.e files using $opt.p from $opt.d and output to $opt.r"

RegExDescriptor becomes ValidatorDescriptor
First the java...

import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;

/**
 * Regular Expression Descriptor used to help
 * validate source files.  
 *
 */
public class RegExDescriptor {

    private String description;
    private Pattern excludePattern;
    private Pattern pattern;

    public RegExDescriptor(String regExString, String description, String exclude) throws PatternSyntaxException {        this.description = description;
        pattern = Pattern.compile(regExString, Pattern.CASE_INSENSITIVE);
        if (exclude != null && exclude.length() > 0)
            excludePattern = Pattern.compile(exclude);
        else
            excludePattern = null;
    }

    public Matcher getMatcher(String line) {
        return pattern.matcher(line);
    }

    public String getDescription() {
        return description;
    }
    
    public boolean excludeFile(String name) {
        if (excludePattern == null) 
            return false;
        Matcher matcher = excludePattern.matcher(name);
        return matcher.find();
    }

    public String getPattern()  {
     return pattern.pattern();
    }
    
    public String toString() {
        StringBuffer sb = new StringBuffer();
        sb.append("Validator description=").append(description);
        sb.append(" regEx=").append(pattern.pattern());
        if (excludePattern != null)
            sb.append(" exclude regEx=").append(excludePattern.pattern());
        return sb.toString();
    }
}

Now the Groovy equivalent, which I renamed to ValidatorDescriptor. Like a lot of Java classes ported to Groovy, it is significantly shorter since we don't need to specify the getters and setters. The toString() method is really needed but I used it in testing to display the ValidatorDescriptor classes created from the properties file.

class ValidatorDescriptor {
    String description
    String exclude
    String regex

    String toString() {
       "regEx="+regex + " description=" + description + (exclude ? " exclude("+exclude+")" : "")
    }
}

Consuming the properties file
The next task is to read the properties file and create a list of objects representing each of the errors I am attempting to capture. If you remember from part 1 of the posting, there can be three properties: regex, description and exclude. The java processing to read the file and create the list of RegExDescriptors was located in two steps: the first was to read the file into a Properties object and then call the setDescriptors method in the Validator class as part of the main method called by Ant.

The idea is to group the properties together by the extension and the number that is part of the key. Each matching set is used to construct the RegExDescriptor. The RegExDescriptor creates a Pattern for the regular expression checking for the error and another for the file name exclusion check. The patterns are compiled at construction time.

props.load(new FileInputStream(propertyFile));
setDescriptors();

// later in Validator
/*
     * Setup the regular expression descriptors 
     * using the extension to determine which properties
     * are to be used when validating the files.
     */
    private void setDescriptors()
    {
        // strip the leading . and add one to end
        String prefix = extension;
        char c = extension.charAt(0);
        if (c == '.')
            prefix = prefix.substring(1);
        prefix = prefix + ".";
        
        int idx = 1;
        boolean done = false;
        
        // build list of validators using the extension to create the property keys
        while(!done) {
            String regexKey = prefix + idx + REGEX;
            if (props.containsKey(regexKey)) {
                String descriptionKey = prefix + idx + DESCRIPTION;
                String excludeKey = prefix + idx + EXCLUDE;
                try {
                    idx++;
                    String exclude = props.getProperty(excludeKey, DEFAULT_EXCLUDE);
                    descriptors.add( new RegExDescriptor(props.getProperty(regexKey),
                                                       props.getProperty(descriptionKey),
                                                       exclude) ); 
                } catch (PatternSyntaxException e) {
                    StringBuffer sb = new StringBuffer(100);
                    sb.append("Exception compiling regular expression key(").append(regexKey);
                    sb.append(") :").append(props.getProperty(regexKey)).append("\n").append(e);
                    sb.append("\nSkipping this expression...\n");
                    log(sb.toString());
                }
            } else {
                done = true;
            }
        }
    }

Now for the Groovy implementation. It starts out the same, loading a Properties object. Next, we use the ConfigSlurper class to group and parse the properties. An empty list to hold the descriptors is defined and then we iterate across the extension (sql in my case) keys and build the new ValidatorDescriptor objects. Notice the use of the GString aware string as part of the key for calling the each closure.

Properties properties = new Properties();
  try {
     properties.load(new FileInputStream('validator.properties'));
  } catch (IOException e) {}

  def config = new ConfigSlurper().parse(properties)
  def descriptorList = []
  config."$extension".each {
     descriptorList << new ValidatorDescriptor(it.value)
  }

In the next (and last) posting in this series, we will cover the following:

Replace the FileValidator class
Using the HTML Builder to generate a report
Complete listing of the script and classes

Java Ant task ported to Groovy - Part 1

2009-12-14T22:10:00.002-06:00

After reading one of Mr. Haki's recent posts, it reminded me of an Ant task that I had written a couple of years ago that we use in our nightly build. I started thinking about how much shorter and concise the code would be if written in Groovy. Rather than doing it all in one large posting, I have decided to break it out into 2 or 3 posts and will include all the final Groovy code in the last posting.

The Ant task: Validator
Our product supports several different databases and while can you generally code SQL to the 'lowest common denominator" for a lot of statements, we were encountering problems with our 'upgrade' scripts mostly due to syntax that was more lenient in Oracle and SQLServer but more restrictive in MySQL. Since the update scripts are not run every day/night, I came up with the idea of using some regular expressions to try and catch these errors as early in the cycle as possible, which is the scheduled nightly build.

Inputs & Outputs
The validator task takes three input parameters and one output parameters.

Inputs

extension - the extension for the file types to be validated
directory - the directory from which to start checking files
propertyFile- a properties file containing sets of properties for each validation to be performed

Outputs

reportFile - name of the output html file to be created with the results of the validation

Java implementation

The java implementation of this tasks consists of 3 classes. The first class, Validator, handles the Ant processing, loads the property file into a list of objects, RegExDescriptors, contains a main loop to visit all directories and files starting at the directory parameter and lastly, writes an output report. The second class, FileValidator, validates each file line-by-line and records any 'matches' against the regular expressions that represent an error in the SQL runtime syntax. Lastly, property settings are grouped together into a RegExDescriptor, that contains the description of the error being validated, the regular expression used to 'match' the error and an exclusion regular expression, which is used to exclude some files from some tests. For example, Oracle scripts using the 'nvarchar' data type would be an error so we only want to run that validation against Oracle scripts and not against the SQLServer or MySQL scripts.

Sample property file
This is a sample file containing some of the regular expressions currently being used. These are subject to change since, in some cases, backward slash had to be escaped in the pattern, like the last example sql.3.regex.

#################################################################
#
# SQL file regular expressions
#
#################################################################

#
# Instances of lines starting with GO
#
sql.1.regex=^GO
sql.1.description=GO statement

#
# Oracle scripts using nvarchar
#
sql.2.regex=nvarchar
sql.2.description=Oracle scripts using nvarchar
sql.2.exclude=SQLServer|MySQL|Store Update

#
# MySQL scripts using ntext
#
sql.3.regex=\\sntext
sql.3.description=MySQL scripts using ntext data type
sql.3.exclude=SQLServer|Oracle

Minor limitation

One minor limitation of this tool is that it only works on a single line at a time. SQL statements that are longer than a single line are not validated as one SQL statement but simply as lines of text. This has not been a problem as you can see from the sample validations listed above. The errors we try to catch are a mixture of syntax and runtime errors we have encountered.

Additionally, this will not be a 'exact' port, meaning, it will not be an Ant/Gant task, but merely a Groovy script, which could then easily be converted for use in those tools.

Port to Groovy
The first part of the port is easy - replacing all the code that recursively starts at a specified directory and creates a list of files matching the extension provided. First the Java version:

/*
     * Process all files and directories under dir
     */ 
    private void visitAllDirsAndFiles(File dir) {
        process(dir);
    
        if (dir.isDirectory()) {
            FileFilter fileFilter = new FileFilter() {
                public boolean accept(File file) {
                    return file.isDirectory() || file.getName().endsWith(extension);
                } };
            
            File[] children = dir.listFiles(fileFilter);
            for (int i=0; i<children.length; i++) {
                visitAllDirsAndFiles(children[i]);
            }
        }

Now the Groovy version

def sqlFiles = new FileNameFinder().getFileNames(dir', '**/*.sql')

Notice the Ant-like "**/*" prefix for the extension. Groovy does all the work and returns a list of files matching the criteria!

Upcoming

In Part 2

Using the CliBuilder for a command line interface (instead of Ant)
ValidatorDescriptor as a replacement for RegExDescriptor
Consuming the property file

In Part 3

Using the HTML Builder to generate a report
Complete listing of the script and classes
Replace the FileValidator class

Data binding for one-to-many association - part 2 (improved)

2009-10-06T22:48:00.005-05:00

Improving on the solution...
In a previous posting, I wrote a small sample of how to handle the data binding for a one-to-many association. While my approach worked, I had a couple nagging issues that I had to resolve. One, based on the code, the form field names and the way Groovy builds the map of request parameters, I had some questions about combining the bindData() method along with also doing objectInstance.properties = properties. Two, I had this nagging feeling that this could be handle better.

Issue 1
Based on the naming of the form fields that represent objects in the collection, these fields actually appear twice in the params map. Field addresses[0].street appears in the params map with a key=addresses[0].street and it also appears in a map within the params map with a key=addresses[0]. This 'sub-map' contains all of the fields of the Address object. My example used this 'sub-map' when calling the bindData() method. This all works fine. After completing the iteration across the addresses[x] sub-map entries within the params map, the last statement in the controller was personInstance.properties = params which then handles the binding for ALL the form fields, including the addresses[0].state field. This means that I was actually binding the association data twice! Using the debugger in Eclispe, I was easily able to verify this.

Below is the output of printing the contents of the params object after posting to the PersonController. The fifth line shows an example of the 'sub-maps' that I mention.

[
addresses[2].city:Arlington, 
lastName:Miller, 
addresses[0].city:Flower Mound, 
addresses[1].zip:66666, 
addresses[0]:[zip:75022, street:1 Main Street, city:Flower Mound], 
addresses[1]:[zip:66666, street:99 Palm Beach Circle, city:Palm Beach], 
addresses[0].zip:75022, 
addresses[2]:[zip:75066,street:1234 Cowboy Way, city:Arlington ], 
addresses[2].zip:75066, 
addrCount:3, 
addresses[0].street:1 Main Street, 
action:save, 
addresses[2].street:1234 Cowboy Way, 
controller:person,
addresses[1].city:Palm Beach,
firstName:Mike, 
addresses[1].street:99 Palm Beach Circle
]

Issue 2
That nagging feeling just wouldn't go away. I believed that there had to be a better way and I could swear I actually read about another option. This isn't really a Grails issue, but more a Spring data binding issue, so with the right set of search terms, my Google searches found what I was looking for (and had actually read before). Matt Fleming has written an initial posting and then a followup on this subject so you can read more there. Short answer: define the list as a LazyList which knows what data type to create when it actually does add elements to the list.

The Initial Problem
The initial problem encountered was a NullPointerException because the collection was not initialized with the correct number and type of entries. My sample solution handled that by creating the Person object first, which contains an empty list. Then it created each of the Address objects and added those to the list. Lastly, we did the binding using the whole params map.

Using Matt's solution, you can see the updated Person object below using the LazyList and defining the collection type to Address.

import org.apache.commons.collections.FactoryUtils
import org.apache.commons.collections.list.LazyList

class Person {

    static hasMany = [addresses:Address]
    
    static constraints = {
    }
    
    Person() {
        //addresses = []
    }
    
    String firstName
    String lastName
    List
addresses = LazyList.decorate(
                                  new ArrayList(),
                                  FactoryUtils.instantiateFactory(Address.class));
    
    String toString()  {
        firstName + " " + lastName + " Addresses:"+addresses
    }
}

The updated controller code is shown below. No reason to call the bindPerson(params), we just create the Person object with the params map and Grails & Spring do the rest!

def save = {
        //def personInstance = bindPerson(params)
        def personInstance = new Person(params)
        if(!personInstance.hasErrors() && personInstance.save()) {
            flash.message = "Person ${personInstance.id} created"
            redirect(action:show,id:personInstance.id)
        }
        else {
            render(view:'create',model:[personInstance:personInstance])
        }
    }

Client needs to help ensure 'guaranteed delivery' with JMS

2009-09-29T22:49:00.000-05:00

We recently received a new defect report claiming that some of our data was not being distributed correctly through JMS. Actually some of the data was not being distributed at all! That's the first time we have heard that complaint - hey, JMS is guaranteed delivery, right? Well not so fast...

Apparently we had a 'slow leak' in our connection pool within JBoss MQ. After a few days of uptime, the server seemed to stop distributing some of our data. The problem helped expose a deficiency in our JMS client code. We had not registered an ExceptionListener with our Connection object and as a result, we were not notified of problems creating a new Connection. I don't know about everyone else but most of the JMS sample client code that I have seen hasn't registered an ExceptionListener so that interface was news to me. Oh well, another learning experience!

So... JMS guaranteed delivery requires some assistance from the client as well, to help handle connection errors.

Using HTTPBuilder to call a RESTful web service

2009-09-24T22:59:00.004-05:00

Our team is delivering a new RESTful web service and so I figured that I would scratch my Grails itch and create a small application for testing the new web service. RESTClient looked like it would be a great match and it would be except that I want to display the XML request being passed to the service as well as the XML response returned from the service. Displaying the XML request is not a problem. The problem is that RESTClient automatically parses the response data and returns a GPathResult NodeChild object.

After reviewing the documentation for RESTClient and exchanging some emails with the RESTClient developer, he made the suggestion of using HTTPBuilder instead of RESTClient. The code shown below is the call to HTTPBuilder to call the new service. This topic came up in another posting and you can see an alternative approach here.

Points to notice

The second parameter to HTTPBuilder is setting the default content type to TEXT.
The request sets the Accept header to 'application/xml'.
HTTPBuilder.RequestConfigDelegate send method sets the content type to XML
Success handler gets response from 'reader.text' and then pretty-prints the result back in to a String for easy display.

def txnCreate(TransactionSpec txnSpec, String userid, String pswd) {

        def conf = Configuration.get(3)
        if (conf != null) {
           def ppos = new HTTPBuilder( conf.protocol+"//"+conf.server+":"+conf.port, TEXT )
           ppos.headers = [Accept:'application/xml', 'ppos-user':userid, 'ppos-pswd':pswd]
           
           try {
             ppos.request( PUT ) {
                   uri.path = conf.context + conf.url
                   send( XML ) {
                    TransactionSpec (training:txnSpec.training) {
                        delegate.transClassName txnSpec.transClassName
                        delegate.locationNumber txnSpec.locationNumber
                        delegate.currencyCode txnSpec.currencyCode
                        AssociateSpec {
                            delegate.associateNumber txnSpec.associateSpec.associateNumber 
                        }
                        delegate.terminalName txnSpec.terminalName
                        LineItemSpecs {
                            for (item in txnSpec.lineItemSpecs) {
                                LineItemSpec {
                                    sku(item.sku)
                                    quantity(item.quantity)
                                }
                            }
                        }
                        TenderSpecs {
                            for (tender in txnSpec.tenderSpecs) {
                                TenderSpec {
                                    tenderCode(tender.tenderCode)
                                    cardNumber(tender.cardNumber)
                                    expiryDate(tender.expiryDate)
                                    track1Data(tender.track1Data)
                                    track2Data(tender.track2Data)
                                    track3Data(tender.track3Data)
                                    preLockNumber(tender.preLockNumber)
                                    preLockAmount(tender.preLockAmount)
                                    amount(tender.amount)
                                    precision(tender.precision)
                                }
                            }
                        }
                     }   
                   }
                   // success handler
                   response.success = { resp, reader ->
                        // pretty print format the response
                        def stringWriter = new StringWriter()
                        def node = new XmlParser().parseText(reader.text);
                        new XmlNodePrinter(new PrintWriter(stringWriter)).print(node)
                        return stringWriter.toString()
                   }
                   // failure handler
                   response.failure = { resp ->
                      return 'Response status='+resp.status
                   }
             }
           } catch(Exception e) {
               log.error("Caught exception:", e)
               return e.toString()
           }
        }
    }

Blogger Minima template wastes space

2009-09-17T21:56:00.000-05:00

I just recently started blogging and so I am trying to figure out what looks good and how to get my blog to look better. I choose the Minima template but noticed that the 'gutters' seemed too wide down the left and right side of the page. Postings with code seemed to caused a lot of the line to wrap and it just didn't look very good. After seeing some other blogs that didn't appear to have that problem, I posted comments asking, "How'd you do that?". Someone suggested adjusting the template and so I took his advice. Below is a just a couple minor tweaks I made to the template and the code examples look much better and the gutters are not nearly so wide!

This all coming from a developer who has spent most of his career in the middle-tier and server-side of applications. For those of you very familiar with CSS, you can probably stop reading NOW.

Updating the Minima template to increase the posting area

Go to Layout --> Edit HTML page
Click the 'download template' link to make a backup copy of your template, for safe keeping
In the editable textarea, make the following changes to the style sheet entries:

outer-wrapper width change from 660px to 1000px,
main-wrapper width change from 410px to 700px
footer width change from 660px to 1000p

Click the 'preview' button to see the changes
If you are happy with the new spacings, click the 'save template' button.

Example Grails data binding for one-to-many association

2009-09-16T22:45:00.008-05:00

Data binding is an integral aspect of all web applications. The responsibility for the data binding generally falls within the scope of the controller. In this following example, I will show all the 'moving parts' that are needed to easily bind a domain object with a one-to-many association within Grails.

I looked around for a simple example of data binding to handle a one-to-many association and never really found a full example so I decided to create an example and share it. The example uses a Person domain object with a one-to-many association of Address objects. All of the data is captured on a single screen. The example uses DHTML or Dom scripting to dynamically create multiple address form fields and submits all the data from one screen.

Domain objects: Person & Address
The Person constructor contains just a first and last name. Notice the Address field implements Serializable and also implements the hashCode() and equals() methods as required for objects that are to be placed in a collection.

class Person {
    static hasMany = [addresses:Address]
    static constraints = {
    }
    
    Person() {
        addresses = []
    }
    
    String firstName
    String lastName
    List<Address> addresses
    
    String toString()  {
        firstName + " " + lastName + " Addresses: "+addresses
    }
    
}


class Address implements Serializable {
    static constraints = {
    }
    
    String street
    String city
    String zip
    
    String toString() {
        street + "\n" + city + ", " + zip
    }
    
    int hashCode() {
        street?.hashCode() + city?.hashCode() + zip?.hashCode()
    }
    
    boolean equals(o) {
        if (this.is(o)) return true
        if (!(o instanceof Address)) return false
        return street?.equals(o.street) && (city?.equals(o.city)) && (zip?.equals(o.zip))
    }    
}

Sample screen shot of create person screen

Below is a snapshot of a the create.gsp screen after clicking the 'Add Address' button twice and filling in the data. The 'Add Address' button calls a javascript function (see further below) to create an additional address row in the address table. (I wanted to provide the entire page .gsp source but could not get it formatted correctly - if anyone has suggestions, I would appreciate hearing from you.)

Add Address javascript function

Below is the javascript function called from the 'Add Address' button on the form. The important part here is that use of the hidden field, addrCount, to add an index to the address form field names and ids.

function addAddr(tblId)
{
  var tblBody = document.getElementById(tblId).tBodies[0];
  var newNode = tblBody.rows[0].cloneNode(true);
  var count = parseInt(document.getElementById("addrCount").value);
  count++;
  document.getElementById("addrCount").value = count;
  var cells = newNode.cells;
  for (var i=0; i
      if (cells[i].firstElementChild.id.indexOf("street") != -1) {
          cells[i].firstElementChild.id = "addresses["+(count-1)+"].street";
          cells[i].firstElementChild.name = "addresses["+(count-1)+"].street";
      } else if (cells[i].firstElementChild.id.indexOf("city") != -1) {
          cells[i].firstElementChild.id = "addresses["+(count-1)+"].city";
          cells[i].firstElementChild.name = "addresses["+(count-1)+"].city";
      } else if (cells[i].firstElementChild.id.indexOf("zip") != -1) {
          cells[i].firstElementChild.id = "addresses["+(count-1)+"].zip";
          cells[i].firstElementChild.name = "addresses["+(count-1)+"].zip";
      }
  }
  tblBody.appendChild(newNode);
}

Here is the initial statement for the hidden addrCount index field.

<g:hiddenField name="addrCount"   value="1" />

PersonController
When the form is submitted, the save action is called. I have created a helper method, bindPerson(), to handle the data binding. The method iterates 'addrCount' times, creating a new Address object, calling the bindData() method provided by Grails with the params starting with the value 'addresses[i]'. Lastly, the address is added to the collection and the Person object is returned.

The controller code below works party because my constructor for the Person object initializes the addresses field to an empty list. Other options might be to create a list in the controller and then set the addresses field to that list, or keep the initialized list and call personInstance.addToAddresses(addr).

class PersonController

  def save = {
        def personInstance = bindPerson(params)
        if(!personInstance.hasErrors() && personInstance.save()) {
            flash.message = "Person ${personInstance.id} created"
            redirect(action:show,id:personInstance.id)
        }
        else {
            render(view:'create',model:[personInstance:personInstance])
        }
    }
    
    def Person bindPerson(params)  {
        def personInstance = new Person()
        def count = params.addrCount.toInteger()
        for (int i=0; i<count; i++) {
            def addr = new Address()
            bindData(addr, params["addresses["+i+"]"])
            personInstance.addresses[i] = addr
        }
        personInstance.properties = params  
        return personInstance
    }

Results
Here you see the results on the show.gsp screen.

Wrap up:

First you need to setup your domain objects, including implementing Serialization, hashCode() and equals(), where required.
Form fields need to be name using index, like addresses[0].city or addresses[3].zip
Finally, the controller needs to bind the association data using the indexed fields.

Note: After writing this up, I finally realized why my Address object looked strange, it is missing a field for state. While important for a correct Address object, it isn't absolutely necessary for this example, so I decided not to go back and add it in.

Note: Please see an improved solution in Part 2

I hope this helps and as always, constructive criticism is appreciated! Due to the large number of requests, I have created a site over at Groovy sites and you should be able to download from here. Download site is https://sites.google.com/site/mikesgroovystuff/

Simple Groovy mistake had me wondering what the heck was going on!

2009-08-30T22:44:00.004-05:00

While 'scatching my programming itch' late one night last week, I made a small mistake which didn't present itself until runtime. When it did occur, it made we wonder what the heck was happening. After fixing my mistake, I just had to figure out what was really happening.

Minor background for readers new to Groovy:

Groovy provides language support to make the routine tasks on collection types easier than Java programmers are used to. For example, creating a map is as easy as:

def map = [key1:valueX, key2:valueY]

which creates a map with two entries: key1 maps to valueX and key2 maps to valueY. The fun starts with the many ways of accessing the maps data. The following are all valid ways of retrieving valueX from the map:

def value = map.get('key1')
def value = map['key1']
def value = map.key1

The first example looks just like Java. The second and third examples are provided by the Groovy language. Now, on to the mistake.

Can you guess what happens?
I have created a very small subset of the code to show the bug. The actual code resides in a Grails controller method that gets invoked on a form submission. In the submitted form, I use DOM scripting to dynamically create multiple line items for a transaction.

Take a look at the excerpt below and see if you can tell what happens when it executes.

def params = [itemCount:'3']  // this was really form data submitted
for (int i=0; i<params.itemCount; i++)
    def lineItem = new LineItem()
    bindData(lineItem, params["lineItem["+i+"]"])
    transactionInstance.lineItem[i] = lineItem
}

Anyone see the error?
The for loop looped about 10-12 times and then stop even though there were only 3 lineItems submitted on the form. Other times, the loop ran forever. The problem: the value of the itemCount entry is a String and not a Integer, which resulted in a strange comparison (more on that later). The fix was easy, convert the String to an Integer.

def count = params.itemCount.toInteger()
for (int i=0; i<count; i++)

But why did the loop behave the way it did?

Operator overload for < (less than)
The next step is to figure out what Groovy's doing. The operator < is basically doing a comparison, so what happens when you try either of the following:

i.compareTo(params.itemCount)
params.itemCount.compareTo(i)

The answer is, both throw a ClassCastException indicating that an Integer can not be cast to a String and vice versa. So how is Groovy coercing this comparison? Time to decompile the code to get to the bottom of this.

Decompile the code!
Below is a snippet of the decompiled code where you can see it calls the ScriptBytecodeAdapter compareLessThan method with both of the arguments as Objects, so we get an identity comparison!

  for(Integer i = $const$0; ScriptBytecodeAdapter.compareLessThan(i, acallsite[1].callGetProperty(params));)

Finally got a blog!

2009-08-28T20:55:00.000-05:00

Talk about being "late to the party", I have finally setup a blog. I plan on blogging on Java, Groovy & Grails and other topics that I would like to share while I "scratch my programming itch".

Hopefully some of my posts will be of interest to others and help them, as so many others have done for me. Please feel free to comment. Life's a learning experience!