Thoughts and Ideas

What is Computer Science?

2011-12-01T12:00:00.000-05:00

"Computer Science is the Science of Computing Machinery."

When I hear this I want to put a stick in the eye of the person who just said it. That's not a definition, it's a slogan. Yet, this is the definition used in most textbooks that even try to tackle the subject.

Let's sit down and try and break this apart. When do we cross the line from just being a machine and becoming a computing machine? Where is the distinction? Perhaps that will tell us something.

If I take a light bulb and plug it in and it turns on, that's not a computing machine.

If I take a light bulb and plug it in and it flickers... not still not computing since the flicker is random and has nothing to do with anything outside the light bulb.

If I take a light bulb and plug it in and it doesn't light. That's not computing, that's broken.

If I take a light bulb and it has a sensor so that when I plug it in, it doesn't light ... until I darken the room. In the dark it turns on, and in the light it turns off.

This machine has now crossed into the realm of computing. Albeit the simplest possible processing is happening: an outside stimulus is causing a response. We could call this a stimulus-response agent. Even so, this is now a computing machine and what is going inside this simple device is the very smallest beginning of something of interest.

Now, name a system that we can clearly define as a computer that does not exhibit some form of this behavior. Even a batch processing system that starts at a given time takes input or stimulus from outside itself, processes this inside itself, and then produces output.

What is Computer Science? We study something called "Computing" perhaps that should have been the name of the field? Computing is really what I've been working with my entire career and that's really what the degree is all about.

The Life Cycle of Plugins & Addons in the Enterprise

2011-11-17T11:59:00.002-05:00

I'm probably one of the few people in the world that has had the opportunity to work with both Grails and Spring Roo on extended projects. In both environments I've discovered a strategy around code reuse that is a bit different from what I normally observe in the Enterprises I consult with.

Both Spring Roo and Grails feature a reusable component system. They are similar enough you can use similar strategies in the enterprise with either Spring Roo or Grails. If you are in an enterprise that uses both Grails and Roo (these companies do in fact exist), I recommend that you develop common functionality in Spring POJO then graft a Grails plugin and/or a Spring Roo addon over the top.

In Grails, plugins are powerful sub-applications. A plugin can provide a command line script to the Grails CLI, a library, a library dependency, to a full set of Model View and Controllers or anywhere in between.

In Spring Roo, we term these reusable components addons because they do not necessarily contribute runtime functionality to the final system. Spring Roo's philosophy of having zero runtime footprint means that you don't have a Roo runtime to hang your hat on. What you do have is a very powerful Command Line Interface (CLI) that you can leverage along with a templating system inside your Roo addon to allow your end-user-developer to generate their own components. Instead of shipping function as you do in Grails, in Roo you provide a mechanism for guiding the developer through generating function.

Whether you are shipping function wholesale, as you do in Grails, or you are providing functional templates as you do in Roo, in an abstract sense I've discovered the same pattern applies to both Roo and Grails. You start with a pilot project for the addon or the plugin and you lift it out of the pilot project allowing it to graduate to a shared enterprise-wide repository.

As time goes by, you work on the addon or plugin as a separate project. Each project that uses the addon/plugin will keep track of its own versions of these addons meaning it is important to keep those old revisions available just in case. You will also want to tag each released revision in the event you need to back port fixes into old releases should a critical project get stuck on an old version of Roo or Grails for some reason.

The advantages to the enterprise would be a single place to revision control the client components of a Single Sign On system, or a single place to revision control and distribute shared client functions, look and feel components, or other commonly reused system components.

The Highly Monitored Grails Application @ SpringOne 2GX 2011

2011-11-01T15:58:00.003-04:00

My second talk at SpringOne 2GX was "The Highly Monitored Grails Application" and it had a great turn out from a large and lively audience. This was fantastic especially considering I was presenting as the very last presenter of the conference. This was one of the best received talks I've ever given.

The talk was demo heavy and so, while there's not much in the slides we spent a lot of time in my IDE, at the command line, and running through ways to monitor and work with live Grails applications. We even manipulated the hibernate settings in a running Grails application! Great fun!

Definitely the highlight of my SpringOne 2GX conference!

Performance Tuning Grails applications at SpringOne 2GX 2011

2011-10-27T20:31:00.005-04:00

I just got done with my SpringOne 2GX talk on Performance Tuning Grails Applications. When I proposed this talk six months ago there was no material on this subject. Since then a fantastic webinar came out. At the start of the presentation I mentioned this video which I would call the essential resource. And, you can watch it any time.

I deliberately cut out of my talk much of the subject matter there, why present the same information in a talk that you can download at any time? Wanting to present new and original material at SpringOne 2GX I rewrote my own talk from scratch. This time focused on the performance tuning of the server side code itself.

I decided to focus on real world performance metrics associated with Groovy Code itself. In the talk I demonstrated four implementations of a service three in Groovy and one in Java. The result is a bit surprising. Restructuring your Groovy code yields order-of-magnitude greater performance improvement than simply shedding Groovy in favor of Java.

The lesson to be learned is that the largest expenses your application will pay are in data marshalling, unmarshalling, network transmission time, and database querying. The Groovy itself can be restructured to be more performant without resorting to pure Java. Pure Java is there for us if we really need it but the majority of its benefit is not in that it is Java but in that it removes the most elegant features of functional programming and dynamic type systems from our pallet. I ask if the trade-off is really worth the small difference we see when simple Groovy code restructuring can give us five times better performance.

Slides:

Code:
https://github.com/hartsock/folksonomy

Speaking at Milwaukee Java User's Group

2011-09-20T23:10:00.002-04:00

I'm honored to be speaking at the Milwaukee Java User Group coming up on September 27th. If you are in the area, I'd be glad to see you and hear your input on the topic. I'm leaving my normal fare of talks on Groovy and Grails and speaking on the topic of cloud computing and Java. Many of my engagements revolve around tuning Java in cloud computing platforms for use in high volume websites. This talk is about my reflections on travelling from mere virtualization to three variations on cloud computing that I've used.

Hope to see you there.

Software Development versus Software Engineering

2011-08-17T12:01:00.003-04:00

One of the most useful things that I have gotten from studying things that I call "Demingesque " is the idea of Kanban. Kanban means "sign board" and is a way for projects to display information about themselves.

In its simplest form the Kanban idea is simply to keep track of things I need to do, things that I am doing, and things that I have done. In addition, because the board is placed in a location that is easy for management to see it can't help but communicate the status of the project or work area associated with the Kanban.

This idea is so simple it hardly seems like it is worth the effort of naming it. It is so engrained in the idea of a productive work team that I can't fathom needing to have a "methodology" associated with it. Yet I see amazingly complex ideas built up around this very simple tool.

Ironically, developers tend to like over complicated things in place of simple things. I suspect that it is because we are smart people and we don't like things that are too "simple" challenge us. Instead we want something hard that forces us to really stretch. So, I'd like to challenge people who think like that: don't think like that.

Instead, we should relish the idea of doing amazingly complex things with the simplest tools possible. In the places I've worked, the Kanban is most frequently implemented using a simple white sheet and a simple "sticky note" approach. I've seen many variations but ultimately you are just capturing the following information:

1. What do you need to do?
2. What are you doing?
3. What have you done?

Anything beyond this is probably a waste of time. This information recorded regularly can give you an idea of how long a project will take (based on the velocity of movement of items from "need to do" over to "have done") and how many "are doing" things are happening at the same time. The manager's job now becomes tracking this information and predicting that if fewer "are doing" things are happening then the "have done" pile grows more slowly.

You can formalize this as much as you feel like, put pretty math on it, or give it buzzword names, but really this is the basis for all software development project management... also sometimes known as "software engineering."

Washington DC Area Groovy User Group

2011-08-14T19:00:00.001-04:00

I'm honored to be presenting at the Washington DC Area Groovy User Group this Wednesday. If you are going to be in DC I would encourage you to come out.

good code, great code, and really great code

2011-08-10T12:00:00.013-04:00

There's great software, and there's really great software. The stuff that's great is the stuff that works well and is well documented. Ever need to figure out something? Go read the docs, if you screw up then the software tells you exactly how you messed up and didn't follow the documentation. That's great software.

The best software (the really great stuff) doesn't require documentation. You can play with it and discover how to make it do what you want. You just have to poke about with it a little bit and it will guide you to a working use case. You don't have to read a book, understand a theory (beyond basic computer sciencey things) and you can get to work almost immediately.

This is accomplished through really good failure modes and support tools that help you figure out how to use the system. It's 100 time the work of documentation but it is 100 times better. The best form of this is really great test coverage. You need to have really great test coverage to show you the use cases the system designers were thinking of. That is the best documentation because if a developer changes their minds the tests must reflect the change in paradigm.

Then there's stuff like this:

This is from the Sonatype Blog. On its own this is a tiny mistake. A blog has had the POM XML fragment stripped out of it. A bit of spelunking on the web produces this link as the definitive documentation for the plugin. Let's go and read the code for the plugin and see if we can't figure out what the missing tags are.

Over on github we find the emma4it plugin's source code. Diving this code we find that the project is actually pretty darn thin... and that's good because it is easy to understand. What does the JavaDoc say about how to configure this plugin? Thankfully there is at least documentation telling us what each of the Java files are for in the Maven lifecycle.

Does any of this include an example like the one Sonatype accidentally destroyed the only canonical example of in their blog? Sadly, no.

So my best guess is the first "report" word shown is a "report" id for the execution since we have more than one execution. My next guess is the phase we use in the documentation from the blog is different from the phase shown in the JavaDoc. The phase here is the "post-integration-test" phase by my guess. Next I have to take a wild leap. My guess is "project.build.sourcesDirectory" is the value set in the sourcesDirectory tag.

This is my best guess at the missing POM fragment:

<execution>
<id>report</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<sourcesDirectory>
${project.build.sourcesDirectory}
</sourcesDirectory>
</configuration>
</execution>

So by my estimation this is good code, that is on the cusp of being great code. That is still a ways from being really great code which wouldn't require you to dig around for documentation to begin with.

The seven stages of dealing with PMD rule violation

2011-08-01T21:09:00.007-04:00

Is your build broken from a PMD rule violation? Has your project has been set up so that a PMD rule violation breaks the build? On occasion even the best developers slip and push code that is in violation of the PMD rules they themselves agreed to.

I have been observing developers under these severe conditions for some time now and I've identified seven emotional stages that every developer who breaks the build due to PMD rule violations goes through. I am cataloging these here in the hope that it helps some developer cope with their broken build and move on in their project.

Shock and denial

You will probably react to learning that your code does not conform to your project's PMD rules with numbed disbelief. You may deny the reality of this PMD rule violation at some level in order to avoid the pain. Shock provides an emotional protection from being overwhelmed all at once.
Pain and Guilt

As the shock wears off it is replaced with suffering. Although it is excruciating, it is important that you experience this pain fully and not avoid it or escape it with //NOPMD comments or by the use of alcohol or drugs.

You may have guilty feelings or remorse for the things you did or didn't do with your code. Life feels scary and chaotic during this phase.
Anger and Bargaining

Frustration gives way to anger. You lash out and lay unwarranted blame for the rule violation on someone else. You may blame PMD itself. You may blame the authors of the rules. You may rail against fate questioning "Why me?" and you may also try to bargain in vain with PMD saying things like: "I will never drink while coding again if you will just accept my patch"
Depression, reflection, loneliness

Just when other developers think you should be getting on with your project, a long period of sad reflection will likely overtake you. This is a normal stage of resolving breaking a build due to a PMD rules violation.

During this time you will finally realize the full magnitude of your rule violation. You may isolate yourself from other developers on purpose to reflect on things you did in the code-base and focus on your memories of past times before you violated the PMD rule.
The upward turn

As you start to adjust to life after a PMD rule violation has occurred, you become calmer, your thoughts organize. Your physical symptoms (these range from mild stomach cramps and headaches to full blown cold-sweats and involuntary spasms) lessen your depression lifts and you begin to formulate a plan.
Reconstruction and working through.

As your mind becomes more functional, you find yourself seeking realistic code changes to be made to resolve the PMD rule violation. You start to work on restructuring your code around the PMD rule violation.
Acceptance and hope

During this last stage you learn to accept and deal with the reality of using PMD on your project. Acceptance of PMD does not necessarily mean happiness. Given the pain and turmoil you have experienced you may never return to a care-free and untroubled code base that existed before you broke the build. But, you will find a way forward in your project.

You will start to look positively on your project again as a whole and actually plan things for future releases. You will once again anticipate writing good code and may even find joy in writing software again.

Rest assured, you can recover from a PMD rule violation. You can put your project back on track. There will be joy in coding again someday.

The one about scope. And tacos.

2011-07-24T18:34:00.011-04:00

Consider the following paragraph.

Mary and Sally are friends. Mary gives Sally a gift for her birthday. Mary hangs up the telephone.

Are there any syntax problems in the above paragraph? Are there any grammar problems? What is wrong with this paragraph exactly? The problem is non-technical in nature. The paragraph references a relationship between Mary and Sally and then describes an interaction that occurs between them. Finally it describes Mary ending a phone conversation. However, we have no idea when or how Mary began talking on the phone. We don't even have a prior reference to a phone conversation.

Consider the following code:

def exchangeIfRelated(def mary, def sally) {
if( canRelate(mary,sally) ) {
mary.exchange(sally)
}
mary.closeConnection()
}

What is wrong here? The isn't anything technically wrong provided whatever connection we just closed was not in use after this method finishes. What is the likely hood this call is made from a different part of the program like this?

public void friends(FooService mary, BarService sally) {
   mary.openConnection()
   sally.openConnection()
   exchangeIfRelated(mary,sally)
   sally.closeConnection()
}

Anything wrong here? No. Not really. It will compile, it will run. In this method we have implicitly taken responsibility for opening and closing whatever these connection things are. In a more concrete example, weight be opening and closing a database connection. In that kind of example we are allowing concerns to bleed between methods. Ideally, I want one unit of code to be concerned with one thing. That thing (whatever it is) is hopefully easily abstracted for use by other units of code without their having to become experts in that other field. The following is better and will likely lead to fewer misunderstandings.


public void friends(FooService mary, BarService sally) {
   mary.openConnection()
   sally.openConnection()
   exchangeIfRelated(mary,sally) 
   mary.closeConnection()
   sally.closeConnection() 
}

Knowing how an ignition system works does not aid my ability to drive a car in rush hour traffic. When I start my car I do not need to have knowledge of what happens when in my car's engine, nor do I need to know how a camshaft functions in order to drive my car down the road. The driver is insulated from these automotive inner workings. In the same way when we build software we want to insulate the layer above us from details about the layer beneath us.

Mixing the scope of when and where you do things is like changing the subject of a paragraph. There are fish in the sea. Nothing is really wrong with it from the stand point of the compiler which really only understands the mechanics of phrases in the programming language. There are tacos in a truck. However, just as properly formed sentences that have little to do with the core thought of a paragraph are disruptive and confusing. I like kites. Similarly, such disruptive code phrases in our code leads to confusion and in a program that kind of thing leads to bugs. And tacos.

GR8Conf

2011-03-23T08:42:00.006-04:00

I'm very pleased to be speaking at GR8Conf this year. I'll be covering two topics. I hope the Groovy communities find them interesting.

The first up, how to use Domain Specific Languages in order to reduce accidental complexity. I have a very long post that I'm holding on to based on this topic (inspired by my trip to Manhattan and Google NYC earlier this year). In this talk I'll really take time to explore this idea practically. I will create a Domain Specific Language before your eyes as I do live coding with Test Driven Development. It's performance art with Unit Tests!

The second topic is on Grails and the JPA. I'm currently working on a lab for this topic. We'll explore some of the nastier Object Relational Mapping problems I've encountered working with Grails and with JPA in isolation and together. If time permits I'll work in some performance troubleshooting in this lab too. In this format we'll have the chance to work together and really get a deep understanding of what's going on with Grails persistence and how you can use GORM and/or the JPA in places you may have thought were impossible. I really enjoy the lab format and I hope to see a good turnout.

For me, getting the opportunity to speak at the GR8Conf is a major milestone and I'm very pleased that Open Software Integrators is allowing me to participate. The fine folks at Open Software Integrators have an experienced consulting staff that can help you with all things JVM related. I've had a great time working with the other consultants here over the last few months learning a great deal about performance tuning and scalability.

So please check me out at GR8Conf and I look forward to seeing you there!

Groovy/Grails: How to disable a controller

2011-02-16T08:00:00.000-05:00

Let's say you have a plugin. That plugin includes a controller. You don't want to use the controller provided by the plugin (for whatever reason) but instead you want use your own controller.

What do you do?

Filters my friend. Filters.

Next to your Config.groovy file in the grails-app/conf directory create for your self a file called DisableFilters.groovy and then add a class definition. Here's a filter definition that disables everything in the application... (making your grails application headless).

class DisableFilters {

def filters = {

all(controller:'*', action:'*') {

before = {

// return false

}

after = {

}

afterView = {

}

Now you'll notice there is a before closure and it returns false. This closure fires before the request is passed to its destined controller/action. The * means match all. So if you had a specific controller name or specific controller name pattern you wanted to disable you could easily disable all these by matching the name here.

Notice the nifty after which would execute after the controller and action complete. Finally you have the afterView closure which would execute after the entire call chain was done and the browser has left. I'll leave the uses for these up to your imagination.

In the filter body you have access to everything you have in a grails controller so you have a great deal of flexibility here with what you can do. You could look at usernames, session data, whatever your heart desires. You can even redirect the request to another URL using the same conventions as you would have in a controller. So have at it!

double entendre

2011-02-09T08:00:00.006-05:00

I have said "The Future of Java is Groovy" and I meant it deliberately as a double entendre. The future of Java is indeed groovy and is indeed Groovy. This does not mean Java will be supplanted by Groovy, or some other language. Instead, the future of the JVM and the technology stack that calls the JVM home is tied more to technological off-shoots like Groovy than it is to any other purist re-engineering of the JVM or the core Java language.

Java is what it is at this point. Tools like Groovy give developers the chance to get a gentle introduction to new techniques such as Functional Programming, Parallel Computing, Domain Specific Langauges, Runtime Metaprogramming, and Compiletime Metaprogramming, as well as a whole host of other techniques and technologies.

The beauty of Groovy as a programming language is how accessible it makes all these advanced techniques to a beginner and how readily it will get out of the way of an expert. This is a beautiful balance of the elegant and the ugly that few other languages manage to pull off well.

Do I think Groovy will supplant Java on the JVM? No, not really. Do I think any other language will make that claim? I seriously doubt it. Do I think Java needs a successor? No. No I do not.

The closest ecosystems that we can draw analogies to for the modern JVM are the .Net ecosystem and the C/C++ ecosystem. The C/C++ ecosystem is probably a better environment to draw lessons from since C/C++ long ago shed its lock-in with a single vendor (that being AT&T long long ago).

I think Java is growing up the same way the C/C++ ecosystem did and C hasn't really changed much over the years. I think Java needs to become the same kind of solid foundation. If other languages come and go on the JVM that needs to be groovy with everyone.

I don't steer any ships in the Groovy navy, I don't direct any plans or strategies, so what I say here is just what I say, and what I say is:

Groovy allows rapid access to advanced computing concepts with a minimum extraneous cognitive load.

This is the Groovy language's strength. It is immediately accessible to anyone with any programming background. This accessibility is the vital engine for the success we are seeing in Groovy adoption. It is also the weakness of the language. It means that Groovy must necessarily avoid the traps of becoming too "beautiful" of a language forcing people into certain forms steepening its learning curve too sharply.

That's my opinion anyway. I happen to be one those kinds of guys who likes what I can do with a language. I tend to not get hung up on whether it is pretty, or if it is pure. It's all groovy man.

A late New Year's post.

2011-02-02T12:44:00.004-05:00

tl;dr I am a consultant now.

It's a quite late, but I did make some new year's resolutions. One was a personal fitness resolution to run a mile every workday. The other was a professional resolution to "get out there" more and contribute more to the Groovy/Grails community. For most of 2010 I felt as if I were under a rock and not able to contribute to the community more and so I've made a couple of drastic moves.

My first drastic move was to step-up my fitness routine. I have been working out 3 times a week now for a few years. This year I have stepped it up to 5 times a week. Yes. 5 times a week. My warm up is a 1 mile run.

So far this year I've managed to run one mile every weekday (I give myself Saturday and Sunday off from running). So far, I haven't missed a day. I was inspired by my son's devotion to his Cross Country training and I have set myself some small goals for my running to try and achieve over the next few years. My big goal is to eventually run a Marathon. This seems utterly absurd to say, I have so very far to go before I can do something like that. Barring any medical reasons I fully intend to achieve that goal.

My results this last month are encouraging. I started the year with a mile time of 15:44 (fifteen minutes and forty-four seconds) and I've managed to bring this down to 11:24 or so on my best day. I typically complete the mile now between 11:30 and 12:30 now. My goal is to make a 12 minute mile an easy run for me. If I can do that then I will begin increasing my mileage over time.

The other one of those drastic moves was quitting my job at a company I dearly loved working at and really respected. It was a very hard decision to make but I decided that I wanted to invest more directly in the Open Source community and open source efforts. I believe strongly in the cause of Open Software and if I believe in it I should put my whole being behind what I say. So I've joined forces with Open Software Integrators which is a young Open Software consultancy based here in the Research Triangle area of North Carolina. We have a strong match for vision and attitude toward software development and I look forward to working with the team here to help companies reap benefits from Open Source software and toward making solid contributions back to the Open Source community.

There have been a lot of preparations going on in making these transitions so I've not been as active here in this blog as I intended. I have made a resolution to also post more regularly here. It was this blog that enabled me to get published in GroovyMag, create the plugins that have taught me so much about living with open source software projects, and introduced me to so many of you.

I hope that I can find a way in my new role to make a real positive impact on the open source community and fill a need that is currently under served. As always I'm open to your feedback and I find your comments valuable. I am currently looking for conferences that would be open to me speaking so your pointers will be welcome. Obviously, I can't attend them all but I will make a real effort this year.

Thank you for reading and I hope I get meet some of you in person this year too!

On the topic of Grails Audit Logging Plugin version 0.5.4

2010-10-14T08:55:00.006-04:00

Last night, I released a new version of my Audit Logging Plugin. Release notes are here. There had been an outstanding compatibility issue with Grails 1.3.x based on JUnit 4's Integration tests. The bug documented here has been open for several months as I haven't had time to come back to the project to work on it.

Normally I don't blog about tickets but I figure this one requires some attention because I've run into a few plugins that I need in my day job that are "broken" in 1.3.x and I'm not sure who else is having these troubles. In the interest of helping the community as a whole I want to cover my experience.

It turns out that GRAILSPLUGINS-2243 (titled "Integration tests with auditable domain class do not observe test transaction boundary") is actually a more accurate description of the actual bug. The symptom is during integration testing you can't delete entities that are audited. If you take the same code and run it in development or production mode you can delete the objects.

The issue with my code is in this block... (extraneous logging and debugging code removed)

Session session = sessionFactory.openSession()
def trans = session.beginTransaction()
def saved = session.merge(auditLogObject)
session.flush()
trans.commit()
session.close()

Keep in mind this block functions exactly as expected in Grails 1.2 under all environments I know of. The one environment this code is horribly wrong is Grails 1.3.x in Integration testing. Only at that time does this code cause a problem.

So let me ask you... what's wrong with this code?

Before addressing the real issue, questions I anticipate:

Why are you opening your own session?
Because on a transactional database in some cases the audit log must fire after the database insert so that I can log the automatically assigned object Id. This means you can't use the open session without invalidating it.

Why are you using merge and not save?
Because there are edge cases where the AuditLog object will be saved twice and I don't want to dictate the logic for that in this block to keep it simple

Why are you flushing the session?
Because we want to make sure the audit log is being written outside the regular transaction cycle and this particular session is going to perform no other job we want to ensure the log is written out to the database

So ... what is wrong with this code again?


def trans = session.beginTransaction()

Apparently, under integration testing in Grails 1.3.x if you start a transaction in the way I have you will cause a problem with the other independent transactions in the rest of the application. This is simply not the behavior I see in my production systems under Grails 1.2 or in my staging systems running Grails 1.3 for that matter.

My solution is a bit of a work around. In the application context initialization for the Audit Logging plugin (the bit that runs just as you start up your grails application I have a block that does this...


if( Environment.getCurrent() != Environment.PRODUCTION && ConfigurationHolder.config.auditLog?.transactional == null ) {
       transactional = false
}

... which simply says in environments that are not production (Notice I'm using the Grails 1.1 API to find that breaking compatibility with Grails 1.0) the boolean flag "transactional" is set to false. I then wrapped the begin and commit calls to the transaction around a check on this boolean. For good measure, I gave the end developer the ability to override this behavior using the setting "auditLog.transactional" in their Config.groovy file. (It would figure that I would release this as a feature and someone needs to use those transactions in their integration testing because they run their testing environment against Oracle or something so here's an out for you to get the old behaviors)

What took so much time to find this bug? It's not clear cut basically. I stumbled on this answer by blindly trying different things until I found a combination that worked. Then I tried to build a theory around this result that fit the facts that I had on hand.

My theory is the JUnit 4 integration environment does something to the session factory so it does not really produce new sessions such that the session that I get in my plugin's listener code is not really a separate session or transaction. I call commit on this transaction and when the session is returned to the normal GORM code it is already committed so the actual entity delete call can't happen and is swallowed.

What is disturbing about this is I didn't invent this technique for Audit Logging I'm implementing here. I created this code after being educated by the Hibernate wiki. This means if my theory holds it's not just a Grails application that will observe this strange-ness ... pretty much any Hibernate application with this kind of Audit Logging will be affected when integration tested under JUnit 4.

Beyond just this anyone developing applications that open their own sessions independent of the GORM session may potentially see bugs during integration testing that do not manifest in other environments. That can be a very difficult situation to deal with. So I've over documented things here for the benefit of the community at large.

In other news...

In other places I've noticed a change in Groovy 1.6 and 1.7's builder API. In particular I use RichUI in one project which makes calls like...


builder.yield(" var ${attrs.yuiVariableName}DataSource = new YAHOO.util.XHRDataSource(\"${attrs?.action}\");\n", false)

... under Groovy 1.6 yeild was a builder method.

But, under Groovy 1.7 this method is not present. That means the HTML rendered by code that used the yield and yieldUnescaped calls now create tags <yield> and <yieldUnescaped> which breaks some pages (but not all).

So far these are the only surprising issues I've run into that has kept Grails 1.3.x off of my production servers. I haven't yet seen and great benefit to 1.3 so I can't say I'm deeply compelled to move to the new version.

I am looking to rewrite the plugin in light of GRAILS-5725 but this will be a breaking change. Frankly, it's such a large change I'm surprised it was not pushed to Grails 1.4 ... however, it does not appear to interfere with the operation of plugins using the listener injection technique I am also using so no harm no foul.

I see I've lost a whole star in my plugin ratings so, my apologies to my end users who have not had a plugin that works with your Grails 1.3 integration tests. I sincerely hope I can earn back that extra star from you. Thanks for your support and help. I am reviewing code submissions for another version of the plugin coming up and I greatly appreciate your support.

TriDroid: QRCode, Linkulator & other "move the data" ideas

2010-09-30T22:27:00.008-04:00

At tonight's TriDroid we discussed QRCodes and other techniques for moving data between the environment and our Android phones. Here are hastily-hand-drawn my slides for anyone interested in them.

Other things we talked about:

Chess Data format based on Algebraic Chess Notation

3D Barcodes for manufacturing

4D barcodes (time phased codes for higher data density)

color barcodes where color is used as a new "dimension" to store data in.

EDIT:

As George points out (and many have pointed this out repeatedly) the Google Chart API has had QRCodes incorporated in it for some time. Once again, being lazy would have paid off much more than working hard. A lesson the universe seems to be trying to teach me repeatedly.

App Inventor

2010-07-14T08:51:00.004-04:00

If you haven't heard about it App Inventor is available to educators wanting to teach students about programming. It allows novice programmers to create Android applications using a graphical programming environment similar to Scratch.

The educational perspective that motivates App Inventor holds that programming can be a vehicle for engaging powerful ideas through active learning. As such, it is part of an ongoing movement in computers and education that began with the work of Seymour Papert and the MIT Logo Group in the 1960s.

This is probably one of the more interesting and exciting ways to engage lay-programmers in software creation. It is possible to view the graphical programming language blocks as a Domain Specific Language but I'm not clear on whether this is actually the case or not. In particular:

the types of statements are limited to available blocks

these statement blocks are limited to guided classifications specific to phone app development

user and device interactions are limited to the palette

In these ways the "developer" is "on-rails" (in the video game sense) limited to a certain type of application. This kind of limitation is beneficial in some cases guiding the developer toward a "good" application. In this case the limitation keeps the developers from heading down paths that require knowledge about the halting problem or multi-threading. For this class of application... that's a good thing.

The OpenBlock abstract certainly sounds like a description of a Domain Specific Language (emphasis added):

Graphical programming systems have been built to lower the threshold to programming for beginners. However, because these systems were designed to make programming more accessible to novices, they were developed with narrower intentions for their users and applications. For example, in StarLogo TNG, a graphical block programming environment, users may create games and simulations, but they cannot use this same system to create programs that can automate their computer processes, like the text-based scripting system AppleScript. Application developers can create their own programming systems, but doing so can take a significant amount of time to design and implement. This thesis describes an extendable framework called OpenBlocks that enables application developers to build and iterate their own graphical block programming systems by specifying a single XML file. Application developers can focus more on the design of their systems instead of oil the details of implementation. The design and implementation of OpenBlocks are described, along with a user study conducted to test its usability and extendability.

On the other hand it could be argued that the graphical programming environment is just an immature environment and is only limited because of how new the tool set is. There is no intrinsic reason a graphic block based programming language couldn't eventually become as rich as a traditional text based language. The real differentiator between a (DSL) Domain Specific Language and a GPL (General Purpose Language) is that the scope of the programming done.

You evolve a DSL either by creating a very rich API that "begins to talk" or you create a deliberately limited subset of your GPL that can be exposed and interpreted using limited API.

So is App Inventor using a new(-ish) Domain Specific Language or a new way to wrap a General Purpose Language?

Three Small Ideas

2010-07-09T11:10:00.008-04:00

I was glad to be interviewed by DZone a while ago. The second interview was on my current Open Source work and I gave what is probably a non-traditional answer on why Open Source contribution is important for a developer. The google appengine project I'm talking about is the qrcode linkulator. And, for the record, I've got a better hair-cut now.

My statistics are from this article on Functional Fixedness at wikipedia. The over-lapping concept I'm trying to tease out is that there are problem solving exercises that are not served by tightly focused environments like business. As a developer you need to be able to work in spaces that allow for creative freedom and the lessons you learn there (for better or worse) translate back into your "high pressure" environment. The result is a better developer that is able to think outside the parameters of the problem.

A nice animation covering the idea popularized by Daniel Pink (who is an infinitely better speaker) is on youtube:

My contribution is: not only is open source a product of intrinsic motivation... it creates intrinsic motivation to improve yourself as a developer. These improvements translate back to your "day job" ... so for purely selfish reasons you *should* create open source code. You will be forced to learn. You'll get interaction with other developers. You'll learn much more than if you stay locked in your office and never get challenged by people who never would think like you do and have no reason to pretend they do.

Even if the project is a failure and your contributions are insignificant... you'll learn something.

Grails + Acegi + CAS + Proxy Ticket

2010-06-29T11:46:00.006-04:00

Should you have Acegi + CAS setup in Grails and you want to proxy a URL secured behind the same CAS instance you should do something like this:


    URL url = new URL(urlString)
    SecurityContext ctx = SecurityContextHolder.getContext();
    CasAuthenticationToken auth = ctx?.getAuthentication();
    Assertion assertion = auth?.getAssertion();
    Principal principal = assertion?.getPrincipal();
    assert principal.proxyRetriever != null 
   // the above line asserts that proxy system is properly setup and working
    String proxyTicket = principal?.getProxyTicketFor(url?.toString());
    assert proxyTicket != null
    URL readUrl = new URL(url.toString() + "?ticket=${proxyTicket}")

This only works if your CAS supports proxying and you have a working proxy call back. For details on troubleshooting this with Grails-Acegi 0.5.3 see GRAILSPLUGINS-2231 which includes a working proxy callback controller. Hopefully this will be fixed in an upcoming version so you won't need this reference for long.

If your setup is not working properly the proxyRetriever embedded in the Principal object will be null. If it is null then getProxyTicketFor returns null no matter what. Once you get your CAS and SecurityConfig.groovy working properly the proxyRetriever starts getting magically injected into your application. This doesn't mean you actually have proxying working yet but you can almost rule out client-side configuration errors.

Your Grails application should also have something at /secure/receptor that can catch responses issued by the CAS server. You should see this URI respond to something just after you successfully log in. That URI also needs to be open so the CAS server can post there without needing to login to itself.

NOTE: your CAS server must be able to resolve the URL of the proxy call back. That means if you are running your proxy callback on localhost the responding CAS server must be able to post to that URL. Why? When you request a proxy granting ticket CAS will send the ticket back to your Grails application at the URI /secure/receptor by default. That means your application's URL must be resolvable by the CAS server for proxy tickets to function.

In practice, I run a small CAS server on localhost at an obscure port with the URL to proxy also on localhost when testing proxy ticket granting and other CAS secured proxy services. I work around the SSL issues by using a custom implementation of javax.net.ssl.HostnameVerifier that is installed during bootstrap in development and test-app modes only. The custom verifier (written in Java) looks like:


import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
import javax.net.ssl.HttpsURLConnection;

public class CustomHostnameVerifier implements HostnameVerifier {
    public String[] exemptNames = {"localhost","testing.com","example.com"}; 
   // whatever hosts need exempting from having a "real" SSL certificate
    public boolean verify(String hostname, SSLSession session) {
        boolean exempt = false;
        if(exemptNames != null) {
            for(String name:exemptNames) {
                if(hostname.toLowerCase().endsWith(name)) {
                    exempt = true;
                }
            }
        }
        return exempt;
    }

    // call from BootStrap.groovy or other opportune initializer
    public static void init() {
        HttpsURLConnection.setDefaultHostnameVerifier(new CustomHostnameVerifier());
    }

}

Other work arounds would involve a staging CAS server that was able to resolve the development or testing host during work on proxying. These are up to you.

When NOT to use a Domain Specific Language

2010-06-24T06:53:00.005-04:00

There's really one rule I can use to sum up why you would not use a domain specific language.

Do not use a Domain Specific Language when implementation specifics out weigh domain specifics.

This one rule covers numerous cases:

when intimate knowledge of API are as important as function

when performance is critical and requires expert knowledge

when implementation details are nearly as important as semantic details

All these cases and numerous variations on this theme are summed up by acknowledging that the choice of using a Domain Specific Language (DSL) is a design trade off. You are deliberately adding a layer to your application stack. This layer intrinsically incurs a maintenance and/or performance cost. (Not all DSL cost additional clock cycles at run time it depends on your implementation.)

With these costs and benefits in mind you can make an intelligent choice about when and where to use this tool. It is far from a panacea but Domain Specific Languages do have the allure of potentially netting you accidental programmers and preserving folk knowledge about your system and its real world uses that few other techniques offer.

Never edit icky JBoss XML files again!

2010-04-16T09:39:00.004-04:00

Do you have a JBoss application server? Do you have configuration files in that server that need to be edited or changed periodically? Do you have an allergy to XML?

Have I got a solution for you!

JBoss has the amazing ability to load *.properties files from any location (on the file system or on the internet) as part of its start up routine. So you can set properties in a simple properties file instead of an icky XML file.

NOTE: I'll refer to the directory you have installed JBoss into as $JBOSS_HOME you may have your JBoss server installed in /opt/jboss or /usr/local/jboss for example. It doesn't matter.

NOTE: I'll assume your JBoss is set up to load the "default" server directory. If it isn't then I think you are smart enough to figure out what to do on your own.

Stop the JBoss server

open the file $JBOSS_HOME/server/default/deploy/properties-service.xml in your favorite text editor.

Find the tag

<mbean code="org.jboss.varia.property.SystemPropertiesService" 
  name="jboss:type=Service,name=SystemProperties">

Add this tag:

<attribute name="URLList">
      /etc/jboss/system.properties
</attribute>

Comment out or delete any other instances of the <attribute name="URLList"> ... </attribute> tag

Great! Now when we start up the JBoss server again it will read the file /etc/jboss/system.properties and set its properties from there. But, wait... the file doesn't exist yet. If I start my JBoss server and that file isn't easily readable it will crash.

So let's create the directory...

$ sudo mkdir -p /etc/jboss

... and the file ...

$ sudo touch /etc/jboss/system.properties

... now we can restart JBoss safely.

We don't get much benefit yet though. Nothing in JBoss is using that file. So let's take a database-ds.xml file and configure it to use system.properties for its connector-url, user-name, and password.

Assuming you have a file like $JBOSS_HOME/server/default/deploy/database-ds.xml you can edit it to look like:


<?xml version="1.0" encoding="UTF-8"?>
<datasources>
   <local-tx-datasource>
      <jndi-name>MyDatabaseDS</jndi-name>
      
      <connection-url>${MyDatabaseDS.connection-url}</connection-url>
      <driver-class>${MyDatabaseDS.driver-class}</driver-class>
      <user-name>${MyDatabaseDS.user-name}</user-name>
      <password>${MyDatabaseDS.password}</password>
      
      <min-pool-size>10</min-pool-size>
      <max-pool-size>20</max-pool-size>
      <prepared-statement-cache-size>10</prepared-statement-cache-size>      
   </local-tx-datasource>
</datasources>

Now edit /etc/jboss/system.properties to hold those values... it should look like this


# config for database-ds.xml
MyDatabaseDS.connection-url=jdbc:postgresql://localhost:5432/database
MyDatabaseDS.driver-class=org.postgresql.Driver
MyDatabaseDS.user-name=username
MyDatabaseDS.password=password
# this example is for postgresql but you could substitute any database.

... when you start up JBoss it will read the /etc/jboss/system.properties file and when it looks at database-ds.xml it will see the ${variable} values and substitute the values you set in system.properties for the ${variable}. If a ${variable} is not present, there will be no substitution... so be sure that every time you use a variable you set its value somewhere.

You can also set system wide system properties this way too. Commonly on servers that I work with the unix host name has nothing to do with the hostname being served. So I commonly set:


server.host=server.example.com

For use in my Grails applications which I then read in Config.groovy like this:


grails.serverURL = "https://${System.getProperty('server.host')}/${appName}"

... but that is another story ...

QRCode Linkulator the interesting bit.

2010-03-30T08:58:00.003-04:00

For me the interesting bit about writing QRCode Linkulator was creating a PNG on AppEngine. The JRE Class White List doesn't include any Graphical libraries. Notice that most of java.awt.* is missing?

Why would Google do that? Well, try downloading a copy of the OpenJDK. Go ahead. I'll wait. Got it now? Great. Now look in openjdk-6-src/jdk/src/share/native/sun/awt ... what do you see? A mess of native C code.

So key components of OpenJDK are written in C. Key components of AWT. That means if you want to do graphics (server side) in Java you need to hit some chunks of C code. I bet that's dangerous in AppEngine.

Combine this with the fact that pngj is the only project I could find that had a pure Java implementation of the PNG specification and the challenge of doing the "impossible" and the temptation to pull this off was irresistible. Consider how many programs rely on those classes and there is NO pure Java solution in existence.

At my current workplace we do a lot with cloud computing. Because of this I've got a special sympathy for the guys working on AppEngine. They probably have some technical infrastructural reason they can't let you get at those AWT classes either directly or indirectly. My guess is it has to do with how they implemented the custom JRE that your app runs in.

That combined with an interesting socio-economic constraint they placed on AppEngine applications makes me think that AppEngine isn't just "free" as in beer hosting but really an interesting way for Google to codify their chief aesthetic concerns about how applications function.

This aesthetic is born of cloud computing on a scale no other company gets close to. At this scale, apparently, an aesthetic similar to that found in embedded systems starts to come through. Which in my mind is extremely ironic. See... I've been doing some Android development lately.

Consider that your app-engine app may be sitting cold with no one using it. Then the user hits the application for the first time... perhaps deep linking into your application. The deep link re-activates the app. It's cold, dark, and if you are a scripting environment... not compiled requiring an interpreter to fire up, load your script and then interpret it.

Huge penalties in that environment. So I hear some clever developers have kept their applications hot by writing Google AppEngine Cron Jobs which hit the web application in some low cost way to keep it warm. That's pretty clever. But only works until your application scales.

Yes. That's right. It works until your application becomes really really popular. Why?

When your application scales your application is going to get provisioned with another "machine" on the Google Cloud. That application will be cold. You'll pay the warm up penalty for that application and that first user gets the raw deal having to wait while your app spins up. Bad news. This happens every time your app scales up one "machine" and the net effect can be Fail Whale like badness.

Besides, inflating your own app's "popularity" by "hitting yourself" seems a tad... well... evil. So please. Stop hitting yourself.

The "right" answer is to somehow mitigate that first page hit cost. You get that first page view cost down as close to zero as you can... then take it a little farther.

How? Anyone have ideas?

QRCode Linkulator speaking Gaelyk

2010-03-28T13:47:00.005-04:00

I've just written my first application using Gaelyk a Groovy based framework for Google App Engine. I ported the qrcode plugin I had written for Grails over to Gaelyk in a matter of hours.

This isn't really an application for iPhone or Android but it relies on the Barcode Scanner on Android or an app like
Barcodes on the iTunes AppStore for iPhone. Combined with a tiny "bookmarklet" I found the result is quite useful for moving a link between my desktop and my phone. I made a demonstration video since this can be a little hard to explain.

You can get a gander at the app here: QRCode Linkulator. I make use of the HTTP referer header property to determine where you came from so if you were to click http://qrcodelinkulator.appspot.com/300 for example then you would get a QRCode for the URL you were at *before* you clicked the link. The error mode (should you have no "referer" associated with your browser (for example if you were to right-click the link and open it in a new window) is just the logo for the app.

You might notice I tested out this feature in my blog. When you hit a given page in my blog there will be a QRCode for the URL you are currently looking at embedded in the page. This is done with simple image tags and no javascript.

Right now, I'm just hosting the app on the free AppEngine account and hoping that is enough CPU + Bandwidth for the application. I'm also hoping to stress-test the app a little to get a feel for how things scale. So far Gaelyk is running error free. I'm quite pleased with the simplicity of the framework and the price is right for App Engine hosting.

I doubt I'll get enough traffic to have to worry about paying for the service if the app gets that popular I'll have to figure out how to get it to pay for itself. Any ideas?

Domain Specific Languages @ TriJUG

2010-03-16T08:30:00.002-04:00

This version of my DSL talk focused on architectural implications of Domain Specific Languages. Less time is spent on language details and I dive right into creating the language.

Speaking at TriJUG

2010-03-02T11:36:00.001-05:00

I'll be giving a version of my DSL talk at TriJUG coming up March 15th.