Step Ahead Software

JDOQL: What is the equivalent of SQL's count(*) in the JDO world?

2020-12-18T17:00:00.004-08:00

Displaying large datasets

There are often times when you need to display data in a table format and the size of the potential dataset is extremely large and so ends up being represented across many pages and displayed with the usual "previous/next" buttons.

Most UI frameworks need to know upfront what is the total count of items you will be displaying so they know how many pages there are and can display some text above the table that says something like:

Items 31-40 of 397

Avoiding a big database, memory and CPU hit

What you don't want to do is force your JDO ORM (or an ORM using any Java persistence standard e.g. JPA) from instantiating ALL of the objects just so that your UI knows the total count of items to be displayed.

To do this you need to perform some kind of 'count' query. In JDO this is done by setting the result set to 'count' but count needs a parameter.

e.g.

PersistenceManager pm = getPm();

Query q = pm.newQuery(MyClass.class);

// Set up any filtering here ...
q.setFilter("value > 300");

q.setResult("count(batch)");

where batch is an attribute of MyClass, the extent of which, you are querying. Now that's all well and good but what if you don't want to perform the 'count' on any specific field.

I just want a total record/object count

In raw SQL you would do something like:

select count(*) from mytable;

but that '*' is an SQL concept only. However, there is an equivalent in JDOQL and given it is OO based you can guess what the equivalent is in JDOQL - 'this'.

So the following will make sure you get the exact count that you are looking for:

q.setResult("count(this)");

If you were to perform a Collection retrieval with the exact same filter then the above count will always match the number of objects in the returned Collection. However, if you use count(attributeName) this may not always be the case based on uniqueness of the attribute's value etc.,

Conclusion

Until we came across the count(this) construct we were having issues with our counts returning more objects than a query with the exact same filter that returned a collection of the objects. I hope this helps anyone else out there who has this issue.

Extra performance hint

Some JDBC drivers which will always retrieve all records to perform a 'count' operation even if you have specified only a count(this) dataset in JDOQL above.

That is not desirable on large datasets!

To overcome this issue with we can, if using the DataNucleus JDO implementation, use its extension that tells it to inform the driver NOT to pull all records to perform a count:

q.addExtension("datanucleus.query.resultSizeMethod", "count");

Enjoy!

Upgrade from MySQL 5.7 to MySQL 8 - Do it! Do it now!

2020-07-21T12:26:00.002-07:00

Our web hosting application platform Pagebloom has been running on MySQL 5.7 for some time now and it runs brilliantly - it's exceptionally responsive and provides great user experience.

The performance of the underlying MySQL 7 has been great except...

Clustered indexing: The double edged sword

Part of MySQL's great read performance is due to an "under the hood" design feature that most software developers who use it are blissfully unaware of.

In the automobile world it would be the equivalent of fuel injection or a turbo/super charger or in German - a Kompresser! These are features that can be added to the design of an engine to provide a significant boost in performance.

For MySQL this feature is what is known as a "Clustered Index". The main concept of a clustered index is that the core data of each record is actually stored within the primary key index. Yes, you heard correctly - actual record data is embedded into the B-tree frames that make up the index - I kid you not! At first this may seem crazy but what it means is that for data retrieval that involves locating a record via its primary key - arguably the most common type of retrieval in a relational database - once you have located the correct B-tree frame in the index you have the data right there as well. In non clustered RDBMS architectures only a location reference to the actual data is stored in the index frame so retrieval of the record data at that point requires an extra disk read but in MySQL the retrieval can be done with a single disk read.

Schema changes are MySQL 5.7's Achilles heel

While a clustered index provides great read performance its write performance has traditionally suffered in the MySQL 5.x versions.

Where the clustered index has caused very noticeable performance issues is when you need to do schema changes that add a column or two to an existing table with millions of records.

If you google "mysql add column slow" you can read endless accounts of slow schema updates with the 5.x series versions. Some people report waiting, literally, for days for it to complete! This can cause real issues with maintenance/upgrades of live production systems.

MySQL 8 to the rescue!

For one recent update to Pagebloom the required schema changes were to core tables with literally millions of records that hold organizations, people, users etc., Whilst performing the schema update in a test system we saw the schema update took many hours on MySQL 5.7 - this was unacceptable as, even if we performed the update at 2AM in the morning, we still did not wish to take the live system down for multiple hours. We do most updates at that time of morning in under 60 seconds.

We had heard that MySQL 8 had made some inroads to this "slow schema update" issue and so we thought a possible path was to migrate from MySQL 5.7 to 8.0.21 FIRST and then do the schema update once we're live on MySQL 8 - it sounded like a plan so we went with it.

The time to shine is 2AM!

Early one Monday morning we did the fairly quick migrate from 5.7 to 8 and then the following Tuesday morning it was time for the schema update which had taken many hours in our MySQL 5.7 tests. How long would it take on MySQL 8 in production?

OMG! It completed in under 5 minutes!!! I don't know exactly what the MySQL team have done with MySQL 8 but it's like they've added a Kompressor/turbo to their clustered index architecture.

This "many orders of magnitude performance improvement" is very rare in the software world but it is very much appreciated and in fact, is basically mandatory for MySQL to be running real, live, production systems in these days of sophisticated database schemas with highly populated tables.

Conclusion

Never be the first to jump in to a new major version release but after it is stable and many guinea pigs have gone before you and suffered all the necessary "burns" required to get a new major release ready for widespread consumption you really should consider upgrading.

In terms of MySQL 8 it is definitely time to migrate from 5.x releases if you have not already done so. The recommended approach is to upgrade to the latest version of the 5.7.x releases first and then the upgrade to 8.0 should be much smoother. Maybe, like us, you will really appreciate the improvements.

The case against IP 6: When scarcity can be a good thing

2020-07-16T17:18:00.016-07:00

With IP 4's limited range of public IP addresses and free IP 4 addresses becoming scarce it makes sense that the world should migrate over to IP 6 ASAP ... or should it?

What's an IP address anyway?

A bit of background for those who did not know that the internet predated the world wide web by a good two decades or more:

IP addresses are kind of like a physical street address. You write a physical address of a place anywhere in the world on the front of an envelope, put an appropriate stamp on it and then one or more postal services around the world will cooperate with each other to, hopefully, get your envelope to the "TO" address you specified.

A public IP address identifies a "place" (e.g. a web server) on the internet where internet "data packets" can be sent. There's a lot of different protocols that can be used to send data across the internet but at the lowest levels they are all based on a simple principle: The internet is basically one huge "data packet" delivery network and, just like an envelope, every data packet has a "TO" address embedded in it at a special location in the data packet header and like many envelopes there is also a "FROM" address indicating where this packet is coming from. The job of the 'data packet routers' that power the internet is to route data packets along a variety paths to other routers towards their final destination given by the "TO" address.

That's basically it. I've just described the lowest level of the internet in one paragraph.

Why would we run out of IP addresses?

In IP version 4, the well established, universal Internet Protocol version that every device on the internet supports today, each IP address that appears as a TO or FROM address in the header of those data packets is stored in just 4 bytes which is essentially a 32 bit long integer.

The maximum number of unique values that can be stored in a 32 bit integer is 2 ^ 32 = 4,294,967,296!

The internet standards declare certain ranges of addresses within this 4,294,967,296 range as 'internal' IP addresses - your data packets can't reach a device on the public internet if you specify at "TO" address which is one of these declared internal IP addresses: every packet routing device on the public internet has been coded so as NOT to pass on any packets with a "TO" address which is within any of the declared 'internal' IP address ranges.

So the maximum number of public addresses is reduced by the number of declared 'internal' addresses (and some other complex technical issues which we don't need to visit here) so let's just say there's "about" 4 billion public IP addresses all up.

Back in the early days of the internet 4 billion seemed like an incredibly large number but now that the internet has really kicked on suddenly ... not so much.

Every publicly accessible device on the planet needs a public IP address. This means publicly accessible websites like google, Facebook, your employer's website, your own website, mail servers etc., all need a public IP address or, in the case of the "big boys", a range of IP addresses.

With many billions of devices on the internet why haven't we broke the internet yet?

Even though there's roughly 7 billion people on the planet everyone seems to be having no problem with using the internet with their multiple devices. You can post a message to someone's Facebook Messenger account half way across the world and their smart phone will instantly "Ping!" them to tell them they've received your post. That might appear like their device is accessible to you or, at least, Facebook from anywhere in the world and you might think their device would need a public IP address for this to happen but, fortunately, with 7 billion people on the planet it is not required for every person's device to have its own publicly accessible IP address!

In regards to websites needing a public IP address - they don't actually need their own unique IP address - they can share IP addresses. Many web hosting companies will locate many smaller websites at the same public IP address. That used to be an issue when it came to SSL/TLS certificates that allow for the HTTPS instead of HTTP URLs and give you the comforting 'Padlock' icon in your browser but for many years now SSL/TLS certificates can work without requiring a unique public IP address for every domain name they secure.

In fact not every worker's PC in an office building needs their own public IP address. In the very early days many organizations actually used to do this but the world soon realized that this was a very bad idea because the organization's potential "attack surface" exposed to hackers become orders of magnitude larger than if the organization only had a few publicly accessible devices.

To avoid this dangerous exposure of internal devices via public IP addresses inside an organization we have the now ubiquitous technology known as 'masquerading firewalls': It allows only a few public facing devices to have "exposed" public IP addresses and everyone else sits nice and safely behind a firewall using those 'internal' IP addresses we spoke of earlier - to which no one on the public internet (e.g. hackers) can send data packets directly.

While improving security 'masquerading' had the side effect of each organization requiring far less public IP addresses than would otherwise be necessary.

What about home internet connections?

Up until relative recently in the history of the internet home internet connections were assigned public IP addresses by their ISPs when ever they connected - this consumed a lot of public IP addresses but in recent years new ISP connectivity technology means that ISPs only need to grant home connections 'internal' IP addresses. It means that their home modem/router is no longer directly publicly accessible from the internet but, again, that's an excellent security feature.

For others who might need a public, unchanging (static) IP address most ISPs offer that option for a small extra monthly fee. This is only necessary for people who want to host, for example, a web server or mail server from their home.

Internet advancements lead to less public IPs required

So what we have is improvements in internet technologies that have both:

Increased security by shielding end user devices both at home and within organizations by assigning them private IP addresses instead of public IP addresses
Decreased the demand for public IP addresses

So what does IP 6 do differently?

Unfortunately IP 6 does a lot of things differently. If all IP 6 did differently was use, no correlation intended, say 6 bytes instead of IP 4's 4 bytes for the TO and FROM IP addresses in every internet data packet it would have been much easier for device manufacturers, software developers and operations teams to adopt but noooooo...

Why would the committee who work on such things make such a trivial, small change that gets us a greater range of public IP addresses very simply when they could change *a lot of things* and make it so much harder for the world to adopt it?

If IP 6 had 6 bytes instead of IP 4's 4 byte IP addresses it would have given the world 65 thousand times the 4 billion public IP 4 addresses we have now. 6 bytes is 48 bits so 2^48=281,474,976,710,656. So about 281 trillion!

Having such a plentiful range of public IP addresses would have been sufficient one would think.

But no, clearly bigger is always better so IP 6 went with a 16 byte IP address - that's a massive 128 bits.

I once heard a professor say that a storage unit that is 128 bits long can store one of 2^128 unique values which gives you enough range to allocate a unique integer to EVERY ATOM IN THE UNIVERSE!

To say the IP 6 committee made sure the universe will never, ever run out of public IP addresses is an understatement. We would actually need more universes to use up that range.

The law of scarcity: bigger is not always better

So, clearly, as time went by masquerading technologies intended to increase internet security actually tempered the ever growing need for more IP 4 public IP addresses and so the internet still works, luckily!

So what's so bad about IP 6 having such a big range of available public IP addresses?

Funny enough, it's once again about security. You may have heard of White lists/Black lists which are lists of IP addresses that a certain device or software will always ALLOW or DENY access to.

These are essential in so many areas of today's internet:

Logon access denial: too many failed attempts to log on from a particular IP address starts to look like some 'bot' attempting a brute force attack on the system. A good system will lock out that IP address by adding it to a permanent or temporary 'deny list'.
Website screen scrapers: some 'bot' apps scan the content of certain websites to try to extract data that they do not wish to spend time or money obtaining in more honest ways. The website administrator can manually or automatically set up blockers that block access to such actors based purely on their IP addresses.
Website search engine bots: if these are drawing too many resources they can be blocked by IP address.
Email SPAM avoidance. You may have heard of IP reputation in the realm of email SPAM filtering. If too much SPAM is generated from a particular IP address it is assumed to be controlled by a SPAMMER/hacker and so earns a bad reputation by the large network of email monitoring services. Many SPAM filters will use that 'reputation' level to determine if you receive email from that IP address in your Inbox, in your SPAM folder or whether it's sent straight to trash.

There are many more cases where IP filtering/blocking is used but the important point is IT WORKS BECAUSE IP 4 ADDRESSES ARE SCARCE!

Scarcity increases value

IP 4 addresses are scarce and like anything that is scarce, like gold, it has much greater value than something that is abundant - like IP 6 public IP addresses. As world governments print more and more money they reduce it's scarcity and by doing so, deflate its value.

It's the old law of supply and demand in practice: Under IP 4 the supply of public IP addresses is very scarce but their demand is high so they become very valuable. It's hard to get your ISP or IP registrar to just give out even a tiny bunch of IP addresses at a time because they are so scarce.

Not so with IP 6 with it's 2^128 possible range... IP 6 public IPs are essentially worthless - they are like a crypto currency where miners can generate 1 new coin every millisecond on a PC with a 486 CPU - the coins end up totally worthless.

If hackers or spammers had unlimited access to large ranges of public IP addresses they would simply run their nefarious activities and eventually get their IPs all blocked or denied or have their SPAM reputation eroded but they would not care. Due to the massive amount of available IP 6 public addresses they would simply request another million every week, day or hour as required. They're virtually infinite, so can't be too expensive.

Have a thought for all those IP blocking tools I mentioned above. Instead of keeping a database of, possibly, thousands of 'naughty' public IP addresses they need to potentially keep a database that's large enough to store a unique number for every atom in the universe...

This would create an almost impossible burden on website administrators, operations staff and mail service operators of maintaining the list of all those 'bad' IP addresses which could likely hit the millions, billions or possibly trillions!

However, in the IP 4 world this can't happen because IP 4 addresses are scarce and very valuable. Organizations don't just hand them out like there's no tomorrow. Under IP 4 spammers, hackers etc., find it impossible to acquire large numbers of public IP addresses so their activities can be limited to some extent.

Stay safe. Stay valuable. Stay small.

Contrary to popular belief bigger is not always better. Smaller is more efficient and, above all, much safer. IP masquerading technologies are protecting more devices and requiring much less public IP addresses than before. The fear of running out of IP 4 IP addresses and the measures taken to overcome that fear, i.e. an infinite range of IP 6 addresses, needs more rational thought and careful consideration of all the consequences and subsequent devaluation that unlimited supply brings.

How to reset MySQL 8 root password on CentOS 7 and 8

2020-07-09T17:37:00.002-07:00

Occasionally you come up with the world's most unbreakable root password for a MySQL 8 installation and you get distracted before you record it somewhere.... and you remain distracted for 3 months.

Then you return to the project where this database was needed and ... whoops! What was that totally awesome password?

Here's how you can reset your root password. This was based on a page that had a similar instructions except none of them worked - lots of errors and incorrect names - like referring to mysql instead of mysqld when stopping the service.

Here we go:

All as root user:

Create a file in /var/lib/mysql called mysql-pwd with the following contents:

ALTER USER 'root'@'localhost' IDENTIFIED BY 'anotherPwd28^$94';

Shutdown mysql

  systemctl stop mysqld

Run MySQL executing the given init file under the mysql user

  mysqld --init-file=/var/lib/mysql/mysql-pwd --user=mysql &

Check the MySQL log to make sure no errors occurred:

  vi /var/log/mysqld.log

Stop the mysqld that was started directly by first finding its PROCESSID:

  ps -efa | grep mysqld

Stop it:

  kill PROCESSID

Start up mysql the normal systemd way:

  systemctl start mysqld

  mysql -u root -p

You should now be logged in as root user

Edit the mysql-pwd file to remove the password value so that root password is not available to people viewing that file.

Asynchronous notifications from the server to the browser - perfect

2019-10-29T13:50:00.001-07:00

This is why I love Wicket Java Web UI framework.

It provides all the power of the latest JavaScript and browser capabilities without ever having to write any code in that crazy, non type safe, interpreted JavaScript language ;)

Wicket components have all of the required JavaScript embedded within them meaning a Java developer of any skill level can simply instantiate a few components, add them to a page and bang! - too easy.

So about asynchronous notifications...

The 'status update' problem

Often within a web app the user will invoke some operation that can take some time. The page or part of it needs to be updated to reflect the new status after the operation has completed.

To do this developers, traditionally, have had to do one of these:

Make a request to the server which the server delays responding to until the operation has completed. This is inviting bad karma to your web server because it ties up a connection and the number of connections in a web/app server are not infinite. What's worse, in many application servers tying up a connection also ties up a thread - which consumes much more resources than merely tying up a connection.
Start up a poller in JavaScript which regularly polls the server to get the status on the operation. When the operation is complete the poller is disabled and the relevant panel(s) on the page are updated. This option has a latency issue: for example if you set the polling period to 2 seconds then the user might have to wait for up to 2 seconds after the operation has completed before they see the update. You can make the polling period very small but this solution will not scale well as with thousands of users all hammering the server with update requests may place too much burden on the server.

A better solution

Quite a long time ago a technology called "Web sockets" was introduced which allows for the server to asynchronously notify a client (browser) of a change of state. e.g. the completion of an operation or an update of the value of a stock.

Wicket provides a very simple "wrapper" for Web sockets which allows the developer to add Web socket support to their app without writing a single line of JavaScript code - it can all be done in Java - the clever Wicket devs have done all the hard JavaScript coding yards for us, once again! (Thank you guys!)

Here is an example of an asynchronously updated graph:

http://examples8x.wicket.apache.org/websockets/behavior

Here is a tutorial on how to write an Web sockets app using Wicket:

https://dzone.com/articles/websockets-with-wicket-tutorial

I'm coming clean. I'm letting everyone know that I'm a bug addict...

2019-10-21T19:19:00.000-07:00

The other day I added a new, rather complex feature to our Filegroove product. I did the normal planning and then analyzed the pros and cons of a few different design implementations.

I then went ahead and implemented the chosen design and ...
... it was a bit of an anticlimax when it basically worked as designed, first go 🙂

Normally I have the excitement of hunting down some tricky bug that needs super detective skills because of some incorrect assumption I had made or due to some oversight but no ... the damn thing just worked!

My reaction made me ponder:

I think I've become addicted to the dopamine hit I get when I solve a tricky bug. I miss that when everything just works - need to get my dopamine hit from somewhere else - maybe some FB pings? LoL!

Here's my official admission:
I'm a bug addict. I've been using bugs for many year now.

I tried to get off the bugs but its just too hard.

After a few days without bugs I can't sleep at night and go looking for a fix - boom tish!!!!!!

Maybe that's why I called myself the OO Junky? :)

Git and Subversion - the working directory is fundamentally different

2012-10-22T15:16:00.002-07:00

One of the fundamental differences between Git and Subversion that I didn't understand at first and, IMHO, doesn't really get much coverage in the documentation is the way the 'working directory' is treated.

Git is all about branching and it is indeed very good at that but what happens when you switch branches in Git is completely different to when you switch branches in Subversion.

You could say that you never really "switch" branches in Subversion. You can create branches and checkout a branch in subversion: In subversion if you want to use a branch you typically 'check it out' to a new clean 'working directory'. This directory is a new place in your filesystem and you perform work on that branch in this new working directory. You can't easily 'switch' a working directory from one branch to another branch.

With subversion if you are working on a number of different branches then you would typically have a number of different 'working directories', each being for a specific branch. In Git things are very different...

In Git you typically have a single 'working directory' and when you want to work on a different branch to the current one you literally 'switch' branches and the source files in your working directory are modified to reflect the state of the files in that different branch. When you switch to another branch the source files are updated again to reflect the current state of every file in that other branch.

At first this seems like a lot of work for the revision control system to be performing but git seems to handle it in its stride - and very quickly. Most of the time the differences between two branches are only relatively small compared to the size of the overall code base. Git is very fast at storing 'deltas' and rendering and backing out those deltas on source files as required by branch switching.

You might ask, "what about changes that I have made to files in the current branch?"

Git won't let you switch branches if you have changed files in your working directory. You have to either commit the changes (not so bad as you're only committing to your local repository) or 'stash' the changes. Stashing tells git to store all the changes you've made in the current branch without a more official and long term 'commit'. You could say 'stash' is the perfect solution for developers who are scared of commitment. You can leave your current branch after a stash and return to it whenever you feel like it.

So the ability to switch between branches in Git means that you can have very quick access to any branch in your repository without having to checkout a new, full working directory on the filesystem. In other words, Git branch 'switching' is a lot faster than Subversion's 'branch' checkout.

I have to admit I wasn't at all excited by the arrival of Git at first - "Oh no! Another revision control tool to master" but many of the open source projects that I use on a regular basis are now hosted on GitHub so I have undergone a forced introduction to Git. I have to say that the more I use it the more I like it!

Classic software engineering mistakes: To Greenfield or Refactor Legacy code?

2012-09-10T23:50:00.002-07:00

Oh, the humanity!

If we were not in the software engineering game but, rather, in the civil engineering game then the equivalent of this article would be called 'Classic Civil Engineering Mistakes' and contain graphic videos of buildings and bridges collapsing and thousands of people running for their lives.

It's hard to get emotional about something you can't see

Unfortunately software is a rather intangible asset that doesn't lend itself to dramatic and emotional visualizations. When it comes down to it the life work of a software engineer, when reduced to it's lowest conceptual level, is the specific arrangement of sequences of 1's and 0's on one or more hard disks (or some form of solid state storage) residing on a server or desktop computer. How those 1's and 0's get there and 'which' sequences work and which don't is not really a black art but it definitely requires some intelligence but surprisingly a lot more common sense.

Some arrangements of 1's and 0's form social networking services like Facebook and Twitter, earning their creators billions of dollars. Other arrangements are from lesser known companies that, while not making billionaires out of company owners and millions for employees, can provide sufficient income to keep families fed and mortgages paid.

Other arrangements of 1's and 0's never earn any money at all. Still others begin successfully for their creators but through circumstance, changing environment or poor decisions the success teeters off either abruptly or over an extended period.

Opportunity knocks

At certain times in a software company's life new market opportunities, constraints or internal political rumblings arise in relation to one or more products produced by the company that demand analysis of various possible strategies going forward. Often the intended end result is an extension or enhancement or refinement of an existing product. The evaluation of a way forward usually involves choosing between one of the following basic strategies:

Creating a new greenfield code base for either all functionality or a portion of it.
Retain a single code base for legacy and new products, refactoring, enhancing, refining the legacy code base where required to provide the extra functionality required by the new product. The same code base can execute in multiple 'contexts' (i.e. products) via well established context dependent coding mechanisms (eg., standard factory patterns, template pattern etc.,) where certain behaviors are abstracted and context/product specific implementations are 'plugged in' at run time.
Copy/paste the legacy code base, creating a completely independent code base for the new product such that there is no requirement to support the legacy system in the duplicated code base - support for the legacy system will be provided via the original legacy code base.
A hybrid of 1 plus either 2 or 3 where new functionality can be sufficiently decoupled that it can be developed as greenfield project but it integrates with the legacy system (2) or a copy of it (3) through well defined interfaces - some of which may not exist yet but can be added, usually with some fairly straightforward refactoring and abstraction of certain functionality within the legacy system.

This article relates my experience during an early software engineering position I held 20+ years ago when I was bright eyed and bushy tailed and very naive (I like to believe that I'm still bright eyed and bushy tailed but no longer so naive!)

Some seemingly plausible but, in the end, damaging strategic decisions were made in regard to the way the software team arranged their 1's and 0's as you'll see...

The Company

The company I worked for produced software to run in sophisticated controllers for at least three different products. Let’s just call them ‘engines’ so as to not make the company identifiable. They all supported very similar functionality but varied in the hardware that they were deployed to so the software for the three different engines was built from the same code base – in places where engine specific code was required we used C’s pre-processor macros to customize the code for the different engines (this was in the infant days of OO and not many people had heard about C++ - which would have been a more elegant solution)

#ifdef ENGINE1
…
#elif ENGINE2
…
#else // All other engines
...
#endif

Example of an engine specific section of code

99% of the code didn’t need this as it was common to all products but some of the source files had it and while inelegant, it meant we only had to maintain a single code base. Any bugs or features added to the common code for one product was automatically inherited, at no extra cost or effort, by the other products. It was very maintainable, highly productive and made a lot of sense.

but... one day a (insert adjective that starts with 'n') software team leader decided it would be much 'cleaner' if we separated the code into three separate code bases and remove all the pre-processor macros. So without any further discussion and much to the amazement of some of the more astute software engineers, he split the code into 3 separate code bases even though 99% of it was common to all engines.

The theory was that by removing the ‘ugly’ engine specifics from the code we’d all be working on ‘cleaner’ code and therefore productivity would go through the roof.

That was the beginning of the end for the software department in a historically successful, medium sized enterprise. I won’t mention the name because I don’t want to offend anyone who was involved in the decisions that pre-empted the escalation in costs that contributed to the eventual closure of its software department.

[Aside: very few of the pre-processor directives were ever removed from the code bases so the perceived 'ugliness' was never removed - we just had 3 separate versions of it :) ]

Three separate code bases: What could possibly go wrong?

In the beginning the three code bases were still fairly identical so deltas and merges were almost manageable but before too long subtle differences meant that simple copy/pastes from one source base to another was almost guaranteed to cause regression bugs. In sections of the code that were previously purely generic (i.e. no engine specifics) bugs were being raised by customers for one product that were not present in the other. It quickly became a dog’s breakfast that was very expensive to maintain.

Within months a hiring spree ensued because now when a bug was found it had to be fixed in three places instead of one and when new features were added it had to be added to three files instead of one - the team needed more developers to perform this laborious 'duplication of effort'. As a developer I recall suffering the classic 'same but different' problem that occurs when you work in different instances of essentially the same thing - there is very little contextual difference as things look so similar so, a few weeks later, remembering which 'instances' you fixed and which one you didn't becomes very challenging.

"Same but different" can be dangerous to your health

There was once a German car racing track that relied on this 'same but different' confusion to trick drivers. It had multiple bends that were deliberately designed to look 'exactly the same' and they were essentially the same except for the last one, the exit of which went in a completely different direction to the rest. You can guess what happened - they had to close the track from too many driver deaths as many drivers turned the wrong way on exiting from the last bend - confusing it with another that went the opposite way. Due to the confusion caused by the 'same but different' phenomenon their brains lost track of which instance of the 'same thing' they were driving in.

Anyway, enough about real things like cars and racing tracks and back to 1's and 0's...

Many more developer hours were required to maintain three code bases than the previous single code base – which makes perfect sense to the average man in the street even though the people initially pushing for the ‘cleaner’ code base(s) found this outcome *surprising*. Nevertheless they arrogantly continued with this 'new' approach to how we arranged our 1's and 0's because to go back would mean an admission that they were wrong and for 'n' type people that is very hard to do. One of the guys leading the team was one of those personality types that would rather run the ship into an iceberg rather than admit they had the ship pointing in the wrong direction.

Pretty soon the quality of all three code bases dropped dramatically as the maintenance effort was overwhelming even the larger team that had been established. The products with the least market share started being treated like second rate citizens with all effort going into the most popular products. Previously, updates in the common code (99% of it) were occurring for the less popular products for free, now they weren’t happening very much at all as it became too expensive and time consuming to port new features and bug fixes across from one code base to another.

The company had effectively close to tripled the cost of maintaining and enhancing its cash cow’s software code base without a single extra dollar in revenue coming in for all that extra effort and cost. Customer's don't really care if you've got pre-processor macros to implement conditional builds in your code so long as the code works.

Rather than realizing the expected, yet completely unquantified, productivity gains from establishing multiple, but ‘cleaner’, code bases the productivity dropped dramatically as did the quality of released products. The sales team were constantly asking “Why is bug X fixed in product Y but is still there in product Z? I promised the customers it would be fixed in the latest release and it's not. You have embarrassed me.” and customer satisfaction dropped accordingly.

This plane’s altitude was dropping quickly but it still had two of the four engines running and could have limped to the nearest airport if they were smart about it but only the bravest and humblest manager would be able to make the smart and responsible decision required at this stage in the game…evidently that type of person didn't make it to the second round interviews at this place ;)

New 'clean' greenfield project: What could possibly go wrong?

What happened next was a very exciting opportunity offered to many of the software engineers and therefore well accepted by all the software team (including me!) – a new greenfield project called the ‘re-engineering project’ (I’m sure thousands of companies have had projects with re-engineering in their title so I’m not giving away any company identification hints here) which, in recognition of the set of 3 separate unmanageable code bases was to rebuild a new “re-engineered’’ single code base from which the software for all products could be produced. Common, reusable frameworks and libaries would be built. It was a very ‘back to the future’ experience – creating a new single codebase – much like we had before the code split except that all the code would be shiny and new and therefore much, much better!

The ‘re-engineering project’ started with great enthusiasm and it was decided early on to avoid using any of that yucky, dirty existing code that had undergone years of development and field proven robustness.

Initially about 18 engineers, including myself were put on the project. It would use the latest 'state of the art' CASE tools whose licensing cost thousands of dollars per engineer - not to mention the cost of training. Everyone was very positive and keen to "get into it". What could possibly go wrong?

Well there was one minor problem - the project would take 12-18 months and would not earn 1 cent in revenue until it had been completed, fully tested and customers started buying it. Meanwhile there was a "dog’s breakfast" of 3 difference code bases that needed to be maintained and the ‘re-engineering project’ just stole about 18 engineers from the maintenance effort. Whoops!

As you may have guessed, in a company with limited financial resources, engineers started getting poached from the re-engineering project and put back on maintaining the old code for the cash cow products. When we were down to about 6 engineers I remember thinking, “Ok, this project’s ETA has just been extended by about another 24 months”. Unfortunately that wasn’t the end of the poaching.

The re-engineering earned ZERO dollars and was of intangible benefit in management’s eyes. Maintenance and quality of the existing products, which WERE EARNING MONEY was dropping as less engineers were available to maintain them. Over the next six months management gave directions to R&D to poach even more engineers back to maintenance to improve quality and turn around times for new features. This continued until there was only a single person left on the re-engineering project…me. While rather chuffed to be the last one standing it was pretty damn lonely and the ETA had now become a company joke: 18 engineers = 18 months so 1 engineer = 324 months!

So what can software managers learn from these types of experiences to make better decisions for how their team organize their 1's and 0's? Well it depends on a number of variables. What may be surprising to some is that a very important variable is a non technical one: the size and therefore financial resources of the company. Here's a list of typical outcomes based on company size:

The little guys

The little guys, small enterprises, rarely have a choice. Their decision is constrained by the economic reality of very limited financial resources and they usually don't have the financial resources for a full greenfield 'rewrite' of their code base. Often their only option is to 'get agile': continually improve, refactor and enhance their existing code base. Over time much ugliness in the code can be replaced by elegant design. The improvements are incremental over time instead of 'big bang' at the end of a 2 year project development - which a small company simply can't afford.

As turn around times for each iteration is usually from 1-4 weeks the improvements to the existing code base are regular and financial benefits can be realized in the very short term. In other words, agile style, incremental improvement of the legacy code base produces a repetitive, financially viable sequence of deliverables with excellent 'return on investment' (ROI).

In my experience this is usually successful as it does not incur large financial commitments on behalf of the small enterprise. My second place of employment was a small but successful company. I wrote a sophisticated Windows application for them in the early 90s. It was ported from 16 to 32 bit when Windows 95 was released and they are still deploying it 20 years later and have continued enhancing and maintaining it over two decades.

My own company is relatively small yet we have developed many reusable frameworks over many years and we have incrementally enhanced and refactored these on a continuing basis. We have never hit any requirement that has caused us to even come close to being tempted to 'ditch it and start with a clean slate'. It certainly helps to design with flexibility built in - often abstracting behaviors into clearly defined interfaces to reduce coupling and increase re-usability - but even with an old code base that wasn't built with this is mind it is usually possible to refactor to repackage legacy code into an implementation of an interface that then allows that behavior to be reused by other modules or even expose that behavior as a service to be reused by other applications. It also allows other modules or applications to provide their own implementation of that interface. The trick for knowing when an opportunity for such abstraction exists is to be ever vigilant against the temptation to copy/paste. If you ever feel the need to copy/paste a section of code then right there you probably have a candidate for some abstraction and re-usability.

The big guys

The big guys, large enterprises typically do succeed with large, 'big bang', greenfield projects. If you’re a Microsoft or Google you typically have enough resources that you can do all the extra hiring you need and don’t need to pull staff off one project to start another and so there’s no need to poach people from the Greenfield/Re-engineering project and put them back on maintenance of the company's existing cash cow product(s) when the legacy code suffers from neglect. This approach can work if you happen to be a Fortune 50 company with infinite funds to invest in product development and so don’t need to poach resources from the Greenfield project back to maintenance roles.

The medium sized guys

Stuck between little guys and big guys is the medium sized enterprises. This is where things get very tricky. They are not like the little guys whose decision is forced upon them due to 'shoe string' budgets but equally they are not like the big guys whose seemingly infinite budgets mean they can fund even the largest of projects.

These guys often do have budgets that can run into multiple millions but lets be clear: millions are not billions like the big guys deal with. Millions can sound like a lot of money but if an average software engineer is paid around $100k/year and you create a new project that needs 8 engineers you've just incurred an extra ~$1,000,000 of costs every year once you include rental space, furniture, benefits, pension contributions, training etc., Has revenue increased by that same amount over that year? Is it expected to increase once new customers use products built with the 'new and shiny' code base or won't most customers notice the difference?

Some managers justify the greenfield approach by saying we're going to have no new hires or only one or two and so the overall cost to the company is not significant. If you need 8 engineers but hire only two extra engineers then you have to pull 6 engineers from an existing product, typically the company cash cow and that has all the risks I outlined above. Poaching is the main risk - every poaching pushes out the ETA of the greenfield project and reduces it's chance of success as the ETA approches the 'Greenfield Window'.

Greenfield Window

I use the term "Greenfield Window" to describe the very limited window in which a greenfield project has to succeed. One that window closes it's game over for the Greenfield project. All the code created is archived in the version control system and rarely ever touched again.

Greenfield projects take lots of developer and financial resources and they don't earn the company ANY revenue until the product is being used by paying customers. For even modest projects this can take 12 months but in larger products this can take many years.

During greenfield development there are all sorts of pressures placed on the project. Pressures from accountants, management, creditors and shareholders who ask questions like "how much is this costing us and how much has it earned us?" Until the project is completed and the first customer pays you something the answers to this question are always "A LOT & ZERO" which really tests the patience of these stakeholders. It only takes one of these groups to lose patience and the 'Greenfield Window' slams shut very abruptly - on your fingers because you were gazing out the window at the time, looking at the greenfield and wondering when your 'crop' would be ready for harvest.

This is a very painful experience both financially and emotionally. In addition to thousands or millions of dollars wasted egos are hurt and egg lands firmly on faces. Morale among the team can be affected and office politics and power-plays take on a new vigor as the dust starts to settle under the mushroom cloud. It's the injuries sustained by egos and subsequent political game playing that can have long lasting effects.

Greenfield Specification Vacuum

Often legacy code has evolved over such a long time and in a relatively undisciplined way that much of the functionality is not documented anywhere. The code has become the specification. Hundreds of boundary cases and error conditions have been 'handled' by the code without any documentation of such behavior. Process/workflows are 'embedded' deep into hard coded state machines that make decisions based on a set of business rules that are perhaps known only to the original developer or a business analyst who has long since left and often for situations that arose in real life scenarios that only they were aware of.

Throwing out legacy code and starting with a clean slate often throws the proverbial baby out with the bathwater.

Most greenfield projects that are designed to replace a legacy code base have no accurate functional specification to work with. Even if one were to be produced it's coverage of business rules, boundary conditions and exceptions that were implicitly, explicitly or even accidentally built into the legacy code would be severely lacking. A thorough examination of vast quantity of code would be required to establish a functional specification with sufficient coverage before the greenfield project can even commence. This is another advantage that refactoring the legacy base has over a greenfield development: the legacy code contains years of valuable explicit, implicit and accidental functional implementations and, even though they may be subtle, undocumented and hard to find, they are working and you can reuse them as they are because you don't have to throw them away and reimplement them.

With incremental improvement/refactoring you can attack these a section at a time over an extended time with each iteration producing better code that customers can start using after the usual QA process has completed. With a greenfield approach you will need to discover, document then reimplement in the new code every undocumented, hidden, subtle business rule and specification. I believe that the calculation of effort involved in this task will grossly underestimated by most software engineers overwhelmed with excitement at the prospect of starting work on a greenfield project.

In fact, if the engineer believes that their estimation of effort to create a functional specification via thorough examination of legacy source code is so large that it will make the project too expensive and therefore jeopardize it's chance of commencement there may be a temptation to deliberately underestimate the effort and thus create a 'surprising' cost blowout down the track. I most carefully used the word "tempation" here: whether an individual succumbs to this temptation or not will depend on their own moral code.

With the nature of typical full time employment arrangements there is rarely any direct financial bonus or penalty associated with accurate or otherwise estimation and/or implementation of functionality within the estimated range. The company takes all the risk for decisions made based on estimates provided by people who have no direct exposure to financial consequences related to the accuracy of such estimates.

Greenfield Partial Functionality

One of the "promises" (given without any justification or quantization) raised by promoters of greenfield projects is that the greenfield system will be "all singing and dancing", "infinitely cool and good" and of course be able to replace the existing system but, like politicians, software engineers are great at making promises but not always able to follow through - especially full time employees for which there is usually no direct financial benefit or penalty for delivering on their promises.

Let's say that the software team, in the face of overwhelming odds against it, manage to somehow create a working new greenfield product that does something like the old system did. Let's just say that the first deployment of the new system was for a slightly different market that only needed a subset of the functionality in the legacy system. What you have now is 'greenfield partial functionality'.

The greenfield project has created a new source code base that needs a team of engineers to support it but guess what - it only represents a subset of the functionality so that it can't replace the legacy system. In supporting only a subset of the functionality it is hard to find new customers to use the new software. The legacy system and the new system both need continued development and maintenance. The yearly costs to the company of the software department have now doubled and because the greenfield project has only partially implemented the features of the old system there is no medium term reduction in developer costs. The company now has two different software code bases to maintain for the long term.

Let's be really generous and say that the new project does implement every feature of the legacy system. We're still not in software nirvana because the new system, while it may have been based on the original class model or database tables has had quite a number of new attributes/columns added or undergone refactors to improve problems experienced with the old model/database. In other words that 300GB database that represents your customer's data can't be easily migrated to work with the new software.

It's like a 20 ton anchor that isn't budging until someone can write the "migration script from hell" to bring the old data into the new system. On the contrary, if the legacy system had been incrementally refactored then at each iteration, the data migration (if required) would likely incur very small model/database changes and so very small, manageable data migration scripts are required to migrate the existing data and hence all the users associated with that data.

Greenfield Paradox

What I have discussed here is the 'greenfield paradox'. It's a paradox because intuitively, nice clean, shiny things are much nicer to deal with and therefore easier to look after and maintain than old, dirty, ugly things. That might be true of cars, boats and houses but it's not true of software. Software doesn't rust or corode. Unlike plastic and wooden decking it doesn't break down in the sun over time.

Software is not like the 'normal' things we touch and feel in everyday life. It can be almost 'immortal' unless the environment in which it exists becomes extinct (Good bye DEC VAX we knew you well!). Software written to run in some banks or insurance environments 40 years ago can still run exactly as it ran back then - and some still does! The 1's and 0's don't corode or rust with time (ok, maybe metal 'core' based memory from the 60's might =]). Software is infinitely mutable. It can be bent, reshaped, refactored, re-engineered. It can be changed a little at a time and therefore it can be incrementally improved over time whilst maintaining the current user base and all of their data.

Sometimes there are paradigm shifts in the IT world which force a new greenfield development - for example moving an application from desktop based to cloud based - but if the legacy software was written with a decent model then even though much of the user interface code will need to be rewritten most of the 'core' of the application can be reused so long as it was written in a language that has wide adoption (eg., Java, C++ etc.,).

Hybrid Strategy

In some scenarios the extra functionality is sufficiently independent that only the new functionality needs to be developed as a greenfield project. A significant portion of the functionality can still be provided by the legacy code (strategy 2, preferrable) or a copy of it (strategy 3, duplication of maintenance effort and cost). The integration between the greenfield component and the legacy software can be via intelligently crafted services/interfaces that are provided by the legacy code base. While the hybrid strategy is not the pure 'clean slate' approach that many engineers might prefer it can result in massive savings and much faster time to market compared to a complete greenfield rebuild.

The time and effort involved in refactoring certain parts of the legacy code to abstract behaviors and expose them as services via generic interfaces is usually insignificant compared with the time and effort required to create a complete greenfield implementation of the legacy code. If a hybrid strategy of 1 + 2, above, is chosen over hybrid strategy 1 + 3, it means that the company's code base is not duplicated which avoids a doubling of the company's maintenance effort and cost.

Refactoring existing functionality so that it can be exposed via a clean interface requires patience, intelligent thought, design and high coverage testing. Not every software engineer has the necessary skill set and, just as important, personality and temperament to be up to this task which is why many will push for a 'complete greenfield' strategy. However, if you have one or two engineers who have the right technical skills and personal skills required to implement the intelligent refactoring required to implement a highly efficient hybrid strategy it could represent savings of thousands or even millions of dollars and massively speed up the time to market of your new product. As an added bonus: any improvements made to your legacy code base are instantly available to your legacy products with no further coding effort/expense.

Let the software engineers decide?

If you ask 100 software developers whether they would rather work on legacy code written by someone else or create brand new code from scratch then 100/100 software engineers would say 'brand new code, of course!'.

My 20+ year’s of industry observation tells me that it takes a very brave and very smart software manager to avoid the mistake of undervaluing the legacy code base. While they can see the potential for a major cost blowout in the construction of a new 'bright and shiny thing' they often find it hard to resist the overwhelming pressure from software engineers and others in the organization to 'restart with a clean slate' and the promises of 'future productivity improvements' that are 'guaranteed' by said software engineers. The guarantees are never in writing and the gains are never quantified.

Software engineers hate nothing more than OPC (Other People's Code). A greenfield project or a 'cut and paste' of the legacy code base with no requirements for backwards compatibility represents an extremely tempting though expensive opportunity for them to create an extra code base that software engineers that come after them can have the pleasure of hating for years to come :)

If ensuring popularity with or appeasing the software engineers is a manager's main goal then they will be tempted to decide to ditch or 'duplicate and change' a legacy code base rather than incrementally improve the legacy code base - and they would be among thousands of managers before them who have made that same decision. How many of those software teams are still around to tell the tale?

Strategies for refactoring legacy code

Incremental

The secret in refactoring existing code is that you want 'bang for buck'. In other words you want a great return for the smallest investment. This is the key to the success of the refactor strategy - you only change what you need to change now. The rest can wait till later. Over time the entire application may end up being refactored and enhanced. That's great but the secret is to make it awesome over time with incremental changes - they are much more manageable and testing their effect and fixing side effects is much easier and indeed more viable than a 'big bang' approach that touches lots of different parts of the code.

Database Access - Persistence

An immediate assumption by people who want to ditch the legacy code base and move onto a new or duplicated version of it is that it is impossible to refactor the old code base. It might use a database access layer that uses raw JDBC/ODBC or it may have no database layer at all: SQL statements appear littered throughout the code. One of the assumptions is that to move to an Object Relational Mapper (ORM) to gain the productivity benefits that they bring requires a new database schema - this is clearly not the case. Most ORM implementations in Java like Hibernate (JPA), DataNucleus (JPA/JDO) and others allow you to map to existing legacy database schemas. Each time you refactor a class to be persisted by an ORM instead of using JDBC/ODBC and SQL directly you actually decrease the size and maintenance cost of your code base. Most ORMs will play nicely within a system that accesses objects both through the ORM and through direct JDBC. This means that it's possible to migrate your code base to a new ORM oriented version in a piecemeal way over time - you don't have to convert the entire application over to ORM access before a new production release can be deployed.

Multiple Products - single code base

We touched on this earlier: it is desirable to have a single code base for the generation of product variants that are essentially very similar in function but might differ in titles, terminology, internationalization, color schemes etc.,
Clever use of patterns like the factory and template patterns allows code that used to include hard coded screen display or functional behavior can be refactored to deal instead with an abstracted interface. A product specific implementation of the interface is 'plugged in' to the application during configuration. New variants of the product can be created very easily by simply providing an new, product specific implementation of the required interfaces and setting up appropriate configuration files to 'plug them in'.

Changing the User Interface framework

A driving force for many a greenfield project is that the old user interface framework is technically 'extinct' like the dinosaurs or no longer 'cool' and much newer 'funkier' ones are calling. Ideally, regardless of the legacy UI framework there would have been a well disciplined MVC strategy applied or at least a good separation between the underlying model and the user interface code. I say ideally but I do understand that the architecture of some UI frameworks (eg., JSP) often encouraged the inclusion of model and business code right into the UI elements. If you fell for this trap then you need to slap yourself in the face repeatedly with a wet fish!

The evils of mixing UI and model or business code have been dealt with many times before and I know you have learned your lesson the hard way so I don't need to mention it again ... OK! What I will say, is that if you are starting with legacy code that suffers from this a good first step is to incrementally start a process of UI/Model separation. Create some separate model classes if you haven't already and start shifting model code, any JDBC/ODBC code and any SQL 'glue code' from your UI elements into those model classes. After some pure model classes have been established you can attempt to incorporate a more modern UI framework by targeting the UI elements that only require model objects that have already been 'purified'.

If on the other hand your legacy code already has Model/UI separation then your task is much easier. If not already the case, then move the model into it's own package and move the UI code into a separate package.

eg.,

Package:
com.acme.myapp.onebigblob

becomes:
com.acme.myapp.ui.jsp <- Any supporing JSP classes
com.acme.myapp.model <- Pure POJO model classes

Once that is complete you can start creating UI elements using your more modern UI framework (Wicket, Tapestry, JSF etc.,). In most cases these should be able to coexist in the one code base (we've done this with Echo and Wicket). The important thing is that while in transition both the legacy UI code and the new UI code both use exactly the same model classes - this is why we need complete separation of UI and model code. The model code must have no references whatsoever to any classes from either the legacy or the new UI frameworks. Create a separate package for the new UI framework (eg., Wicket). All classes related to that UI framework go into that package.

eg.,

Package:
com.acme.myapp.ui.wicket <- All Wicket page/panel classes

If your app is non trivial then you should create subpackages under wicket to organize individual logically separate areas of functionality into their own package.

eg.,

Package:
com.acme.myapp.ui.wicket
com.acme.myapp.ui.wicket.user <- Login form, account preferences
com.acme.myapp.ui.wicket.borrow <- Book borrowing and related forms

Together we stand, divided we fall

Divided we greenfield

Choosing to perform a new greenfield implementation of the legacy code divides by creating two separate teams. One consequence of creating a new greenfield project with a new software team is that it can create an 'us and them' scenario where the developers left on the legacy code team can feel like second rate citizens as they have been relegated to the relatively unexciting role of maintaining old code that uses a bunch of old technologies that no longer have the developer 'street cred' that they used to.

The developers on the greenfield project get to experience all the 'thrill and excitement' of skilling up and using all the latest and greatest technologies and the benefits don't stop there - their resumes undergo a potentially very useful enrichment.

Splitting the team into legacy and greenfield can cause divisions that can affect morale of developers 'stuck' on the legacy team - especially the aspirational developers who want to improve their skill set and who want to help the company improve and enhance its software assets.

In some teams there are developers who are happy to merely clock in each day and aren't at all interested in the challenges of learning to use new, exciting and usually, much more productive, technologies. The consequence of having these types of developers on any team won't be covered here: it would require a completely separate article to discuss issues related to that.

Together we refactor

A decision to perform a separate greenfield development will definitely create two distinct teams and can expose the company to the issues discussed above. On the other hand, a decision to refactor and enhance the existing code base as a unified team can create positive outcomes for everyone involved. The 'buzz' created by the challenge of adding new features and incorporating newer tools and technologies affects everyone on the team. Eventually all developers will gradually be exposed to the new technologies so no one is being left at the station as the technology train departs - everyone goes along for the ride. When the 'together we refactor' strategy is taken we remove the problem of developers on the legacy software feeling rejected or 'left behind'.

Tough call

In this article I have related my two decade old, real world experience of the consequences of choosing 'code base duplication' and 'greenfield development' over refactoring and improvement of an existing code base. Hopefully this has been informative enough for software managers to help them understand that making a decision that is popular with their software engineers is not the same as making decisions, however unpopular, that can drastically affect the financial viability of the software team in which those same engineers work.

Unless you're a Microsoft or Google with unlimited financial resources to spend on a strategy that costs a lot of money and does not give any return in the short or medium term then you really need to make decisions that avoid major increases in the size and/or number of your software code bases because creating and maintaining software is not at all cheap. Choosing the correct strategy for managing the production of your company's 1's and 0's is a tough call.

Share your experiences...

Please let us know of your experiences with software teams that had to make this crucial decision. You don't have to mention the company name but please let us know whether it was a small, medium or large enterprise and the decision they made and whether it was as success or not - i.e. did it make it before the greenfield window closed?

As a Java UI Framework let me say Wicket Rocks!

2012-06-26T06:04:00.005-07:00

Everybody knows that Wicket rocks as the best Java UI framework available. Sure there's people out there still using JSP (with/without struts), Spring MVC (Huh?), JSF, even raw servlets and a host of many other Java frameworks but let me repeat; "Everybody knows that Wicket rocks as the best Java UI framework available".

I suppose I should clarify that a bit.... everyone who has ever used Wicket knows that it rocks....

You see, if you're a Java programmer who "gets" the whole Object Oriented paradigm and loves the reusability that comes from plug and play component architectures and from clever use of inheritance then you simply won't find ANY other framework that satisfies your need for such architectural candy at the UI level but that still allows the visual candy that expert web designers can create when they have full control over HTML and CSS.

All I can say is "Get into it already". Get started. Read up on it. We've been using Wicket since 2006 and have never looked back. The community is alive and vibrant and any issues are usually fixed within a day.

The current release is version 1.5 although version 1.6 is under development. If you're starting out I'd stick to version 1.5 as that and version 1.4 are what most developers are using at the moment.

Here's the Wicket website

Here's an excellent piece on Wicket best practices

Java package name structure and organization - best practice and conventions

2012-06-07T15:30:00.000-07:00

When it comes to conventions regarding how to structure your packages in Java (or any other language in fact) there is a wide variety of opinion. Making the right decision can promote productivity, understandability, quality, flexibility and the construction of reusable frameworks.

I've seen some people promote 'package by feature' over 'package by layer' but I've used quite a few approaches and found 'package by layer' much better than 'package by feature' but further from that I have found that a hybrid: 'package by layer then feature' strategy works extremely well in practice and here's why I think that's so.

Package by layer then feature

The 'package by feature' people tend to want to put all user interface, model classes, persistence classes (eg., DAO - if you're into self torture ;]) for a particular feature into the same package and according to many examples, the same directory. i.e. they don't use a different directory with the same package as is excellent practice for test source code because the test code can then access package level items to test them.
They then claim that their 'separation of concerns' comes from having separate classes in the same package. Huh? That makes no sense whatsoever - 'separate' in my book means completely separate - like get them the hell away from each other already!

'Plug and Play' Layer implementations

Let's say I design and implement a really sophisticated object model and I use a particular Java Web UI framework (say some popular but old fashioned one) to build a web app around this model. Consequences are:

'Package by feature' places my UI code in the same package as my model code.
'Package by layer' places my UI code in a separate package to my model code.

Now let's say I want to change to a more modern component based OO web UI framework like Wicket. Consequences are:

'Package by layer' lets you start to build up a separate package for the Wicket UI classes. When it's working you can simply delete the old UI package and use the Wicket package instead - all the time nothing in the model package ever needs to get touched. This actually happened to one of our projects and it worked like a treat. We even had both frameworks in use in production at the same time as we gradually replaced screens using the old UI framework with Wicket screens.
With 'package by feature' this would have been an unholy mess - both JSP and Wicket classes in the same package/directory.

'package by layer' makes it easy to build reusable frameworks/libraries

Another advantage of 'package by layer' is that we can build shared libraries/frameworks that require both model and UI and we can distribute different versions of the framework with the same model code but with different UI libary implementations. You can't do this easily if you've followed 'package by feature' because all of your UI code for the different UI frameworks ends up in the same package/directory - ugly!

Ugly breakdown of encapsulation and decoupling with 'package by feature'

There's one thing about 'package by feature' that is promoted as a feature but is IMHO a complete breakdown of everything you've ever learned. Using the same package for say model and test code is great for test code because it gives that code privileged access to test items with package level access in your model code which would be impossible otherwise. The very reason that those items were made package level was to enforce separation of concerns and reduce coupling. The developer of those package level items in the model wanted to allow access by other model objects in the same package but DENY access to other classes outside the package - like UI classes!
'Package by feature' places UI items directly in the same package as model objects meaning you'll be throwing away any language supported enforcement of decoupling that was placed in the model by the designer. You just can't promote a packaging scheme that breaks down language enforced decoupling and be taken seriously.

Naive assumptions about features and layers

'Package by feature' assumes that the UI code for a feature has a nice one-to-one relationship with the model code for that feature. This goes against all of my experiences. In most apps a particular UI element pulls in various features of the model. Often there are 'core' model objects that are used by just about every UI element. Which 'feature' package do you place those UI elements in?

Summary of suggested approach

So IMHO the best approach is a hybrid 'package by layer' and then, after you've split things up by layer, then the next level in your package splits them up by feature. I also use a slightly different approach for reusable 'frameworks' or 'libraries' than the one I use for apps.

The format looks something like this:

revdomain.moduleType.moduleName.layer.[layerImpl].feature.subfeatureN.subfeatureN+1...

Where:
revdomain Reverse domain e.g. com.mycompany
moduleType [app*|framework|util]
moduleName e.g. myAppName if module type is an app or 'finance' if its an accounting framework
layer [model|ui|persistence|security etc.,]
layerImpl eg., wicket, jsp, jpa, jdo, hibernate (Note: model layer has only single implementation so leave out for model layer)
feature eg., finance
subfeatureN eg., accounting
subfeatureN+1 eg., depreciation

*Sometimes 'app' left out if moduleType is an application but putting it in there makes the package structure consistent across all module types.

eg., framework where some UI elements require access to common model objects in feature 1 but UI elements for feature2 model objects have very specific feature 2 references

 com.mycompany.framework.myFrameworkA.model.feature1
 com.mycompany.framework.myFrameworkA.model.feature2
 com.mycompany.framework.myFrameworkA.ui.wicket
 com.mycompany.framework.myFrameworkA.ui.wicket.feature2

If you decide to make your framework available for multiple UI libaries you would add separate packages for each of these and ship each with the shared model code.

com.mycompany.framework.myFrameworkA.ui.jsp
com.mycompany.framework.myFrameworkA.ui.jsp.feature2

com.mycompany.framework.myFrameworkA.ui.jsf
com.mycompany.framework.myFrameworkA.ui.jsf.feature2

I usually rename 'framework' with 'util' if it's more of a basic utility more than a framework with its own model and UI.
If you write code using a persistence technology that supports transparent persistence (e.g. an ORM) then your model code can be written in a fairly ORM agnostic way but you'll put supporting, ORM specific repository/service classes (or DAO classes if your into that) in a separate package for the persistence layer
e.g.

com.mycompany.framework.myFrameworkA.persistence.jdo.feature1
com.mycompany.framework.myFrameworkA.persistence.jdo.feature2

com.mycompany.framework.myFrameworkA.persistence.jpa.feature1
com.mycompany.framework.myFrameworkA.persistence.jpa.feature2

eg., for apps I drop the 'framework' and follow mycompany with the app name:

com.mycompany.app.myApp.model.feature1
com.mycompany.app.myApp.model.feature2
com.mycompany.app.myApp.persistence.jpa.feature2
com.mycompany.app.myApp.ui.wicket
com.mycompany.app.myApp.ui.wicket.feature2

Modal X 1.3 for Wicket released

2011-08-04T01:57:00.000-07:00

We have recently released a new version of the open source Modal window extension framework for Wicket called ModalX. We have also added an getting started guide, a live demo and an example application that demonstrates its main features.

Feel free to leave any comments about the library here.

Take a look at the ModalX home page here:
ModalX Home Page

See the live demo here:
ModalX Live Demo

Wicket DateTimeField - trick for young players

2010-11-24T19:59:00.000-08:00

If, like me, you tried to use the DateTimeField component in the Wicket extensions library and saw an extra spurious TextField appear in your form and you didn't know why... then read on:

It turns out that the DateTimeField is actually rendered as a Panel by Wicket. Being a field in a Form I naturally (according to what I saw in some online example code) just entered markup within my form like this:

<input type="text" id="startDate">

but of course that can't work. Wicket can't render a panel into an input element (well not with any success at least!).

It turns out that as a Panel the DateTimeField is best entered as a <div> in your markup:

<div id="startDate" />

That allows the DateTimeField to be rendered correctly. Yeehah!

Once again I want to congratulate the Wicket team on the world's best object oriented, component based, Java User Interface framework.... a big claim I know, but try it and I think you'll agree!

Feel like a Pizza?

2010-08-31T23:53:00.001-07:00

Try Gabriel's Pizza and Ribs, using the munchopia online ordering service:

Click here to: Take a look

2010-08-26T19:00:00.001-07:00

Want to save the planet by saving on your power consumption?

Then you need to install LED lighting in your home or business:

EO LED

New website for Step Ahead Software Development

2010-06-09T16:37:00.000-07:00

The software development section of Step Ahead Software now has a new website. It's at:

http://develop.stepaheadsoftware.com

If you need any software or web development let us know. We have been providing outsourced software development since 1994 and have a large range of satisfied companes from small businesses to large corporations like Canon and Lucent.

We specialize in development in the following fields:

Windows Desktop
Web application development
Enterprise website application development

We are experts in the following languages, frameworks and databases:

Java, JDO, JPA, Wicket, Echo, JSP, J2EE, DataNucleus, Hibernate
C++, MFC, OWL, .Net, GDI, GDI+
MySQL, Oracle, MS SQL Server

We also have 'off the shelf' modules for fast 'time to market' of a variety of projects:

Entity management framework: For extremely fast implementation of websites or desktop applications that require users, people, groups, organizations, authentication, products, etc.,
Visual editing framework for creating applications that allow users to manage a visually oriented view of the topology of a particular problem space.

Step Ahead: Authentication in the Matrix!

2008-11-20T09:10:00.000-08:00

A little while back one of our customers, Matrix Education, expressed a desire to improve the security and protection of their rather sophisticated intranet based enterprise management software. To this aim we discussed integrating our entity management system into their web based intranet application.

Our entity management system is a powerful role based system that allows the management and authentication of entities including users, actors, organizations etc., It comes with a powerful, intuitive web based user interface to allow easy creation and management of entities and such as users and their roles and 'access rights'. Our separate workflow management system plugs right into the entity management system for customers who want to manage processes within their organization efficiently and automatically.

The customer understood that integrating our well established 'off the shelf' user management and authentication framework would provide a more reliable, high quality and less expensive alternative than building such a system from scratch.

Yesterday, after only a few days effort we completed the integration of our entity management framework into their system so now their web based enterprise system has modern, powerful user authentication built seemlessly into their existing JSP based application. The use of our access managing servlet filter allows us to hook into the 'smarts' of our entity management framework from just about any Java based web application including those built with servlets, JSP, Tapestry, Wicket and even Echo.

pagebloom's Media Asset Manager shines!

2008-10-29T05:58:00.000-07:00

Motivation
About this time last year ago we conceived an idea to enhance our pagebloom enterprise content management system (ECMS) to help manage the media assets (images, Flash intros etc.,) for the websites that it hosts. The idea was to create an advanced and powerful auto replicating media asset management system that could serve images from the most appropriate media host chosen from a range of available media hosts located around the world. That system was implemented and integrated with the pagebloom hosting service earlier this year.

How it works
If a particular host goes down or is under high traffic then the media assets can be served from alternative media hosts - all without site visitors experiencing any negative affects from the loss of performance on the degraded or lost media server.

Concept payback!
Since the integration of the new media asset manager none of the media hosts have experienced any problems, luckily. Well that was until today! And pagebloom carried forth without a problem. The US based media server that went down was removed from the list of available servers and media assets for pagebloom hosted sites were transparently retrieved from alternative media hosts - all without site visitors noticing any problems. From the perspective of a software engineer it truly was a thing of beauty! We're all very satisfied and proud at how well our design was able to keep the pagebloom ECMS working perfectly even though a key component of the system was not available for a period of time.

Welcome to the Step Ahead Software blog

2008-10-29T05:53:00.001-07:00

This blog is where we'll keep you up to date with all the news and happenings, product releases and product updates of Step Ahead Software.

I'm your host, Chris, so sit back, relax an enjoy the journey!