Enter RubyRep, a Ruby-based replicator that already supports both Postgres and MySQL and has plans to also support Microsoft SQL Server, Oracle and IBM. The author has published a screencast where he enables and demonstrates working replication in less than 5 minutes.

It handles composite primary keys, single primary keys, sequences/uniqueids and even tables without primary keys with a touch of configuration. It works using triggers which it installs and maintains automatically. There are some limitations with MySQL since it only supports one trigger per table which could collide with an existing application trigger. We’ll have to see what performance is like but it’s pretty compelling so far.

Postgres 8.4 RC1 is out too - might be a good opportunity to roll out a test environment and start playing.

Incorporating Codex as part of your code

My preferred method for including a third-party code base is to use Subversion’s svn:externals. Specifically I have been using a “local external” repository which gives me the benefits of svn:external without the deploy-time dependency on a remote repository that may not be available.

Thus, in my application, I have an svn:external that points like so:

http://myrepo/externals/codex/tags/trunk_r298 help

Where the codex/tags/trunk_r298 folder is simply a copy of the Codex trunk at revision 298. When I run svn update on my application, it extracts the Codex codebase into a /help directory. All’s great right? If you don’t want to modify the look and feel, then you’re done.

The problem with modifying Codex

Most people however are going to want to customize the layout and CSS at a minimum so you’ll need to modify the layout and view CFM files. That’s pretty straightforward but what happens when Codex releases an update? And what about the configuration file - where will you put that? What a train wreck - now you need to merge your files into theirs and hope there are no conflicts.

To embed Codex (or really any ColdBox application) into your app, you’ll need to adjust three things:

1. Move the coldbox.xml.cfm file to a location outside of the Codex tree
2. Relocate the view files
3. Relocate the layout files

A Better Way

This may be common knowledge to Coldbox developers but I stumbled upon it through trial and error. There is a way to externalize your configuration and your views such that your external views only serve as an override rather than the default. This lets you take advantage of updates that come from the Codex team while keeping your modification footprint as small as possible. Let’s start with a partial directory tree where Codex has been checked out into the /help directory and my Model-Glue application has its own /views directory structure:

/
+ gfx
+ help (this is Codex, checked out via svn:external)
    + config
    + handlers
    + ...
    + layouts
    + model
    + plugins
    + views
    Application.cfc
+ controllers
+ views
    + help
         + views
             + tags
             footer.cfm
             header.cfm
             ...
         + layouts
             Layout.html.cfm
             Layout.Main.cfm
             ...
Application.cfc
index.cfm 

Each ColdBox application has a configuration section called “Conventions” that are optional adjustments to the defaults. The general strategy is to turn Codex back on itself by making it look for the default in your external folder and then look for the override in the standard Codex folder. In this manner, if you have a file in your own views folder then it will be used, if not, it will go searching for the default file under the Codex directory tree. You’ll need to add this section to your Codex config file:

<Conventions>
	<layoutsLocation>../views/help/layouts</layoutsLocation>
	<viewsLocation>../views/help/views</viewsLocation>
</Conventions>

Conventions will work with a relative path so I am pointing my default views to be in my Model-Glue views folder instead of the Codex directory. Codex (or more specifically, Coldbox) will first look in my /views/help/* directories instead of /help/layouts and /help/views.

Perfect! Now my layout and views are external so I can manage them independently of Codex upgrades. But, what if I only want to make small changes, like put my logo in the header? It would be a bummer to manage all of those files for such a simple change! Thankfully Coldbox has a Setting called ViewsExternalLocation:

<Setting name="ViewsExternalLocation" value="views" />

Put alongside the other Settings in your configuration file, this tells Codex to override any default views with those found in the standard Codex views directory. At the time of this writing with Coldbox 2.6, there is no corresponding LayoutsExternalLocation but Luis Majano said it would be coming in 3.0. In the mean time, you will need to include all of the 5 or 6 layout files in your custom layout folder but you only need to include the views you wish to override or add.

You can verify this works by creating a single file like (using my directory structure) /views/help/views/wiki/show.cfm (which is used to display a wiki page) and adding some extra text to it. Reinitialize Codex, view a wiki page and you should see your changes.

Moving the Configuration File

Since Coldbox is all about conventions, you must override the default to point at a configuration file in a different location and this will take some editing. This is one place where you will need to make one small change each time you update Codex or script it for your deployment using something like Ant. Automating your deployment is a good idea if you’re not already doing it.

The /help/Application.cfc defines where the configuration file lives. Each time you update Codex (either via svn:external or unzipping of a file), you’ll need to modify a single line like so (or, see the Coldbox docs):

<cfset COLDBOX_CONFIG_FILE = "../myconfig/codex.xml.cfm">

In this case, I’m using an example directory of /myconfig with a copy of the configuration file that I renamed from coldbox.xml.cfm to codex.xml.cfm. Move your file, save the Application.cfc and reinitialize and you’ll now be running off the XML file managed by your own version control rather than the one provided in the standard Codex distribution.

That’s it, you’re ready to modify the look, feel and configuration of your Codex wiki (or really any other ColdBox app) without resorting to modifying the actual distribution. This will make future upgrades much easier and less fragile which will encourage you to do them more frequently.

Bonus Tip for Automating Application.cfc Edit

If you’re using Ant, you can automate the replacement of the Application.cfc file on every deployment using the following task:

<replace file="/help/Application.cfc">
	<replacefilter token="<cfset COLDBOX_CONFIG_FILE = \"\">" value="<cfset COLDBOX_CONFIG_FILE = \"../myconfig/codex.xml.cfm\">" />
</replace>

Now you can update with impunity and your Ant script will automatically fix the configuration file pointer during deployment.

Amazon EBS Volumes

• $0.10 per GB-month of provisioned storage
• $0.10 per 1 million I/O requests

Well, I’d like to know how much it’s going to cost to run my database before I move it over and I can’t find squat on the Intarweb on how to estimate this cost. But assuming you’re running Linux, there is an easy way by looking at /proc/diskstats:

[brian@db2 ~]$ cat /proc/diskstats
 104    0 cciss/c0d0 5946064 12358605 1783920022 267694420 26411136 296537705 2583591570 34712288540 53639757 3738914277
 104    1 cciss/c0d0p1 553 2156 2 10
 104    2 cciss/c0d0p2 194565 1556520 192647 1541176
 104    3 cciss/c0d0p3 18109735 1782360706 322755967 2582047424
 104   16 cciss/c0d1 6579320 1005537 566670186 27193242 13685423 33336126 376172656 26751430 023642414 53937189
 104   17 cciss/c0d1p1 7584779 566669554 47021570 376172552

This is a device with hardware RAID so we want to ignore the individual partitions (c0d1p* for example) and look at the disks c0d0 and c0d1:

cciss/c0d0 5946064 12358605 1783920022 267694420 26411136 296537705 2583591570 34712288540 53639757 3738914277
cciss/c0d1 6579320 1005537 566670186 27193242 13685423 33336126 376172656 26751430 0 23642414 53937189

According to these directions on reading the numbers, you can add the first and fifth numbers of that row to get total reads and writes respectively since the system was started. To calculate your total I/O requests per month (what Amazon will charge you), just plug your numbers into this formula:

Cost = $0.10 * 30 days * ((first number) + (fifth number)) / (days of uptime)

You can get this number another way by using iostat as well which will give you the average number of transactions per second since boot:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0        1.56        86.17       124.80 1783920022 2583601386
cciss/c0d1        0.98        27.37        18.17  566670378  376184592

If you take the average TPS for your disks and multiply them out, you get roughly the same number (good enough to make a cost estimate for EC2):

Cost = (TPS of all disks) * 60s * 60m * 24h * 30d

You can see this particular box is under very light load with only 6,583,680 I/O requests per month. With EBS that would cost about $0.65. I think we can afford that.

• Look at replacing my validate() methods with Bob Silverberg’s ValidateThis framework which has centralized validation rules. One thing I’ve struggled with in centralized validation is how to handle the situation where a user and an admin have differing validation requirements depending on who is doing the editing. Bob solves this with configurable “contexts” which look promising.
• There are a few places in my system where I am working with collections of ~100-300 objects and using Transfer to manipulate these can be slow. Peter Bell’s Iterating Business Object, which is simply a fancy wrapper for a query, is a performant way of working with collections of this size. The fact that we, in CF land, have to worry about this makes object oriented design a compromise but that’s the way it goes. It’s time to evaluate this as an option.
• Refactor to Abstract Classes. Bob Silverberg gave a presentation on building an object oriented model which included discussion of Abstract Classes. I have used one as the basis for my Transfer decorators for some time which defines a generic populate() routine (influenced by Bob in the past) but it’s time to look at these for the remainder of my objects in an effort to reduce my total code footprint and centralize the similar code. Because I refactored from a query-based framework, my gateways and services leveraged a lot of that code and I wound up with gateways that have their own queries and services that repeatedly proxy calls to those gateways along with calls to Transfer for persisting my objects. One of the things I mentioned in my frameworks migration is starting with my original code may have been a limitation in my migration and the kind of blind cut-and-paste that is time efficient but not particularly elegant is one of those places. Now it’s time to go back and optimize.
• ColdFusion 9 CFC performance is supposed to be substantially better - I have the beta so I need to do some head-to-head tests of Transfer on CF8 vs. CF9 as well as Transfer on CF9 vs. the built in Hibernate support. Gains here may make an IBO unnecessary.
• Get my Trac install updated. I’m still running some ridiculous 0.9.5 version while the latest is up to 0.11.4. Getting 0.9.x Trac to install and run was such a dependency-based nightmare that I swore off touching it but that was 3+ years ago and I’ve got a new server being built to replace the hardware anyways. This older version is not only compromising the feature set but it’s also preventing me from updating my Eclipse setup to take advantage of Mylyn so I can work more from Eclipse against my outstanding tickets.
• I sketched out the definition for a reporting overhaul we’ve been talking about for a year. I already have a pretty sophisticated CF-based reporting system that emulates a lot of what you would get out of a Crystal Report or CFREPORT but does it without requiring the end user to install a report builder tool or have any knowledge beyond the data they’re working with. I’m interested in exploring BIRT further but the user interface for generating reports must be user-friendly. Actuate’s version of BIRT has a very nice UI but costs some dollars.
• The Writing Testable Code: Real-World TDD session by Marc Esher was a great kick in the butt to improve my unit testing. The way he took some very reasonable code and demonstrated how to refactor it to be more testable (and thus, more reliable) was eye opening despite the fact that it was simple. Sometimes the best ninja tricks are. And, he finally explained concisely how to use a mock object in testing that I grokked so it’s time to implement some of these ideas.

Presentation Slide decks

As promised, I’m publishing the slides from my presentations for any attendees who want to grab the details. Thanks for coming and be sure to fill out the CFO survey. The slides, particularly the Postgres preso, were bullet-point heavy which is not my favorite style of presentation (and when compared to say, a Marc Esher presentation, is a bit embarrassing) but when you are walking people through the features of a package, I consider the feature-list-with-commentary to be the equivalent of showing-code-with-commentary. You don’t use pictures to represent your code, now do you? The framework migration slides include my recommendations for surviving a migration of a code base but spare the reader my personal pain and suffering from my launch last year. You only get the actionable stuff and none of the emotional scarring! Don’t forget: build a functional equivalent at all costs!

Move over MySQL, Make Room for PostgreSQL
ColdFusion and MySQL have been best friends since MX and many developers use it as their day-to-day database. Brian made the leap to PostgreSQL in 2004 and never looked back: it has everything MySQL has plus most of the enterprise features found in Oracle, SQL Server and DB2 wrapped in an open source package with a great community and support. Come learn how these packages are more similar than different and why your next project should be backed by Postgres.

Download Slides (PDF)

Planning your migration to a Framework
Still waiting to make the jump to an object-oriented framework like Model-Glue, Mach-II or Coldbox? As the sole developer and survivor of a seven-month migration from a home-grown framework to Model-Glue, Coldspring and Transfer, Brian will share four key areas where unexpected complexity resulted in catastrophe and how you can avoid them in your own applications. This is your survival guide for moving code to a new framework.

Download Slides (PDF)

If you have any questions or feedback about my presentations, feel free to hit me up here.

As it turns out, the road took a fork and I found he had become a citizen and with a copy of a declaration of intent to become a citizen, I was able to file my paperwork and receive my dual citizenship and passport last year. Here’s what the letter says:

This is in respnose to your Freedom of Information Act/Privacy Act (FOIA/PA) request received in this office March 4, 2008 regarding Prospero Ghidinelli.

We have completed our search for records that are responsive to your request. The record consists of 4 pages of material and we have determined to release it in full. The enclosed record consists of the best reproducible copies available.

Please be advised that the National Records Center does not process petitions, applications or any other type of benefit under the Immigration and Nationality Act. If you have questions or wish to submit documentation relating to a matter pending with the bureau, you must address these issues with your nearest District Office.

If you should have any additional questions about your request, please direct your inquiries to this office at the above address [etc]

I originally filed in late 2006 or early 2007 but my application was denied for being incorrect in some way so I had to refile and thus the 2008 date above. By the time I refiled, I already knew he had become a citizen but I sent it off anyways for completeness. The packet includes four photocopies:

• Department of Labor Form 124A, issued 7/28/1919
• Certificate of Naturalization
• Petition for Naturalization
• Declaration of Intention

The copies are not great but of interest is the Declaration of Intent had his name as Dante (what everyone knew him as and what’s on his tombstone in Italy) and the other documents all have his legal name, Prospero. Clearly someone informed him to get his act together.

I have received a few emails from people who are going through the process themselves and are seeking help but otherwise I haven’t thought much about this lately. I had a busy few months of work and then getting married last month and this was a fun surprise to receive in the mail. I will admit that I had a flash of panic when I first saw it, thinking somehow I had done something wrong, but alls well that ends well!

I’m still itching to use my passport for the first time…

Why would you expect someone to provide a higher level of availability and more features at a cost lower than you can do it yourself?

You can argue about economies of scale but fundamentally it’s on the mark. Amazon (and the other grid/cloud providers) have to pay for the same hardware and infrastructure that we do except they have even more strenuous requirements. And since they’re not a charity, there has to be some profit in there, too.

But many of the reasons to wait on cloud computing, at least the Amazon variety, changed today with the announcement of three new services:

• CloudWatch - Web services-based resource and performance monitoring of EC2 services at $0.015/hr or $10.80/month.
• Auto Scaling - A key feature of add-on services like RightScale, Auto scaling will add or remove server instances on the fly. Most web sites experience some amount of seasonality whether it be time of day, day of week or month of year and this will reduce costs by scaling up only as needed. It’s included as part of CloudWatch.
• Elastic Load Balancing - Yes! Instead of setting up an instance and running keepalived or haproxy to route traffic between multiple servers, Amazon now has load balancing at $0.025/hr or $18/month and it can span multiple availability zones (~physical data centers). You also pay $0.008/GB of data transfer handled by the load balancer. 100GB/month would cost you $0.80.

Why is this news?

It means EC2 just got cheaper as you scale up.

For my company, we run a two-node cluster primarily for high-availability. To actually keep this highly available, we also have to run a pair of load balancers managed by keepalived. This works quite well but it’s 4 separate boxes with power supplies and disks consuming power and taking up rack space. Those are all the downsides to having your own rack space. At least in California, power is the most expensive part of physical hosting and also the biggest restriction. It took a fairly exhaustive search to find a datacenter where we could get 30A into a single cabinet.

If we had our current architecture in EC2 today, my interpretation is we could eliminate our two load balancers to save $144/month (small instances are ~$72/month) and replace them with $11 worth of Elastic Load Balancing. If we added monitoring, we should be able to add web servers as demand dictates for additional savings by not having a second web server running 24/7.

Most smaller web companies want five nines but can’t afford it nor manage it. Historically we run our service just short of 99.99% uptime which is fairly expensive and difficult to achieve but provides a scant 3.5 hours of downtime per year. Most companies can tolerate this. These new features of EC2 are making it inexpensive to take advantage of EC2s promise: on-demand scalability with high-availability.

How to save money with EC2

When I agreed it was insane to expect someone else to do something for you for less money, it was if you compare apples to apples. Everyone has a base architecture that has something like a database server, web server and mail server. You simply cannot expect to save much, if any, money in your initial configuration. The only meaningful way to save money with cloud computing is to reduce the incremental step in costs as you add capacity. If a colocated solution offers you additional 10 units of web server performance for $1000/month and you can buy the same performance in 1-unit increments for $100/month, then you save money anywhere between 10 and 20 units. Virtualization and auto-scaling make it possible to reduce your investment to the minimum required capacity on a minute-by-minute basis to drive the most value from cloud hosting.

I think it’s time to start running some tests…

Working with Mark Mazelin over the better part of a year, we’ve created a developer-friendly set of components with a simple API. We’ve made it easy to create new gateways on top of our core and most importantly, we’ve battle-tested this code processing over $1 Million USD using these components.

Whether you are an OO nut or a spaghetti coder, our simple interface makes processing credit cards and e-checks (EFTs) easy like:

<cfset config = {path = "bogus.gateway", username = "user", password = "foo" } />
<cfset core = createObject("component", "cfpayment.api.core").init(config) />
<cfset gateway = core.getGateway() />

<cfset cc = core.createCreditCard() />
<cfset cc.setFirstName("Joe").setLastName("Blow").setAddress("888 Nowhere Lane").setPostalCode("94103") />
<cfset money = core.createMoney(5000) />< !-- $50.00 in cents -->

<cfset response = gateway.purchase(money, creditcard) />

<cfdump var="#response.getParsedResult()#" />

I’ve put more than five years of e-commerce experience into the design and reliability of this library and Mark has helped me make it better every step of the way.

I’ll be at cf.Objective() in Minneapolis through Sunday so please don’t hesitate to come find me and talk shop.

1. Free for Education and Students - Adobe is giving away free full-blown ColdFusion Enterprise licenses for schools and students. They are also providing curriculum. Not shelling out $7500 to learn ColdFusion means the long-term future of ColdFusion developers is looking a little brighter. It’s available at FreeRIATools.com
2. CFaaS - ColdFusion As A Service. The underlying out-of-the-box functionality that CF provides like Mail, Charting, Documents, Database interaction, etc. is being opened up to SOAP web services and AMF for Flash/Flex. This will eliminate building a middle tier for RIAs that wrap ColdFusion tags and it opens up ColdFusion as a RAD service layer for any technology that can speak SOAP like .Net, Perl, Ruby, PHP, etc.
3. Object Relational Mapper - ORM. This has been said before so isn’t new but if you haven’t heard, CF9 will have ORM support via Hibernate baked in with a lot of stupid easy syntax that will eliminate the need to write SQL. I’m a Transfer user today but am watching this carefully.

As a developer, there were a handful of things that are going to be great for day-to-day development:

• Server.cfc and onServerStart(). The ability to run code when the server first starts will eliminate the dreaded first hit for any application with a lengthy initialization process (that’s you, frameworks!). This isn’t a game changer, but it sure makes restarting a server more consistent.
• Implicit struct/arrays can be used everywhere. The gross speed advantage of defining a struct or array in a Javascript-like syntax with:
  myArray = ["Brian", "Joe", "Frank"];
  was less helpful because you couldn’t create it as part of a method call like so:
  methodThatTakesAStructure({name = "Brian", age = "32"});
  That code would fail in CF8 but will work in CF9. This makes the syntax more consistent with other languages and more natural to use which is the kind of sugar that makes day to day development a little easier.
• Assignment chaining! Good god, more than a decade later we can finally say:
  a = b = c = 0;
• Direct access to the elements of a returned array. This wouldn’t work in CF8 but will in CF9:
  <cfoutput>#getSomeArray()[2]#
  This makes working with dynamically returned arrays a little easier by avoiding an intermediate assignment.
• ColdFusion Components have a new automatically defined LOCAL scope (the equivalent of a CFMs “variables” scope). So you don’t need to var every variable if you use the local scope instead:
  <cfset local.myPrivateVariable = "foo" />
  The other part of this is that you can now put a var’d variable anywhere in your method declaration, no longer just at the top! Another syntax tweak that will make day to day development a little nicer and will make reading code a little easier.

There was a lot of stuff that Adam and Terry covered, including a cool demo of the new CF IDE code named Bolt, but there were two specific things that I am looking forward to:

• Multi-server Server Manager. Like the multi-server server monitor in CF8, the Server Manager will let you administer multiple servers from a single interface. For anyone who has to admin even two servers, there are two bitching features: apply changes (e.g., new datasource) to more than one server at a time and compare the settings of any two servers with one click. The latter will help track down why one server is performing exceptionally well or poorly.
• Object Cache - Adobe is going to ship a built-in, pluggable caching framework. New methods cacheGet(), cachePut() and cacheGetMetaData() combined with fragment caching should eliminate a lot of the homegrown caching frameworks. Just like CFQUERY caching will look at the variables passed in and determine if the cached results are still applicable, these tools will do the same and are going to work with CFCs.

More to come… my presentations are tomorrow on PostgreSQL and Saturday on Migrating to a Framework.

Personally, I’m most excited about Bean injection, copyToScope() and Typed Events. Those three things will make me faster and less error prone.