I found out not long ago that my good friend and college housemate Jerry Tam had made the cut and would be appearing on Season 5 of Bravo TV’s Project Runway. I’m currently in Europe right now so I went looking to find some details about the season premiere last night. Knowing Jerry as well as I do, he was either going to win it all or get kicked off the first episode and unfortunately it was the latter.

Tam has an incredible amount of talent and vision wound up into a 150 beats-per-minute kind of frenetic energy. This is a guy who went from growing up in Montana bussing tables in his dad’s Chinese Restaurant to start his own fashion label in Manhattan, FORM. That was not the shortest path between two points.

People think succeeding in business just requires a great idea but ideas don’t blossom on their own. Success takes hard work, long nights, persistence and an undying conviction that you can do it better than everyone else. While he may have lost out on Project Runway, I know Jerry has those qualities required to succeed.

You’re still 1 of 16 buddy and we’re all proud of you here on the left coast!

<cfinvoke component="cfide.adminapi.administrator" method="login">
	<cfinvokeargument name="adminPassword" value="..." />
</cfinvoke>

<cfinvoke component="cfide.adminapi.runtime" method="clearTrustedCache" />

Since updating to CF8 on my Windows XP laptop, running this code as either part of my reinit process or in a standalone template results in the following coldfusion.security.SecurityManager UnauthenticatedCredentialsException error:

The error occurred in runtime.cfc: line 656
-1 : Unable to display error's location in a CFML template.

Stack Trace
at
cfruntime2ecfc1355728568$funcCLEARTRUSTEDCACHE.runFunction(E:\cf8_updates\cfusion\wwwroot\CFIDE\adminapi\runtime.cfc:656)
at
cfApplication2ecfc221588290$funcONREQUESTSTART.runFunction(Application.cfc:85)

coldfusion.security.SecurityManager$UnauthenticatedCredentialsException
	at
coldfusion.security.SecurityManager.authenticateAdmin(SecurityManager.java:1704)
	at
coldfusion.runtime.RuntimeServiceImpl.clearTrustedCache(RuntimeServiceImpl.java:1518)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at coldfusion.runtime.StructBean.invoke(StructBean.java:511)
	at coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2300)
	at
cfruntime2ecfc1355728568$funcCLEARTRUSTEDCACHE.runFunction(E:\cf8_updates\cfusion\wwwroot\CFIDE\adminapi\runtime.cfc:656)
	at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418)

Many people on the CFGURU list including Charlie Arehart and Cameron Childress made several suggestions of things to check but everything came back looking correct. I was prepared to chalk this up to a random issue with my installation until I asked others to try it on their CF8 install and was surprised by the results. While several people were unable to reproduce the error, we had confirmations of the same CF8 error on the following platforms:

• Enterprise 8.0.1 multi-server on Windows XP
• Standard 8.0.1 on Windows 2003
• Enterprise 8.0.1 multi-server on Mac OSX

There are some people on 8.0.1 who have it working successfully but with three different people experiencing the issue it doesn’t seem to be a one-off installation error. If I find a resolution I will post it here as a follow-up but at this point I am going to comment out that line in my reinit since on my laptop I don’t use the trusted cache in development.

I have some timing in my Application.cfc and index.cfm in my model-glue app that produced some interesting results different from my expectations. The difference is between reinitializing an application on a running CF8 server and starting up the CF8 server from scratch. I track the time it takes to load beans into Coldspring and for Model-glue to initialize with the following code in application.cfc:

<cfset cnt = getTickCount() />
<cfset application.cs.loadBeansFromXmlFile(expandPath("coldspring.xml"), true) />
<cflog file="application" type="information" text="Finished ColdSpring load in #(getTickCount()-cnt)/1000# seconds" />

And in index.cfm:

<cfif NOT structKeyExists(application, ModelGlue_APP_KEY)>
	<cfset booInit = true />
	<cfset cnt = getTickCount() />
<cfelse>
	<cfset booInit = false />
</cfif>

<cfinclude template="/ModelGlue/unity/ModelGlue.cfm" />
<cfif booInit>
	<cflog type="information" file="application" text="ModelGlue #ModelGlue_APP_KEY# initialized in #(getTickCount()-cnt)/1000# seconds" />
</cfif>

Now my expectation is that it would be slower on a cold start to load these applications than restart but the reverse is what I see by a factor of 2-1. My reinit routine performs the following:

• Clears session key
• Clears trusted cache via Admin API
• Clears Transfer cache
• Deletes Coldspring instance
• Recreates Coldspring instance (this is timed)
• Clears all Model-glue Application keys to force reload
• Reinitialized session
• Runs index.cfm to initialize Model-Glue (this is timed)

The times below are from starting CF8, loading my application home page, then immediately performing a warm reinit:

Anyone know why this would be the case? I would think the classes had already been loaded so a warm reinit should be faster than the cold start.

If this is true in general, then it’s better to restart the CF8 server when deploying code than running a reinit procedure.

<cfquery name="select" datasource="#variables.datasource.getName()#">
	SELECT *
	FROM roles
	WHERE id = '#id#'
</cfquery>

<cfloop query="select">

	<cftry>
		<cfquery name="update" datasource="#variables.datasource.getName()#">
			UPDATE roles
			SET id = '#target#'
			WHERE id = '#id#'
			AND role = '#role#'
		</cfquery>

		<cfcatch type="database">
			<cfquery name="delete" datasource="#variables.datasource.getName()#">
				DELETE FROM roles
				WHERE id = '#id#'
				AND role = '#role#'
			</cfquery>
		</cfcatch>
	</cftry>

</cfloop>

The problem is that when the UPDATE failed due to a duplicate key, the CFTRANSACTION aborted the rest of the routine. I received this error in the console:

Error Executing Database Query.
ERROR: current transaction is aborted, commands ignored until end of transaction block

SAVEPOINT Solution

Via Google, I learned about PostgreSQL SAVEPOINTs, which allow you to back up your transaction to some intermediate point and continue as if it had not happened. They were easy to implement:

<cfquery name="select" datasource="#variables.datasource.getName()#">
	SELECT *
	FROM roles
	WHERE id = '#id#'
</cfquery>

<cfloop query="select">

	<cftry>
		<cfquery name="update" datasource="#variables.datasource.getName()#">
			-- declare my savepoint
			SAVEPOINT troles;

			UPDATE roles
			SET id = '#target#'
			WHERE id = '#id#'
			AND role = '#role#';
		</cfquery>

		<cfcatch type="database">
			<cfquery name="delete" datasource="#variables.datasource.getName()#">
				-- a collision occurred, skip back to the savepoint and then delete
				ROLLBACK TO SAVEPOINT troles;

				DELETE FROM roles
				WHERE id = '#id#'
				AND role = '#role#';
			</cfquery>
		</cfcatch>
	</cftry>

</cfloop>

Note it requires a semi-colon to run both statements in a single CFQUERY. This says that if the CFCATCH runs, to skip back to where we were just before our UPDATE caused an index error (duplicate keys) and instead run the delete to eliminate the duplicate data (which in my business process here, is OK). Now collisions could be handled without breaking the top level transaction.

There are other ways to do this like use a SELECT query first to see if the collision is going to happen. In this process however, collisions are very rare but datasets can be large so for performance I decided it was “easier to ask for forgiveness than permission”.

I don’t know if this is required in other databases besides PostgreSQL but I believe save points are available with most vendors. My member merge process runs about 70 queries to join together two accounts and all of their data before it declares the operation complete. This a neat feature to help control that lengthy process.

<cftransaction>
acct1 = transfer.get("member", id);
acct2 = transfer.get("member", id2);
memberships = acct1.getMemberships();

for (ii = 0; ii < memberships.length; ii++)
{
  membergateway.mergeMemberships(memberships[ii], acct2);
}

membergateway.mergeAccounts(acct1, acct2);

if (success) cftransaction commit
else cftransaction rollback
</cftransaction>

Which resulted in the error:

Datasource convert verification failed.
The root cause was that: java.sql.SQLException: Usernames and Passwords for all the database tags within the cftransaction tag must be the same.

I was stumped! I don’t even use username/password in my CFQUERY blocks! Then I realized that Transfer did, and since it was between the CFTRANSACTION block, it was screwing up my manual queries. This is what my CFQUERY in my gateway normally looks like:

<cfquery name="select" datasource="#variables.datasource.getName()#">
	SELECT foo
	FROM sometable
	WHERE id = '#arguments.id#'
</cfquery>

Where variables.datasource is a bean provided to my Gateway by Coldspring as defined in my coldspring.xml:

<bean id="datasource" factory-bean="ormService" factory-method="getDatasource" singleton="true" />

This is the bean that Transfer uses and it contains the DSN, username and password. Even though my username and password are stored in the CF admin, I had to add them to each of my CFQUERYs that appeared inside of the CFTRANSACTION to make it play nicely with Transfer:

<cfquery name="select" datasource="#variables.datasource.getName()#" username="#variables.datasource.getUsername()#" password="#variables.datasource.getUsername()#">
	SELECT foo
	FROM sometable
	WHERE id = '#arguments.id#'
</cfquery>

After that change, the error messages disappeared. Also check my post on something else I learned today about CFTRANSACTION with SAVEPOINTs.

Building Forms in Pieces

Reuse is central to development and frameworks purport to aid in code reuse. In my application, there are users, organizers and superusers. Users can edit a subset of the overall data available in the database while organizers can edit almost everything. Superusers have a few additional fields at their disposal. The question is: how do you build views and events to be able to only define the form fields a single time? Oh, and don’t forget client-side Javascript validation or effects too.

Before Model-Glue

Before Model-Glue, I combined my CRUD into a single CFM file with some conditional handling. Here is a simple example of how my forms used to look. The ui:qform is a custom tag that outputs a SCRIPT tag and instantiates the qForms validation library. The form posted to itself and with a structKeyExists(FORM, “uidClass”) at the top, it would have handled server side validation and persistence. I really like this approach as it makes finding the right code quick and easy.

Note my conditional delete statement checking for URL.uidClass which was the convention indicating it was in “edit” mode and would show the delete link with a Javascript “are you sure…” confirmation. I don’t always use these checks but in some circumstances they make sense from a usability perspective. I also try to use the LABEL tag for accessibility and usability purposes whenever possible.

<form action="#cgi.script_name#" method="post" name="frm">

	<input type="hidden" name="uidClass" value="#uidClass#" />

	<table class="form">
	  <tr>
		<td class="labelrequired"><label for="vchClass">Name</label>:</td>
		<td><input type="text" name="vchClass" id="vchClass" value="#HTMLEditFormat(vchClass)#" size="25" maxlength="25" /></td>
	  </tr>
	  <tr>
		<td class="label"><label for="vchClassDescription">Description</label>:</td>
		<td><input type="text" name="vchClassDescription" id="vchClassDescription" value="#HTMLEditFormat(vchClassDescription)#" size="40" maxlength="255" /></td>
	  </tr>
	  <tr>
		<td></td>
		<td>
			<input type="submit" value="Save Class" />
			<cfif structKeyExists(URL, "uidClass")>< a href="#cgi.script_name#?uidDelete=#uidClass#" class="delete" onclick="javascript:return confirm('Are you really, really sure you want to delete this class?')">delete this class</cfif>
		</td>
	  </tr>
	</table>
</form>

<ui:qform name="frm">
	// give better descriptions
	objForm.vchClass.description = "Class name";

	// make these fields required
	objForm.required("vchClass");
</ui:qform>

Post Model-Glue Solution

In my prior system, I would sometimes abstract out everything between the FORM tags into a custom tag so that I could re-use the form in multiple contexts. This worked OK and was an approach I could have used again under Model-Glue but I wanted to try and find something more “Model-Gluey”. Specifically, I wanted to accommodate the following requirements:

• Use the same HTML form for user, organizers and superuser editing, some of which may have different fields visible or editable
• Forms must post to different events in each scenario
• Validation rules (client and server-side) may be different or even optional
• Not all instances may permit deleting the object

Each unique use of the form includes its own form tag. This is so I can pass the exit action in from model-glue.xml. The XML configuration is pretty simple:

<event-handler name="vehicle">
	<broadcasts>
		<message name="needVehicleMakes" />
		<message name="needVehicle" />
	</broadcasts>
	<results>
		<result do="view.render" />
	</results>
	<views>
		<include name="formVehicle" template="shared/vehicle/frm.vehicle.cfm" />
		<include name="validateVehicle" template="shared/vehicle/val.vehicle.cfm" />
		<include name="main" template="staff/members/edt.vehicle.cfm">
			<value name="xe.process" value="vehicle.do" />
			<value name="xe.delete" value="vehicle.delete" />
		</include>
	</views>
</event-handler>

The val.vehicle.cfm and frm.vehicle.cfm stay the same while the edt.vehicle.cfm changes from context to context. The edt.vehicle.cfm “master” view looks something like this:

<cfset myself = viewState.getValue("myself") />
<cfset vehicle = viewState.getValue("objVehicle") />
<cfset xe.process = viewState.getValue("xe.process") />
<cfset xe.delete = viewState.getValue("xe.delete") />
<cfset frmName = viewState.getValue("frmName", "frmVehicle") />

<form action="#myself##xe.process#/uidVehicle/#vehicle.getUidVehicle()#" method="post" name="#frmName#">

	< !-- bring in the HTML form, which uses its own <table> -->
	#viewCollection.getView("formVehicle")#

	<div class="submit">
		<input type="submit" value="Save Vehicle" />
		<cfif vehicle.getIsPersisted()>< a href="#myself##xe.delete#/uidVehicle/#vehicle.getUidVehicle()#" class="delete" onclick="javascript:return confirm('Are you really, really sure you want to delete this vehicle ">delete this vehicle</cfif>
	</div>

</form>

< !-- bring in the validation rules (if any) -->
#viewCollection.getView("validateVehicle")#

You can see how I’m bringing in a lot of parameters from my model-glue.xml file to make it configurable without modifying the HTML making it as reusable as possible. The frm.vehicle.cfm and val.vehicle.cfm look like what you would expect, but by breaking them out into different files I can apply the appropriate validation to each scenario: admins have fewer required fields than end-users, for example.

That’s just client-side. Obviously we need to address the server-side too. If my object has different levels of validation (say, full and basic), I structure my .validate() method something like this:

<cffunction name="validate" access="public" returntype="array" output="false">
	<cfreturn validateBasic() />
</cffunction>

<cffunction name="validateBasic" access="public" returntype="array" output="false">
	...
</cffunction>

<cffunction name="validateFull" access="public" returntype="array" output="false">
	<cfset var errors = validateBasic() />
	...
</cffunction>

The controller calls the appropriate method based on context.

Bonus Advantages

What’s not so obvious is this also gives me the ability to modify what fields are available depending on context. There are several possible approaches for this including some kind of security context check wrapped in a CFIF to determine whether or not to show certain fields but there are scenarios, like sensitive financial details, where the more paranoid might opt to not include those fields in the view to prevent any possible data leakage. Both approaches are valid and in fact I use both which is why I like this method - it’s flexible!

Here’s an example view how I might combine two different views to facilitate editing additional, sensitive attributes:

<form action="#viewState.getValue("xe.formAction")#">

  #viewCollection.getView("basicForm")#
  #viewCollection.getView("organizerForm")#

  <input type="submit" value="Save Now">
  <cfif obj.getIsPersisted()>< a href="javascript;" class="delete">delete this object</cfif>

</form>

#viewCollection.getView("validateForm")#
#viewCollection.getView("organizerValidateForm")#

Future Improvements

Today, I use tables to layout my forms. Standards-weenies be damned, I think forms are a perfectly acceptable way of laying out the tabular notion of a form and they afford many conveniences (like auto-adjusting widths) that XHTML solutions can’t.

However, I’ll also admit to having fallen in love with Uniform and Quackfuzed’s tagset for generating the forms (which has a recent v2 release). My goal is to migrate to XHTML form layout (and potentially jQuery for validation… especially if Dan Switzer would just get on with it and port the ever amazing qForms to be a jQuery plugin ).

What About You?

How do you handle reusing forms or views in general? The idea of reuse is at the core of frameworks like Model-Glue and yet I feel like it can still be a challenge to figure out a successful strategy.

There are two things that I like about this plugin in particular:

1. Full support for keyboard input
2. Designed to handle a lot of data

I have been working with RIAs for a long time, trying to bridge the gap between the power of desktop applications and the distribution model of web applications. When trying to build data-heavy applications, we were regularly disappointed by either the lack of design for lots of data or the poor performance experienced once we exceeded a sample screen of data. I’m glad to see more emphasis on this type of performance at the requirements gathering phase.

I must give great credit to Mark Mandel who has listened to my frequently crazy and incomplete bug reports, found and resolved many bugs ahead of the 1.0 release last week. When I first relaunched MotorsportReg.com six weeks ago, we were getting a lot of strange errors, particularly during application startup and under load. Having just checked my email this morning, it’s refreshing how empty the inbox is totally devoid of our automated bug report emails.

Transfer’s Yin and Yang

There is, however, one area in which Transfer is a clear and direct tradeoff: performance. In some ways, Transfer can boost your performance with its caching model that will return fully composed objects in just a few milliseconds that would otherwise require many database queries and createObject() calls letting you take the hit once for creating the objects and then speed along on subsequent use.

Sean Corfield has mentioned many times that using Coldspring as a transient beanfactory would be extreme overkill and not very performant due to all of the other things Coldspring does beyond createObject(). I am now coming to grips with Transfer performance for large batch operations or where the beans you are persisting will not be read back (frequently, if ever). These are cases where I am finding Transfer may also be overkill.

My Example

My application has an email blaster that organizers can use to send email to their attendees. I record each campaign with the subject, body and sender in the database and then separately record each delivery so that I can use a unique ID in my return-path header for automatic bounce handling and reporting for the original sender. Typically an organizer will send emails out to less than 100 people at a time but there are very valid scenarios where they will send 1000-2000 messages.

Before the conversion to MG/CS/Transfer, even these very large sends would complete in a matter of seconds. It was effectively an insert for the campaign and then one insert per delivery and spool the message to disk. Figure at 20ms per delivery + spool, you can send 1000 messages in approximately 20 seconds.

Since the conversion, we had received an increasing number of timeouts during blasts. At this point, I have our timeout set up to 1200 seconds to try and let these blasts go out and some are still not making it. I have done a bit of instrumentation using CFLOG and debugging to see how long each step is taking averaged out over 3 deliveries in a single campaign:

The absolute numbers aren’t that important; it’s the relative size of them that matters. The actual hard time to save the delivery data and send the message is right around 15% of the total time. Now, this is my dev/test machine, CF7/Linux running older hardware (dual P3 800mhz processors) after a recent restart. Here are some numbers from a production server which is dual 2.8ghz Xeons after it’s been running for a few days and is using ~600mb of RAM (presumably a significant part is the Transfer cache, set to Application scope):

You can see the faster hardware in the create() and the CFMAIL tag. I listed the three SQL inserts as they were all over the place; probably a result of other load on the box whereas the test server was only used by me. Clearly the 6ms INSERT time experienced on the test server (which points to the same database server) is feasible. Without being able to instrument Transfer internally, my only guess is that these much longer save times are related to the large cache size. Anecdotally, we do notice the site seems to run more quickly after either bouncing CF or restarting the MG application which includes a rehash of Coldspring/Transfer.

Conclusion

I have come to love Transfer over the past 9 months. The things it can do, especially if you are building smart decorators that create intelligent business objects, is quite powerful. It has especially helped to reduce duplication of validation checks and centralization of key business rules (like no duplicate email addresses in our system, for example).

But you don’t get everything for free. The overhead incurred for this functionality hurts high-performance operations. As I am scaling the learning curve with these frameworks, the saying “if you have a hammer, everything looks like a nail” comes to mind. With only a few months of experience with Transfer and now just six weeks of production history, the judgment to know when and where to use that hammer is still lagging.

With this research under my belt, I am going to revisit some key places in my application:

• Anywhere I run Transfer.save() in a loop to evaluate the potential total number of objects
• Anywhere I run Transfer.save() without first running .validate() on my bean (I use decorators for almost every object in my app with a .validate() routine)
• Anywhere I use Transfer to persist data but don’t interact with it in single-object form (e.g., I have an audit log system that I use transfer objects to persist but I pretty much exclusively display it in aggregate query output)
• Our JVM is currently using less than half the RAM allocated to it; if the size/age of the cache is a factor in the performance, switching the cache to be session scope may potentially resolve some of these issues (but will probably create others…)

These are probably “duh” moments for experienced users of these frameworks but when migrating an existing application to a new framework, my instinct was to convert everything consistently to simplify the process and take advantage of the new tools. The good news is that I still have my SQL-based code in subversion so going backwards should be easy.

Update 6/8: The announcement is out and the code is available for download. I’ll be upgrading our production tomorrow. Remember kids, don’t forget to delete all of the *transfer* files out of your generated directory when upgrading!

2008 called and it wants its common sense back. Pronto.