Software and Architecture

Seam + {jboss 5.0 | jboss 5.1} = ?

2009-11-09T17:21:00.001-08:00

About a month ago I completed the upgrade of all my Macs to Snow Leopard. This generally went smoothly -- not a surprise for a release that has been charactered as "more refinement than upgrade."

One exception was a jboss 4.2.2/seam 2.1 application that would hang when generating a list view of objects that included images. My initial reaction was that this provided an opportunity to upgrade to jboss 5.x and partake of whatever enhancements that offered.

This proved to be a task that ended in frustration. I spent ~ 40 hours on it and eventually gave up. I fell back to the earlier version and upgraded to jboss 4.2.3 which solved the problem (which I think was related to using Java 1.6).

I thought I'd share some of my experiences, just in case someone else finds it useful:

The first glitch was that the version of seam I was running didn't appear to work with jboss 5.x so I upgraded to version 2.2

The attendant upgrades caused me to change some of the DB mappings
e.g.,

change blob annotations (mysql specified)
From: @Column(name = “data”, length = 8000000)
To: @Column(name = “data”, length = 8000000, columnDefinition = “mediumblob”)

I also tried switching to jboss developer studio to see if that would help me uncover the problem -- this had no real impact.

The core symptom was that nothing was coming back from pages that generated a list of items in the DB and no Hibernate queries showed up in the back end stream.

I eventually tried to go in to one of the more "internal" pages

http://localhost:8080/webtwo/Competition.seam?htmlTitle=competition&competitionId=2&cid=6

(a real advantage of Seam's rest interface) and finally saw a hibernate query on the background stream with the warning:

WARN [Param] could not create converter for: competitionId javax.el.PropertyNotFoundException: Target Unreachable, identifier ‘competitionHome’ resolved to NULL

This warning was similar to the error I was getting that the authenticate method resolved to NULL.

What appeared to be happening is that the seam annotations weren't being processed correctly (specifically @Name("competitionHome") ).

After searching on this error I found this link
http://www.seamframework.org/Documentation/WhatHappensWhenYouDeploySeamAppInJBoss5
which made me think that things are basically broken.

As I said, rolling back to the original code, the problems went away in jboss 4.2.3. However, I must admit that I'm surprised that the issue exists in the newer versions of seam/jboss

Jboss
5.0.0.GA	Stable	104 MB	2008-12-05	LGPL	 134971	Download	Notes
5.1.0.GA	Stable	130 MB	2009-05-23	LGPL	 181731	Download	Notes
Seam
JBoss Seam 2.2	2.2.0.GA	Production	111 MB	30.07.2009	LGPL	Notes	Download

I know that this is "unsupported" code, but I would still think that there would be better testing. After all, 5.1 was out in May of this year and Seam 2.2 claims

Seam 2.2 examples target JBoss Application Server 5.1.

Now, I do realize I could have been anywhere from 1 minute to 1 month away from a solution for this problem (if anyone has a solution, I'd be more than happy to try it), but I have two closing thoughts:

Allocate more time than you might have expected towards making the transition

It would have been appreciated if the various teams involved paid more attention to migration tools (even documentation) and/or backward compatibility. My various searches trying to solve this problem turned up a lot of people having obscure issues with the transition: this is not the way to encourage wide uptake of a tool set.

XCode 3.x

2009-10-20T06:17:00.001-07:00

I've been coming up to speed on iPhone development and thought I'd share some of my experiences.

The first is that Beginning iPhone 3 Development is a very useful starting point. I tried a couple of other resources but finally settled on this. I do like books better than video when learning a new environment but this book also has the advantage of being up-to-date and accurate. I hate trying to learn an environment/languiage when the examples are wrong! Beginning iPhone 3 Development employed a technical reviewer who worked and verified all of the examples. Shouldn't every book like this have a technical reviewer? -- the world would be a better place.

A few observations on the development environment, which although reasonable is a bit more primitive than netbeans or eclipse, especially around refactoring
e.g.,
renaming a class/header file doesn't rename all of the imports throughout the project, and there isn't a "Refactoring" capability that I've found that does this.

The C aspects certainly harken back to an earlier era, e.g., one has to define a function in a .m file and declare it in a .h file for it to work correctly. At least Code Sense minimizes the chances for mistyping in this case.

XCode only allows you to view the interface specification (the .xib file) via the interface builder--however it is useful to realize that the .xib file is really an xml file that can be viewed in a normal text editor e.g., emacs.

Misspellings count and don't seem to generate errors:

Surprised that Code Sense doesn't prompt when overriding methods from the superclasses, which causes the the classic "why wasn't this method called debugging session"

Similarly the compiler doesn't tell if you're calling undefined methods

Misspelling accessor e.g.,
childController.tltle = prez.name;
vs
childController.title = prez.name;
gets the error: request for member ‘tltle’ in something not a structure or a union

Also can't believe that there isn’t enough introspection so that you still have to do these:


#pragma mark NSCoding
-(void)encodeWithCoder:(NSCoder *)encoder{
	[encoder encodeObject:field1 forKey:kField1Key];
	[encoder encodeObject:field2 forKey:kField2Key];
	[encoder encodeObject:field3 forKey:kField3Key];
	[encoder encodeObject:field4 forKey:kField4Key];
}
-(id)initWithCoder:(NSCoder *)decoder{
	if(self = [super init]){
		self.field1 = [decoder decodeObjectForKey:kField1Key];
		self.field2 = [decoder decodeObjectForKey:kField2Key];
		self.field3 = [decoder decodeObjectForKey:kField3Key];
		self.field4 = [decoder decodeObjectForKey:kField4Key];
	}
	return self;
}
#pragma mark -
#pragma mark NSCopying
-(id)copyWithZone:(NSZone *)zone{
	FourLines *copy = [[[self class] allocWithZone:zone] init];
	copy.field1 = [[self.field1 copyWithZone:zone] autorelease];
	copy.field2 = [[self.field2 copyWithZone:zone] autorelease];
	copy.field3 = [[self.field3 copyWithZone:zone] autorelease];
	copy.field4 = [[self.field4 copyWithZone:zone] autorelease];
}

On The Bright Side

@synthesize obviates a lot of useless typing.

Categories seem cool and I plan to explore them further. Categories let you add methods to an existing class -- the source code of the existing class is not required.

From the xcode 3.1 doc Categories

Provide a simple way of grouping related methods. Similar methods defined in different classes can be kept together in the same source file.

Simplify the management of a large class when several developers contribute to the class definition.
Let you achieve some of the benefits of incremental compilation for a very large class.

Can help improve locality of reference for commonly used methods.
Enable you to configure a class differently for separate applications, without having to maintain different versions of the same source code.
To declare informal protocols.
See “Informal Protocols ,” as discussed under “Declaring Interfaces for Others to Implement.”

The doc also contains a suitable caveat:

Although the language currently allows you to use a category to override methods the class inherits, or even methods declared in the class interface, you are strongly discouraged from using this functionality. A category is not a substitute for a subclass.

That is, categories are powerful and can blow your foot off, the "power tool" version of shooting yourself in the foot, if you're not careful.

You can schedule actions to happen in the future and then cancel them when superseded by a subsequent user action.

[NSObject cancelPreviousPerformRequestsWithTarget:self selector:@selector(singleTap) object:nil];
			[self performSelector:@selector(doubleTap) withObject:nil afterDelay:.4];

It is very nice to have that capability just "built in."

Patterns in Network Architecture

2009-09-14T17:17:00.001-07:00

I recently finished reading Patterns in Network Architecture by John Day. It's an attempt to rethink network architectures and polish up "the unfinished demo" that is the internet.

Now, I'm not a network guy, so I can't evaluate the quality of his proposed solutions in any detail, but I liked his thought process and found it a useful read for anyone interested in a good example of thinking through a hard problem and coming up with a disciplined "minimal covering" solution.

Day focuses upon discovering the appropriate layers and layer structures necessary for communication. He works up from interprocess communication on a single machine to processes communicating across multiple machines.

The implications of this analysis are interesting in and of themselves and closely resemble structures seen in other systems. His metaphors are primarily in terms of name lookup and binding, using compilers and operating systems as examples (I have to admit that this only feels partially correct to me: I think the full problem is more akin to providing the data/instructions to a processor and therefore needs to include the mapping from a "memory location" to an actual address accessible by the chip's execution unit e.g., it should take into account caching, TLBs et al.).

The first and foremost conclusion in Day's opinion is that there is one layer that provides interprocessor communication and it replicates. That is, the structure of each network layer is the same, but the policies and optimizations differ depending upon the particulars of what the layer is connected to. Every layer has three parts, data transfer, IPC (Interprocess Communication) control and IPC management -- where control is short cycle management.

In his words

"Layers have two major properties that are of interest to us: abstraction and scaling (i.e., divide and conquer). Layers hide the operation of the internal mechanisms from the users of the mechanisms and segregate and aggregate traffic. But most important, they provide an abstraction of the layers below"

When moving from a shared to a distributed environment the core new functionality is an Error- and flow-control protocol (EFCP). This protocol replaces the shared memory mechanisms of the single system to ensure reliability and to provide flow control in the environment of communication between two systems. An EFCP PM is a task of the IPC process. Although in theory, such a process could be included even when communication is on a shared processor, in practice, this communication is so reliable as to make it redundant.

I think that the core insight/technique was to frame communication from the network perspective as being from application to application and not as interacting with the network e.g., in the figure below (6-15 from the book) communication is conceptualized as being across, that is between applications at the same layer, rather than down through the network and back up to the other application.

The application concerns itself with developing a shared state with its partner application. The N-1 layer provides an an abstract aggregated API to support the application's view of the communication and performs whatever aggregation and abstraction necessary to develop a shared state with the N-1 application on the other side. It then hands off details to the N-2 layer which gets it to the N-2 layer on the other side, etc.

Yes, it is just encapsulation all over again, but as we all know, finding the right thing to encapsulate and doing it in a practical way takes a lot of work.

I'm eliding a number of the other key findings of the book such as

The observation than an address only needs to be unique within a (distributed) application layer
A connection is made only after authentication has been obtained and the connection authorized etc.

All are developed in a thoughtful way showing deep insight into the problem.

Although not the easiest read for someone without a strong networking background, it is an interesting and useful exercise to watch someone so well versed go through the process so thoughtfully.

Flavors of Architects and Analysts

2009-08-24T18:00:00.001-07:00

I was recently involved in a discussion on the difference between architects and business analysts and decided to put together my thoughts on the subject.

Here they are: for each category “Architect”, “Business Analyst” there are a number of sub-categories

I normally think of three levels of Architecture:

Application – addresses evolution and delivery of a single application (small set of highly related functions), activities include:
- Partitioning functionality within an application.
- Developing best practices.
- Assuring flexibility to meet current and immediate business needs.

Platform – addresses evolution and delivery of a multiple application for a particular business area (applications grouped by functionality/user community), activities include:
- Assuring a commonality of results.
- Providing for fine grained interoperability.
- Developing frameworks that allow multiple applications to ship on a common substrate.
- Building in flexibility to meet business developments on the planning horizon (this year/next year goals) for moderate-sized groups within the company (~100 people)
- Assuring that a substrate achieving these goals is in place so the applications can pick it up at the appropriate time.

Enterprise:
- Identify core data elements and services that will be important over the strategic timeframe.
- Assure that there is an appropriate mix of flexibility/capabilities to meet strategic goals e.g.,
  - If acquisition of companies is a strategic goal, methods for rapidly merging personnel, purchasing, and operational information systems are important.
  - If acquiring products is a strategic goal, capturing data about supply and delivery chains etc. is important.

On the business analyst side I similarly think of three levels:

Department Level:

What are the processes involved in performing a function: including as is and to be states?

Division Level:
- What is the external business goal that is being addressed?
- Is this the right way to address it?
- Should functions be merged/refactored?

Corporate/Strategic:
- What are the strategic business differentiators going to be in the business that we want to be in 3-5 years?
- What must the business look like to support them?
.

At all levels there should be some time spent to look at potential inflection points that might radically change the structure of delivery and build in flexibility to address that potential, e.g.,

For architecture think outsourcing, software as a service, location aware computing, etc.

For business think increased competition in product acquisition, competition from generic products, regulatory/legal landscape.

How these various functions are actually assigned to people depends a lot upon the scale of the problem, the level of risk/uncertainty, the talents of the people involved, and the flexibility of the organization. At one extreme, a star performer building upon a solid platform architecture (in the sense used above) can be a combination business analyst/application-architect/developer for a system serving 50+ users in a non-validated environment.

I think it important that business analysts are able to understand the business processes and vocabulary well enough so that there is a good transmission of information between those expressing the needs and the analyst. This implies a greater stickiness between the analyst and the user community than is necessary for the architecture or project management functions.

Similarly there are commonalities in the level of abstractions used in the architecture level (Application, Platform and Enterprise) that imply that levels are sticker than business areas or technology.

In theory, project management is more transferable, but the stickiness here revolves around legal requirement, diversity of end use (geography, user types), system novelty.

Open Source as an Architectural Driver

2009-07-26T07:18:00.001-07:00

Phillip Longman's post about open source products in healthcare (specifically, the VA's "health IT system") talks about how the Midland Memorial Hospital's installation of the VA system went well because it was easy to use, and it was easy to use because it was open source.

Well, the first one wasn't a surprise. Well-designed easy to use software is well, easy to use. The fact that this leads to successful system uptake/user adoption should be no more surprising than the fact that people like their iPhones. The second factor being easy to use because you are open source is a bit of mental speed bump: Easy to use open source? Well, maybe if you are a developer. The article states that the ease of use stemmed from its ease of modification. Now I can't comment on that since I am unfamiliar with the product and have never been involved in a hospital centric system.

However, open source and ease of modification? Yes, that fits. A successful open source project, by definition, must be relatively easy to modify: An interested developer should be able to jump in, modify the code and stand up a running test build in short order. Otherwise the project won't attract enough attention to survive. More importantly, I think a system that is easier to modify will leap ahead in functionality even if it starts out behind.

This is one of the reasons we've seen such useful build/test tools come out of the open source community e.g., ant/junit/maven etc.. All open source projects needs tools like these to succeed since they are critical situations where you cannot afford a dedicated buildmeister or QA organization e.g., you're a developer modifying the code to satisfy your needs.

Similarly a clean, modular, layered structure is going to be favored, codependencies (A depends upon B, B depends upon A) are going to be rejected, since they require understanding two pieces of code and their interrelationship to perform a successful modification.

Both of these issues can be more easily compensated for within a "closed source" shop, since revenue-generating projects employing full time personnel can invest the time and discipline to keep things working, even if the software has a few points of poor structure. In addition, if the points of 'bad architecture" are manageable there is little incentive to fix the problem, since it might easily cost more to fix than it's worth, given the costs of running a full-time development team.

One of the strongest examples I think we've seen of this is the EJB vs. Hibernate controversy with the resultant "conciliation" of EJB3. Hibernate, being open source, could give developers what they needed rather than what they "should want" & it won due to its simplicity and speed.

This argues for a bias towards an open source style, even in a closed source system. For example, can your consultants (internal or external) add/modify deep system functionality? Designing your system in a way that supports such modifications will make the architecture better and help keep the product fresh.

Again, why this works for hospital systems is beyond me, unless there has been an undocumented rash of coding by doctors and nurses.

iPhone: changing the way we think

2009-06-30T14:45:00.001-07:00

I'm struck by how the iPhone has changed the way we think about what can be done with software based devices assisting us as beings-in-the-world. I'm doing a Heidegger reference here because the iPhone is more than just ubiquitous computing: a device always at my side that could answer those important questions like:

Is there good coffee close by?

What's the weather going to be like later?

How old was Kennedy when he was elected?

Although it certainly is that, it has become a lot more, changing both the economics of software delivery and what it means for software to be delivered.

It's not just that there are a billion apps (or so it seems) in the app store, but the economics of iPhone software is such that a small gaming company can do a novel game e.g., tying rope around wooden blocks, get traction with it and make money. That didn't sound that amazing until I read an interview with the developers in Gamasutra that reminded me how hard it was to make money in computer games pre-iPhone. Not that it is easy now, but compared to the stories I heard when I attended a few Game Developer conferences earlier in the decade, it is trivial. Let's just say that the economics of doing a platform (XBox, Playstation, wii) or PC based game were daunting, to say the least, and the likelihood of getting paid for your game was minimal, even if the game was successful.

The core of the iPhone's difference is as a platform that is easy to use, location aware and ready-to-hand -- more like a hammer than a computer.

As a platform it is sufficiently distinct that it is also effecting the way we think about delivering healthcare. Looking at this list highlights core features that are "new," not "new" in the sense of being completely unheard of, but new in the sense of being practically available for use by the overwhelming bulk of the user community -- sort of like the difference between having a generator kit/knowing about electricity and having an electric grid that you can plug your device into.

As a user-assistant, the iPhone allows me fully exploit the affordances of my current location. I can see where I am on a map, look at overhead imagery of my current neighborhood to see if there is something that I want to photograph, and, if there is, use a small application to grab the geo coordinates so I can later tag the photos I took with my (non-GPS enabled) camera.

The end result is something that is always with you, knows who you are, knows where you are, has connectivity both up (3G/internet) and down (bluetooth to local devices) while providing a simple effective mechanism to easily add functionality in small increments.

I think this makes it the biggest game changer since the rollout of the internet to the general public. However, I also realize that this means that it is time to code up a small test application for the iPhone.

PS: I don't have any experience with the Google android platform or the Palm Pre; these observations may apply equally as well to them.

Linked Data

2009-06-01T18:37:00.001-07:00

Finally, thanks to a discussion with Eric Neumann a few weeks ago, I'm beginning to understand what Linked Data is all about. First a caveat -- although I credit Eric for helping me see how linked data fits into what I'm doing, the following interpretation is strictly my own as are errors of omission, commission or orthogonality, although I think my view is supported by the Design Issues document.

The short story is that linked data provides stable identifiers for stuff (a more abstract form of things). These stable identifiers then allow you to say things about this (particular) stuff without necessarily making a strong ontological commitment.

I like this. It provides for interoperability and integration. It does not provide any inference guarantees which is fine by be, and something that I have been advocating for a while. The Linked data site also has links to a number of datasets which publish stable identifiers for useful stuff. The site also gives examples of how to publish your own data.

Hopefully data.gov will provide its data in this form in the near future.

Wolfram Alpha

2009-05-17T18:42:00.001-07:00

Wolfram Alpha is supposed to be launching in the next few days and has been getting a lot of publicity. For background, here's a link to a short YouTube demo of Wolfram Alpha, and a NY Times article and Doug Lenat has a nice post on his impressions.

From what I can see (and I don't have access) even though it doesn't live up to some of the early hype, it achieves a very interesting result: it allows retrieval of general computable information using a simple natural language processing (NLP) interface.

This allows for analysis similar to that permitted by a data warehouse, but within different design space. The design goals of WolframAlpha, unlike those of a data warehouse, preclude prestructuring the data in marts to allow rapid querying of the data in relatively well defined ways. However similar to the mart/warehouse situation you must still provide a speedy response to the quantitative queries to prevent users from drifting away while waiting for an answer.

The question is how is this done? Rumors on the net indicate that the underlying data is an RDF triple store, which makes a lot of sense since RDF Triples constitute a vertical, model free storage approach. In operation, I imagine that the queries provide nice entry points for initiating a spreading-activation fan-out process on the graph. When the activations intersect you can proceed to roll back up to the initiation points suggested by the query, clustering in a bottom-up data-driven fashion along the way. The clustering also affords a natural way to structure the data for presentation to the user.

Although I'll admit that this is just an educated guess as to the mechanism, it does suggest an interesting set of technologies involving fast linking and roll up of data for ad-hoc queries without requiring a lot of effort to tune the data to a specific query.

Generating a set of vetted and annotated data is a different problem, but hopefully would not require a significantly greater level of effort than the ETL portion of current warehousing efforts.

Wolfram Alpha therefore constitutes another factor leading me to be more vertical in my storage designs. In the coming months, I'm hoping to run some benchmarks on production hardware/datasets so as to ground the practicality of this approach and then get permission to publish the results.

Update 18 May 2009: I did try Wolfram Alpha today and it failed on my first try "age distribution of England vs UK," not so much from any idiosyncrasies in parsing my query, but because it appears to be encoded with the identity "England == UK." This just goes to show how important it is to be spot-on with your identity information aka "synonym tables are easy, antonym tables on the other hand......aren't."

launchd

2009-05-06T17:08:00.001-07:00

I just upgraded to a new laptop (driven mostly by the need for more RAM -- hopefully 6G will be adequate for a couple of years). It got me thinking: even though it's great that the Mac will copy all of your old apps over effortlessly to your new machine, it also happily copies all your old unused cruft over to your new machine, and that's not so great.

So, in the spirit of good hygiene (and H1N1 preparedness), I decided to open up the console and look to see what I might find. I discovered that I had a couple of launchd jobs that referenced executables which didn't exist on my system any more e.g., carbon copy cloner.

I have been able to rid myself of all the launchd issues, by cleaning up the Launchdemons/launchagents under the Library folder but
I still haven't been able to rid myself of all of these

/Applications/Safari.app/Contents/MacOS/Safari[54428]: Warning: accessing obsolete X509Anchors.

This is even after searching the web a couple of times. I think the problem starts up after I open an article from NewsFire, but I'm not completely sure. This is definitely a space in which I believe correlation is not causality.

If anyone has any ideas on how to fix this, I'd appreciate it.

BTW it is really nice to have a built in tool like Console: it is simple and effective with just that little bit of extra functionality (string filtering) that makes all the difference in usability.

Java Concurrency

2009-04-20T15:12:00.001-07:00

A predictable side effect of having (way too) many years of experience in Java is that certain "new" features escape your notice. This is particularly true if the IDE's don't pressure you into changing your previously successful, and still functional patterns (the way they do with generics).

I realized this when reading Java Concurrency in Practice. It's a very good book -- I can't say it really opened my eyes on concurrency since I had done some work on multi-master VME based real-time systems years ago, but it is spot on, well written, and a nice refresh. In addition, it made me aware of the thread/concurrency capabilities available in newer versions of Java such as ThreadPoolExecutor

I recently built a file crawler/hash-calculator/storage system as part of my namedData work using an ArrayBlockingQueue and explicitly created threads. ThreadPoolExecutor appeared to allow an easier approach with cleaner shutdown/interrupt semantics.

Java tips has a clear example -- the primary change that I would make to this example is to size the thread pool based upon the number of processors available (on my laptop this returns the number of cores).

It took me less than an hour to make this change, test the code, etc. The final product is a lot cleaner, has better shutdown behavior, and even feels like it runs faster. Definitely the right way to go.

owl:sameAs is a very strong assertion

2009-04-06T17:53:00.001-07:00

There's been an interesting discussion on the public-semweb-lifesci mailing list with the subject "blog: semantic dissonance in uniprot" which, appropriately enough, was spurred by a blogpost entitled semantic dissonance in uniprot. This post talks about a uniprot entry which listed a Drosophila (fruit fly) protein sequence as having been isolated from "a young sporophyte contained within a seed."

The point being that although one doesn't find fruit fly genes in plants, following the owl:sameAs link leads directly to that conclusion. This generated a very long, fairly thoughtful and minimally flame based conversation on owl:sameAs and identity in general.

As the discussion progressed, the problem with associating identity across graphs (ontologies/systems of data developed by different organizations) was noted, e.g., (in pseudo annotation) mySystem:itemA owl:sameAs yourSystem:itemX, the issue being that the use of the terms is usually subtly (and often not so subtly) different between the two systems. This problem is especially apparent when making assertions about real objects which exist independently out in the world. For example: "gold" may have a property, but does the property adhere to a single molecule, or a group of gold molecules and if so what characterizes a group of the appropriate size? For example given:

A nanotechnology view of gold (still under development)

A semiconductor view of gold (probably reasonably well characterized)
A jewelry view of gold

what are the precise boundaries of their applicability? The issue doesn't arise in a system developed for nanotechnology, semiconductors, or jewelry. The problems surface only when these systems are linked together.

My thought is that the difficulty centers around the extreme power of owl:sameAs which indicates that things are identical in all contexts. However in the physical world not only is context everything, but context is also inherently incompletely specified.

In practice many of us heuristically treat identity in the physical world as operating as if identity means indistinguishable in this context, with the context being implicitly dependent upon the issue being considered. I would claim that this is the only reasonable way to proceed when reasoning in a practical manner about what is true about particular objects in the world (abstractions can obviously satisfy stronger conditions since they are abstractions -- with the context factored out to any level desired).

In the physical world, we cannot assure that even the ability to track a particular item with unlimited precision would allow us to make statements about that item which would hold through time. For example, although we might make assertions about a particular atom (#0x177FFEAA) of gold and its behavior, some if not all of the assertions may fail under unexpected conditions, e.g., after an event that alters the structure of the nucleus (nuclear collisions, extremely high temperatures etc.). Exhaustively specifying all of these conditions is impractical at best -- which is one of the reasons the phrase ceteris paribus has remained with us for so long.

In my own work, since I never worry about tracking individual atoms. I gravitate toward weak rather than strong assertions of identity, trying to be very attentive to context. This is very much in the spirit of the middle distance as developed in Brian Cantwell Smith's On The Origin of Objects. Smith's point is that our intuitions are well tuned to objects about our size that we interact with frequently. In data integration and architecture work (I had to get there eventually) it implies that integrating across fields that interact to some degree in the "world" is going to be more feasible than integrating across those that don't interact. The give and take of the practical interaction has allowed us to identify the particular features of each item that are important in context.

OSX Performance Analysis: Instruments

2009-03-23T18:03:00.001-07:00

I started working with OSX's Instruments performance analysis tool, partly out of curiosity and partly because I had just fixed a performance problem in an application using an ad hoc a priori analysis. It happened to solve the problem, but I have enough experience with performance issues to know that the a priori guess is often wrong.

Instruments is heavily related to dtrace and shares a lot of its core attributes. The key attributes are that it is low overhead and works with (almost) anything running on your systems (OSX apparently has the capability for some applications to turn off monitoring for security/DRM reasons).

There's a lot to like here: you can easily get it up and going on your system and it the analysis section is very user friendly:

Especially nice features include

Low overhead: the peak CPU usage I saw for the tool was ~ 16%
The ability to display exactly what is going on under the read head (the upside down triangle above the graph)
Being able to display parameters that you didn't think of turning on during the run. All parameters are captured. The selection only impacts the display -- a godsend for anyone who has had to rerun a test because they forgot to capture a parameter

That said, I couldn't get any particular instrument to focus only on the process specified. As you can see, all of the instruments capture all of the activity, even though they were set to focus on different processes. Additionally, the "default action" kept resetting whenever I dragged a new instrument onto the display.

It is still a very worthwhile tool, but if anyone has any tips as to how to get around these issues, I'd appreciate it.

Kindle

2009-03-04T18:26:00.001-08:00

It's a bit off topic, but I thought I'd point out how useful a Kindle can be for consulting. You can carry at least 500 reference books on it (and who needs more than 490 anyhow?). It is also very light and easy to read.

I do have a couple of qualms. It is a page oriented display (no scrolling), no touchscreen and has no spatial indexing e.g., the top side of the right page half way in, but other than that it's a win.

An important note on utility: O'Reilly e books can be read on the Kindle. The truly great thing about O'Reilly's e-books is that you get both the Kindle compatible mobiPocket files and the more aesthetically pleasing PDF files (for me, aesthetics matter--even in a SQL guide).

You can mix and match reading and reference between the formats depending upon your preference. Thankfully the files aren't copy protected. Thanks, O'Reilly, this is a very nice touch.

Seambay modifications to access Seam Annotations

2009-02-17T08:56:00.001-08:00

This post extends my last one about accessing Seam from the command line. Here I describe the transition from using EntityManager to using EntityHome.

The first thing I did was to create a new folder for the webSevice sources, which meant that I had to add this directory into the build.xml file and add all of the libraries into the compile path in NetBeans (both of which are obvious, but both of which I always forget to do).

The next was to make my action work similarly to an .xhtml page and interact with a home object rather than directly with the EntityManager
going from:

      
if (fileData == null) {
            fileData = new FileData();
            // various actions on fileData
            entityManager.persist(fileData);
        }

       
if (fileData == null) {
            fileDataHome.create();
            fileDataHome.persist(); //side effect of creating the defined instance
            fileData = fileDataHome.getDefinedInstance();
            // various actions on fileData
            fileDataHome.update();
        }

which also required adding these lines to components.xml

xmlns:transaction=”http://jboss.com/products/seam/transaction”   
....    
<transaction:ejb-transaction/>

None of which was particularly difficult and I was up and running in a hour or so.

Seam From a Command Line

2009-02-02T17:36:00.001-08:00

I recently wanted to access some seam derived functionality from a command line java program (something that I could run via cron). I ran into a few minor problems and thought I'd share their solutions.

The first problem was that seam annotations like @Logger won't work. I guess it isn’t that surprising, but the jboss seam annotations are unavailable to a command line program (at least not easily) since the portions of the framework that enables these annotations are designed to operate within a server.

This was disappointing. The @Logger annotation is really useful, but I couldn't come up with a way to get it going.

This pushed me into wanting to use web services as much as possible to take advantage of other annotations that I had built into my system, e.g., the ability to automatically stamp an object with time modified and time created to support temporal data operations.

I did not find the seam documentation about accessing seam web services particularly clear (especially when using netbeans) so I turned to the netbeans tutorial and was quickly up and running with the seambay example.

Note:

and

Data Deduplication

2009-01-05T18:10:00.001-08:00

IEEE Computer recently published a short survey on data deduplication. Conceptually deduplication is isomorphic to the named data approach I posted about a few weeks ago.

The vendors discussed include

ExaGrid

FalconStor

NEC DataRedux

Quantum DXi

Sepaton DeltaStor

Symantec

Since I'm not sure how long the pdf of the article will be available, I'm posting this as a follow up to my previous post.

My preference would be to see these capabilities built right into the internet/operating systems rather than separate utility servers.

Discussing Architecture

2008-12-23T18:20:00.001-08:00

If you're like me, you are forever grappling with finding the right format for discussing architecture with end users.

There are the standard architectural diagrams, but they really don't capture:

How the end users relate to the system
How all of the components are strung together to support the overall business processes
What artifacts (data, documents, etc.) are produced

I think there may finally be a "better mousetrap" available in ArchiMate which has a suitably small number of patterns/best practices that allow you to capture what's important in a large system at the appropriate level of detail. The most important aspect of ArchiMate is that it is a mixed mode modeling language with a distinctive, simple shape for each type of artifact.

ArchiMate allows for seven different types of things: Services, Processes, Organization, Products, Information, Infrastructure, Applications, and Functions.

As of yet I am not completely clear on the breakdown of these categories and to be honest I'm not sure that it really matters. ArchiMate is a communication tool. If it helps you communicate, it has served its purpose. If your way of breaking down the architecture is slightly different from their recommendations (which I strongly feel could use more examples) it may have no more impact than if your class structure for a domain is slightly different from someone else's. Don't get me wrong: I'm a big fan of standards when they have been properly vetted and heavily used, but until they hit that point it is important to be flexible in using them. Flexibility allows a standard to grow and cover a sufficient portion of the domain; otherwise it will wither from lack of use.

The presentation format that speaks to me the most is the layered diagram as shown on p 11(figure 12) of the Enterprise Architecture Development and Modelling paper. See below:

Here's my simplified take for a clinical trial system used by physicians and patients.

What I like about this format is that all of the elements on a particular layer are at the same level of abstraction and are easily placed in relationship to the other few things that are at that same abstraction level. Simultaneously, you can see what supports (and is supported by) a particular component. Each perspective (which appears on the same diagram) only requires concentrating on a small number things at a time and can easily be held in your short term memory.

One of the key things about the layers is that they distinguish externally available from internally consumed data interfaces -- especially highlighting those that cross abstraction boundaries. These external interfaces are distinguished from those that support multiple applications at the same level in the stack. Such "external" interfaces (which are internal to a particular level of abstraction) are more easily altered since they are more tightly coupled organizationally. This nicely foregrounds the implication that care should be taken in designing the external, level crossing APIs and lifecycles since changes to them will be harder to coordinate given the diversity of the interested parties.

Name that data

2008-12-08T17:28:00.001-08:00

There is an interesting trend that I feel has the potential to fundamentally shift the way we think about how data is used in networking and applications.

It involves attaching unique names to data, i.e., referring to the data by its SHA1 hash value. Unique identifiers allow a number of things. For example, they allow you to retrieve the data from the network without worrying where it resides on the network, as described in
A New Way to look at Networking in which Van Jacobson discusses breaking out the content of the page from the page itself.

At 42:47 in the talk there's the section on
Dissemination networking

in which
data is requested, by name using any and all means available (IP, VON tunnels, zeroconf addresses, multicast, proxies, etc),
Anything that hears the request and has a valid copy of the data can respond. The returned data is signed and optionally secured, so its integrity and association with the name name can be validated.

Rather than getting all of the data from a particular location e.g., as specified by a URL, we get the identifier for the content of the data and then let the network supply the data from the location "nearest" (using the appropriate distance metric) to its use.

A similar idea exists in ZFS' implementation of a copy-on-write transactional model:

ZFS uses a copy-on-write transactional object model. All block pointers within the filesystem contain a 256-bit checksum of the target block which is verified when the block is read. Blocks containing active data are never overwritten in place; instead, a new block is allocated, modified data is written to it, and then any metadata blocks referencing it are similarly read, reallocated, and written.

This design allows small changes in large files to be reflected by changing only the blocks that have been altered rather than by rewriting the whole file to a new location. This simplifies backup procedures, reduces R/W bandwidth requirements, etc. Part of what's significant here is that we're dealing with abstractions of the data, e.g., checksums rather than the data itself. If we use cryptographic hash functions rather than checksums the ideas become isomorphic aka, get me this block from wherever it is.

This has a number of potentially interesting applications depending on the granularity of the "named data". As a simple example, answering the obvious: "am I working with the copy of the file that was emailed to me last Friday, or an older version?" Even with coarse grain naming it would be possible to create mashups of music already on a users computer -- just transmitting offsets into and segment durations of existing content gets past DRM issues entirely (morally if not practically-- since much DRM protected media is encoded).

Uniquely naming the content is not just about networking, nor is it just about data but is really about the cross product of the two: the data and its location/retrieval. Data location and retrieval is most of what is involved in computing: unless data is being actively processed by a computational unit (in this case, I'm talking about the integer and floating point units on the chip) the rest of "computation" is about the retrieval and storage of data e.g., do I put this in a frame buffer, in the cloud or in a shredder?

Imagine an ecosystem of data that represents everything that you own/care about -- you could partition this data into multiple overlapping categories based on any number of attributes such as:

temporary: make no copies

pieces of a larger whole: applications would minimize the number of named datasets that they change when they perform updates e.g., editing out that first minute from your video won't change every block of the video.

number of independently survivable copies required, with what longevity?
coupling the data to geographic position: How closely should the data follow me as I move around the planet? Can it stay where I put it, or should it go where I go and be available for high bandwidth processing.

This are just my first pass ideas. This concept of divorcing the data off from its location and higher level structural organization opens up the potential for a whole new set of applications which provide enhanced user functionality by pushing a lot of these data management, replication and caching issues deep into the infrastructure.

Semantic Interoperability: from Mashups to Inference

2008-11-24T17:33:00.001-08:00

My last post looked at semantic interoperability from the standpoint of the CDISC BRIDG model. Thinking back on it, I found that writing it left me with more questions than I had before I started.

I have to admit to being unclear as to what is meant by “semantic interoperability,” since I have heard it used in a number of different ways depending upon the audience. (apparently I’m not the only one: the wikipedia entry on semantic interoperability has the caveat All or part of this article may be confusing or unclear.)

"Semantic interoperability" puts requirements on the data, on the models, and on the processes of using them. How we respond to those requirements implies different interpretations of what it means to be semantically interoperable.

I think that there are three basic ways of using data that are "semantically interoperable".

“hands-off” data integration between designated well curated systems--this is the way in which I think it is used most often.

“hands-off” data integration between any systems sharing common identifiers e.g., publish an interface and allow anyone to use it.

The dual use of integration with any published interface that provides the data that you're looking for - which I think is less common e.g., I'll use any map, or any book information service rather than Google/Amazon (or Yahoo/BN). I haven't seen this that often and I think it sounds a bit sketchy.

Using OWL reasoners etc. for inference across systems to generate new information.

The requirements around these things are pretty different, both in data quality and in the congruence of the requisite component models.

The first “hands-off” data integration between any systems sharing common identifiers doesn't really require any similarity of models other than around the key integration point(s). You need the name of the referent of the data, the name of the data item and the format of the returned data e.g., "The first president of the United States"; "date of birth" returned in ISO 8601 format. Of course the more points that you want to make referenceable between the systems, the more the models have to match e.g., your model of US presidents has to contain a way of dereferencing the person and that person's date of birth. The more independently developed and maintained your systems are, the more quickly you want to start using RDF to give you very stable identifiers for your referents.

If the systems are required to do some curation/analysis of the data, the exported models need to match more closely so that you can derive the correct metrics to perform the analysis and understand the relationships between individual data points. A good example of this comes from Nick Malik who points out

So, if you look in a database and you see a purchase order... has it been approved or not? The answer depends on the business unit that created it.

Your models can be in a number of different forms (UML, OWL, etc.) and be wildly divergent from the underlying reality, but if the delusion is shared you can achieve some synergy.

Inference of course requires (at least) a locally full up OWL ontology since that's the only modelling language that permits inference. Models also have to more closely resemble the shared "current best understanding" of reality (which is of course a moving target in a scientific domain) or the resulting inferences will be worthless, or at best amusing.

However, doing an ontology is a big deal (see The Joy of Ontology by Suzanna Lewis for a discussion). The increment of commitment that we're making here is decidedly non-trivial, especially if the domain that we are trying to model is of substantial size.

I think the clinical trial domain is a good example of substantial size. BRIDG took a long time to do, it is still undergoing revision and does not allow inference. I would argue that given the continued refinement of some of the base terms (sex and gender were recently updated), even if there was an ontology, hands-off inference is not something that lies in the near future, simply because the ground doesn't provide a sufficiently firm foundation.

Just for clarity -- this doesn't mean that turning loose an inferencing bot over a sufficiently sized test set would not yield interesting and perhaps even transformative results. It just means that the inferencing would be part of a web research project rather than a production operation.

I could be wrong on this (and gladly so), but I did live through AI winter an I can no longer utter the phrase "sufficiently smart compiler" without irony.

Semantic Interoperability: Adverse Events

2008-11-11T10:28:00.001-08:00

When reviewing the Bridg Release 2.0 Static Elements report.RTF I did look in a bit of detail at the adverse event model.

Here's a summary:

The AdverseEvent
class is decribed as having the following connections and attributes

Connections

Association link from class PerformedProductInvestigation

Association link from class Subject

Association link to class AEOutcomeAssessmentRelationship
Association link to class AECausalityAssessmentRelationship

AEActionTakenRelationship

Generalization link to class PerformedObservationResult

Association link from class PerformedProductInvestigation

Generalization of PerformedActivity adding an evaluationMethodCode attribute; PerformedActivity captures the duration of the activity

Association link from class Subject -- the clinical subject (An entity of interest, either biological or otherwise.)

Association link to class AEOutcomeAssessmentRelationship
links the AE to an observation
For example, recovered/resolved, recovering/resolving, not recovered/not resolved, recovered/resolved with
sequelae, fatal or unknown

Association link to class AECausalityAssessmentRelationship
links the ae to an observation.
For example, when an adverse event occurs, a physician may evaluate interventions that may have caused the
adverse event.

Association link to class AEActionTakenRelationship.
Specifies the link between an adverse event and the steps performed to address it.
For example, study dose reduced, protocol treatment change, etc.

Generalization link to class PerformedObservationResult
links all observations/protocol deviations etc together with a report.

The AE itself has the attributes:

gradeCode

severityCode
seriousnessCode

occurrencePatternCode
unexpectedReasonCode

expectedIndicator

highlightedIndicator
hospitalizationRequiredIndicator

onsetDate
resolutionDate

The end result is a structure that has a formal relationship that is well thought out, should cover all situations and permits systems to interoperate.

In my mind, this is not the same as assuring semantic interoperabillity. For semantic interoperability to really occur, the grade codes must be comparable across sites, hospitalization criteria must be identical (or at least commensurable) etc.. Achieving this comparability requires continuing education and harmonization efforts, constant feedback of metrics to practitioners etc.. It therefore represents a much higher bar.

This is not in any way a criticism of BRIDG. You need something like BRIDG -- a well vetted industry standard -- to be even be able to begin such an attempt. However, true semantic interoperability involves not only the structure of the data, but the data itself.

CDISC/BRIDG

2008-10-27T18:52:00.001-07:00

Last month I attended the Boston Area CDISC Users group meeting (BACUN). All of the presentations were interesting and useful. However, I found that the one by Lisa Chatterjee on BRIDG stood out as particularly informative.

The BRIDG Domain Analysis Model is a representation of protocol-driven biomedical/clinical research.

One of the goals of the effort is Semantic Interoperability - I don't think that this means that "following the model" guarantees semantic interoperability, but rather that BRIDG constitutes a starting point from which a semantically commensurable system can be built. The bridge team appears to view the model as a foundation for other more problem specific representations (CDISC/HL7 etc.). The idea being that if you can map BRIDG <-> HL7 and BRIDG <-> CSDISC the HL7 <-> CDISC mapping is (relatively) straightforward.

There is no question that BRIDG represents an excellent starting place for using data in an interoperable fashion.
All in all it shows a very inclusive approach
-- and a surprising openness to modifying the model to ameliorate difficulties encountered in use.

The core modeling language is UML and spreadsheets are used to track much of the mapping (there already is a draft version of a spreadsheet that maps the BRIDG R2.0 model to RIM2.18).

From the presentation:

I have to admit that I haven't examined the model in complete detail. However, from what I've seen almost everything that you need for the target domain is there and the level of abstraction feels right: low enough to be relatively easy to implement, but high enough so that you don't get wedged into a corner from the get-go.

I did look in a bit of detail at the adverse event model,which is represented in the Bridg Release 2.0 Static Elements report.RTF.

What we see in Figure 5 : View 4 - Adverse Event is ~ 90% of the complete domain model with a number of classes added in support of recording adverse events and tracking their eventual analysis/resolution.

Since adverse events raise some issues about semantic interoperabiltity that I want to talk about in detail I will cover them in my next post.

BTW on my mac, the only application that could open the .rtf file with the figures was open office 3. MS word 2004 elided the figures.

Seam: the ftl advantage

2008-10-13T20:24:00.001-07:00

I finished the first pass schema for the flexible drug discovery framework and pointed the most recent (2.02) GA release of Seam at the database.

My hope was that it would produce pages with the latest ajax-friendly table sorting headers -- sadly it didn't. In addition, this version retains the practice of generating pages that use labels and column headers wired to a particular language (English) rather than allowing the headers to be language dependent messages, even though the pages fully support localization (via language specific message_x.properties files in the resources directory).

I was in the process of fixing these issues manually via emacs using these instructions (I find that emacs makes it easier to select the files that I want to edit -- Netbeans picks up too many files and deselecting 70 or so "extra" files is too much work). While making these changes, I was discussing a new system with one of my clients and recommended that they consider an open source solution since they could change it (or hire somebody to change it) if at some point they encountered something that they didn't like. I thought this better than the alternative of waiting for a potentially unresponsive vendor to pay attention to their problem.

As was saying this, I thought to myself: "Seam is open source, maybe I should try to fix the code rather than edit the result." I'm usually hesitant to go down this path. Sometimes it works, but my a priori estimate is that the process of understanding the code structure, getting the build environment running etc. costs a day (or more) before anything productive comes out the other end.

However, as a proof of principle, I thought I should give it a try and see how it went.

I'm very happy to report that given a combination of good architecture and good tool selection on the part of the Seam team, making these modifications was almost trivial.

Let me explain why, and encourage you to "do this at home."
The core reason is that the seam generate-entities command operates in a number of phases, one of which generates the .xml and .xhtml files using freemarker template files (.ftl). This allows the required changes to be made without either looking at the java involved or developing any understanding of the calling structure, etc.

The .ftl files (in ./seam-gen/view/) are pretty much self-documenting (which is handy, given the level of documentation included in the files) and very easy to change -- errors thrown by the freemarker engine are clear and easy to work with.

A couple of (minor) caveats

The .java files in the src/action will need to be removed between runs of seam generate-entities.
I think it is a good idea to point the seam generator at a different target directory (specified by ./seam-gen/build.properties -- "workspace.home") while debugging so that you don't accidentally overwrite edits that you've already made (oddly I thought of this BEFORE I ran the generator)

In summary, if you find yourself making a lot of changes to seam generated files, change the .ftl files instead; it can be much, much easier.

Suddenly mysql won't start

2008-09-29T07:21:00.001-07:00

Suddenly is a bit of an exaggeration. This happened on my desktop -- I use mysql on my laptop almost daily, but only once every few months my desktop. They are similar environments: latest Mac OSX 10.5 patched intel based macs. The mysql on the desktop was transferred from my previous PowerPc based desktop, but has been used a few times since the transfer.

In any case, the normal startup action

 sudo /usr/local/mysql/bin/mysqld_safe

failed with the following output.

Starting mysqld daemon with databases from /usr/local/mysql/data /usr/local/mysql/bin/mysqld_safe: line 395: /usr/local/var/: Is a directory /usr/local/mysql/bin/mysqld_safe: line 401: /usr/local/var/: Is a directory STOPPING server from pid file /usr/local/mysql/data/rdf-8-Tower.local.pid tee: /usr/local/var/: Is a directory 080910 15:39:05 mysqld ended tee: /usr/local/var/: Is a directory

At this point I said to myself "well this hasn't been upgraded in a while, I should upgrade mysql" !!bad idea!!

The upgrade didn't solve the problem. After tracing through the script in more detail I did a

sudo ./my_print_defaults

which reminded me that

Default options are read from the following files in the given order: /etc/my.cnf /usr/local/mysql/etc/my.cnf ~/.my.cnf

/etc/my.cnf contained

[mysqld] log = /usr/local/var/mysqlLOG.log #If no specific storage engine/table type is defined in an SQL-Create statement the default type will be used. default-storage-engine=myisam max_allowed_packet = 16M #Enter a name for the error log file. Otherwise a default name will be used. log-error=/usr/local/var/ #Enter a name for the slow query log. Otherwise a default name will be used. log-slow-queries=/usr/local/var/

Which I changed to

[mysqld] log = /usr/local/var/mysqlLOG.log #If no specific storage engine/table type is defined in an SQL-Create statement the default type will be used. default-storage-engine=myisam max_allowed_packet = 16M #Enter a name for the error log file. Otherwise a default name will be used. log-error=/usr/local/mysql_ERROR_LOG.log #Enter a name for the slow query log. Otherwise a default name will be used. log-slow-queries=/usr/local/var/mysql_SLOW_l_LOG.log

 /usr/local/mysql/etc/my.cnf

didn't exist, nor did
~/.my.cnf

This allowed the db to start but I couldn't log in with any of the user accounts normally available (including root). It appears that using the mysql-5.0.67-osx10.5-x86.dmg file to update caused the db user information to get hosed.

Fortunately I'm pretty neurotic about backups and so I easily recovered just by copying /usr/local/* from my last backup.

NOTE: this backup was to a separate disk performed using SuperDuper -- TimeMachine isn't going to get files in /usr (which helps explain why the disk space used by TimeMachine is smaller than I expected).

Seam on Amazon EC2

2008-09-15T17:41:00.001-07:00

I just completed putting up a demo of my seam work on Amazon Web Services EC2 service. I primarily did this to ground my advocacy of EC2 as a good option for small biotechs that may need occasional bursts of compute power but have neither the cash to buy adequate servers for peak compute load nor the staff to maintain them.

I thought that putting up my jboss/seam/mysql demo would also be sufficiently non-trivial to give me a good feel of what it is like.

There are similarities between EC2 and other virtualization options (EC2 is based on XEN after all).

The core differences in my mind revolve around having S3 as a backing store. Since S3 is on Amazon's servers you need to pay more attention to security keys etc.

My recommendation is the go through the Getting Started Guide -- even to the point of saving a modified image. This will assure that you have the proper accounts set up both on EC2 and S3 and you have a bucket set up on S3 for storing your image.

I found it easiest to create a bucket using the Python examples -- even though I have never done much more than a "hello world" program in python (yes one that consists of

#!/usr/bin/env python
print "Hello World"

). The python code is the most self contained and required the fewest downloads of ancillary libraries in my environment (Apple OSX 10.5.4)

Building and saving an image takes a while -- I would recommend doing a reboot of your virtual machine to make sure that all of the changes take hold (boot processes start as designed etc.).

Not to overstate the obvious, but the image that you start with has a tremendous impact on the time it takes you to get up and running. I settled upon an image that already had mysql5, jboss 4.2.2 installed, and it made things much easier. In general I didn't feel that the images were particularly well documented. Not being that familiar with Fedora I thought that the difference bettween the Fedora-core-4 and fedora-core-8 was how many CPU cores they were optimized for, not the revision number. My initial foray with the fedora-core-4 image stopped when I realized that it had mysql4 rather than mysql 5.

All in all the experience wasn't too bad (I don't think that I could ever call one of these experiences "good" -- if it were good I would just be able to click a "do it" button that would do exactly what I wanted), and would have been much better if I hadn't "lost my keys".

TOGAF and Evaluating Architectures

2008-08-29T06:37:00.001-07:00

The old adage is that you always have an enterprise architecture even if you never designed one -- the point being to encourage an organization to spend the time to design one. This is all well and good, but given an ongoing enterprise, what's the best way to determine what enterprise architecture you have, where you want it to go and most importantly, how to get there.

For various reason I've started looking at this issue again and have just refamiliarized myself with TOGAF (The Open Group's Architecture Framework). I had forgotten how much I liked it: it is pragmatic, highly tailorable and focused on open cross-organizational solutions. I'm not going to do a detailed analysis of TOGAF vs. other frameworks -- it's not really my interest as I'm definitely in satisficing mode here. What follows is my (long) elevator pitch of the TOGAF take home message.

The top level graphic of the TOGAF process captures the flavor pretty well

The preliminary phase is key but very easy to overlook. TOGAF suggests that this phase consists of defining the overall objectives and scope

Define Objectives
- Assure that everyone who will be involved in or beneﬁt from this approach is committed to the success of the architectural process
- Define the architecture principles that will inform the constraints on any architecture work
- Define the ‘‘architecture footprint’’ for the organization — the people responsible for performing architecture work, where they are located, and their responsibilities

Define Scope and Assumptions
- The business units that are involved
- The level of detail to be deﬁned
- The speciﬁc architecture domains to be covered (Business, Data, Applications, Technology)
- The time horizon that should be addressed by the architecture.

What I find attractive about this whole approach is its focus on getting getting buy in from the key players in the organization, defining their roles and developing a shared set of expectations around what's going to be done as part of the architecture effort. The preliminary steps give the team some initial criteria for driving the architectural vision, but then TOGAF immediately requires them to ground it as supporting the needs of the business users. I find this grounding critical; often the business thinks that architecture efforts are worthless and often they are right because the architectural model hasn't been grounded in the business process. Note: in this case "business process" and "user's scientific process" are equivalent.

The key to the success of an architecture effort is to address current pain points as they will be reflected in the business processes that will be in place when the architecture rolls out. Sorry if the tense of the last sentence was a bit torqued. What I mean is that the architecture needs to hit a mark to support business operations as they will be in the future, not as they are now, and that some of the problems that are being experienced now will only be exacerbated by these planned changes.

The dialog with the Business Unit leader sounds something like

We're planning to do more collaborations in the future, but with our current collaborations we have a terrible time registering new users and tracking responses to our questions about the data. However, if we put an architecture in place which uses our new authentication mechanism that supports OpenId it will radically simplify the process of adding new users.

In addition, if we use vendor X's implementation of the Life Sciences Industry Architecture, queries will be automatically tracked.

Our ability to handle more collaborations on the back end is increased as our new system allows us to share extra capacity across multiple business units, thereby sharing the cost of reserve capacity to meet any unanticipated surges in demand.

Such a "political/operational" model for rolling out an architectural analysis implies that everyone who contributes to the effort should get something out of it (this is a goal, but the closer you can come to meeting the goal, the more self-organizing the system becomes).

As one proceeds around the TOGAF loop, you pick and choose what makes sense given the decisions made previously (which of course you are always free to revisit) analyzed to the level of depth that is appropriate.

Think of TOGAF as providing a (partial) checklist of processes to use and things to consider that help you reach the end state of
Boundaryless information flow (tm)

I think of it as being similar in spirit to the way the Software Engineering Institute's Risk Management Taxonomy provides a comprehensive checklist of things to consider when undertaking a project -- it keeps you from forgetting something that would be obvious in retrospect.

An aside:

My favorite quote from one of their pages:

Another company is developing a flight control system. During system integration testing the flight control system becomes unstable because processing of the control function is not quick enough during a specific maneuver sequence.

The instability of the system is not a risk since the event is a certainty - it is a problem.