Jesper Joergensen's BEA blog archive

Customer Commitment (Honeymoon phase warning)

2008-09-09T00:16:00.000-07:00

3 weeks in, I think it still counts as the honeymoon phase of my new job although I wish my own honeymoon lasted that long. So bear with me while I muse over something I find truly great in my new habitat.

Today I stumpled upon this post by Scott Hemmeter who figured out the URL that would return all the entries on ideas.salesforce.com that have been implemented in the next upcoming release (Winter '09).

First of all, it's great to see this information pulled out and emphasized by a 3rd party. But to me, still learning the ropes here, it's even more astonishing that this list actually exists.

Think about it. Not only can you go and find all the feedback, ideas and complaints that people have provided for the salesforce.com products on ideas.salesforce.com. You can actually see which of these ideas will be implemented in the next release. That, to me, is true commitment. Commitment from customers to the platform (telling salesforce.com what they love and hate). And commitment from salesforce.com to customers that the company is listening and responding.

If you know of other companies that are doing an equal or better job at this, drop me a comment. Like I said, I know I am in my honeymoon phase, but I am not sure there are many companies out there that take this commitment as seriously.

Voting with my feet: Can anyone ignore SaaS anymore?

2008-08-29T19:45:00.001-07:00

On Aug 11, my little break after BEA Systems ended when I started as product manager at Salesforce.com for the Integration and API product areas in the platform group. I think my new gig will give me a lot to write about and I hope to get some writing done, but we'll see how it pans out.

I don't think it's particularly visionary to go work for a SaaS company at this point. It's been pretty obvious for some time that SaaS will be a paradigm shift in the enterprise software world. Of course this doesn't mean the big enterprise software vendors will go out of business or move wholesale to SaaS any time soon. SaaS is still pretty new, particularly in the enterprise. And on-premise software will probably never go away completely, so we'll just have to wait and see where the new equilibrium will end up at.

There is a hilarious article on ZDNet where Lawson's CEO, Harry Debes predicts the demise of SaaS in 2 years. It's not so much the prediction as his reasoning that put me on the floor dying from laughter:

"Getting signed up as a SaaS customer is fast, but getting out is just
as fast, whereas traditional software is like cocaine — you're hooked.
It's too difficult and expensive to switch providers once you've
invested in one. If it were easier to jump ship, a lot of people
would've hit the eject button on SAP a long time ago."

(there's more silly stuff in the article). So on-premise is a better model because it's more like cocaine? You can't make this stuff up.

Ryan Nichols over at Appirio wrote up a nice rebuttal including this description of the joy of being a product manager in a SaaS company:

"Now look at the happy life of a SaaS product manager. After launching a
new feature, they get immediate, direct feedback on real-life usage
patterns. They see what works and what doesn't. They can even launch
two versions of a feature to see which works better. Their development
teams can fix problems instantly, without having to issue patches or
service packs."

This is his words, not mine. But it sums up pretty nicely why I am really excited about my new job.

So what about BPM? Well, it's as relevant as ever, but I will no longer be focusing on on-premise BPM Suite style products. As a "hobby" I will be thinking about how to bring more powerful, yet easy-to-use process modeling to the force.com platform but my main focus will be on the API and integration features.

Can Spectral Clustering Help Partition Social Networks?

2008-06-20T23:42:00.001-07:00

Twitter has lots of problems with their architecture that they're trying to solve and the service can undoubtedly be made to perform well by fixing the most obvious problems they have today.

But ultimately Twitter will run into a scaling problem that's much harder to solve: It will have to efficiently partition the social graph, so it can serve segments of the graph on different servers while minimizing the communication between these servers. This problem is more difficult than the usual cases where caching and user partitioning will fix it all. In fact, the more pure mathematical solutions to the problem happen to be NP-complete, which means that you won't know how long time it will take for a computer to solve them.

However, I came across a post on Spectral Clustering for Graph Partitioning over on the Social Media Research blog which describes how spectral clustering can be used to segment social graphs. The methods have been developed for other areas where graph partitioning is of interest and I am not sure if they can be applied as successfully to a problem like Twitter or even larger networks like LinkedIn and Facebook. Specifically I don't know how large the computations would get when you apply the algorithms to a network like LinkedIn with 22M nodes and 120M edges (as reported by Oren Hurvitz from LinkedIn's session at JavaOne). Another problem seems to be that algorithms like this one in MATLAB seem to apply only to symmetric, undirected graphs whereas Twitter is asymmetric, directed (you can follow people without them following you).

But I am pretty excited that there might exist a practical way to do social graph partitioning and I'd like to look more into it. I am curious if anyone out there have thought along the same lines...

More Twitter: Wouldn't this work or am I missing something?

2008-06-18T00:55:00.001-07:00

Pouring through more posts on the Twitter problem, I still didn't see anyone suggesting the solution I proposed. They seem satisfied with stating that this is a hard problem to solve.

Hueuniverse describes the problem from a base assumption that messages are not duplicated out to individual subscribers and correctly states that, with this architecture, delivery of messages is simple while retrieval is very hard to scale (because you have to execute complex queries across distributed databases to retrieve a time line).

The option of duplicating messages is rather bluntly dismissed:

One simple but painfully restrictive solution is to duplicate the data for each user. Basically what this means is turning the service into an email system. Each user is given a mailbox and whenever someone they are following publishes a status, it is copied into their inbox, as well as into all the other followers’ inboxes. This brings the solution in line with existing systems such as webmail services where data partitioning alone can accomplish great scaling. But this comes at a price. First the data storage grows significantly faster and requires much more disk space, together with increased backup cost. Second, it makes other actions much more complex such as erasing a message once sent.

[...]

A big problem with this solution isn’t technical but business-wise. If building a scalable microblogging service simply means a heavy investment in servers and storage, and little actual hardcore technology, the only barrier left for competitors is user acquisition.

The blog post continues over 3 parts and ends without offering an alternative solution which leaves me asking: What's so bad about duplicating messages if it keeps Twitter in the air? Who cares if it's "hardcore technology" or user acquisition that gives them a competitive edge?

I am not sure what kind of usage pattern the author had in mind because there are no examples (i.e. are we talking Scoble or just John Doe). This post by Israel L’Heureux comes closer to describing the real problem with duplicating messages:

Nothing is as easy as it looks. When Robert Scoble writes a simple “I’m hanging out with…” message, Twitter has about two choices of how they can dispatch that message:
PUSH the message to the queue’s of each of his 6,864 followers, or
Wait for the 6,864 followers to log in, then PULL the message.
The trouble with #2 is that people like Robert also follow 6,800 people. And it’s unacceptable for him to login and then have to wait for the system to open records on 6,800 people (across multiple db shards), then sort the records by date and finally render the data. Users would be hating on the HUGE latency.
So, the twitter model is almost certainly #1.

(btw, I am convinced Twitter's model is closer to #2 today, but it's irrelevant to the point about duplication). Israel's point is that when you have a LOT of followers, duplication becomes a really big problem. There is a certain point where you can no longer dismiss the amount of disk resources and DB writes required with this model:

Do you see a scaling problem with this scenario?
Scoble writes something–boom–6,800 writes are kicked off. 1 for each follower.

Michael Arrington replies–boom–another 6,600 writes.

Jason Calacanis jumps in –boom–another 6,500 writes.
[...]
And here’s the kicker: that giant processing & delivery effort–possibly a combined 100K disk IOs– was caused by 3 users, each just sending one, tiny, 140 char message. How innocent it all seemed.

Since Israel wrote this, Scoble's fan base has grown to 27,000+ followers, so this problem is just getting bigger.

These two blog posts are pretty representative of what I've read about Twitter in the blogosphere. People "get" the problem but they don't really propose a solution to it.

So I am curious what people think about the hybrid solution I proposed:

So let's look at it another way. Scoble is actually an example of a different usage pattern, let's call it "broadcasting". So why not create a hybrid architecture to support each separately? For "sparse populations" (few connections, few posts) a copy scheme is used. But broadcasters" should be identified and treated differently. For example, the top 10% by number of followers could be treated as broadcasters. A broadcaster's messages are stored centrally and each user pulls them when building the view. Caching increases performance and sharding helps too. Broadcaster's messages can either be pushed to or pulled from each shard, so individual user queries do not have to cross nodes. This would only create as many copies as there are shards/nodes.

I haven't seen anybody else propose something similar, so I am guessing there is some obvious flaw. Let me know...

By the way, I think this hybrid solution is really just an example of coming up with a design that matches the usage pattern. If you design a communication solution so it optimizes the use of resources according to the way people tend to naturally communicate, then you're system will be more resource efficient and thus more competitive.

Social graphs should look similar to how we are socially organized in real life. They have clusters of strong interconnectedness (neighborhood, work, school) with fewer links between these clusters. Overlaying this is a "one-to-many" pattern where many people are also connected to a few "celebrities".

Optimized communication is a matter of detecting these patterns in the graph and adapt the way messages are delivered and consumed accordingly.

I am a Jetty Contributor (I almost forgot)

2008-06-15T17:38:00.001-07:00

Back in the startup days around 1998, we decided at one point to build our next project in Java Servlets (this was before J2EE) and migrate away from the mix of perl and C that we had been using until then.

Our most successful app was a Chat Server written in C and being used at the busiest public Web Chat Servers in Denmark. We had spent a long time tuning this app for performance and we wanted to get the best possible performance out of Java as we migrated.

At the time, the Jetty Servlet Engine was already of high quality and definitely the best choice on the market. I suddenly remembered that I actually contributed a little bit of code to the software. During profiling we found that Java's built-in method for case-insensitive string comparison took up a non-trivial amount of time during request parsing and processing, so I replaced it with some more optimized code (I honestly can barely remember how it looked) and this put me on the list of contributors where I apparently still am today!

It pleases me to see that Jetty has done well since we started using it about 10 years ago. My hat off to Greg Wilkins and everyone else on that team.

If Twitter wants to support two usage patterns, it should use a hybrid approach

2008-06-04T13:17:00.001-07:00

If you're sufficiently tuned into the geek channel, you're probably getting either tired of or curious about the same old song playing again and again: the one about Twitter and its ongoing performance problems. If you're like me, then you can't stop yourself from trying to come up with a better strategy for them even though you're not getting paid for it and they probably have thought of it all already (since this is what they wake up to every morning).

Anyways, just for my own pure satisfaction, I thought about it and I will share my thoughts here even if I am repeating other blogs or if the Twitter boys have already figured it out.

First of all the presentation by Blaine Cook (who is no longer with Twitter) from last year shows that the Twitter team knows all the right strategies.

| View | Upload your own

Blaine comes up with 5 parts to a better future. Although to me they read more like these 7:

Testing and benchmarking
Getting the 101 stuff right, no bad coding, correct indexing
Denormalization
Partitioning
Caching
Messaging
Dealing with abuse

These are all important areas, some of which Twitter has already addressed. I am going to focus on partitioning and then broaden out from there.

Moving to a partitioned model

Twitter needs to move to a "user shards" distributed database and server architecture. It's clear that they understand this and it's also clear it's not an easy rewrite. But it's essential.

With user shards, users have a "home node" where their data is stored and a load balancer directs a user to his home node when logging on. This helps isolate problems and load surges and it can be beefed up with a master-slave or master-master configuration per node. There will no longer be one big database holding all messages for all users. Each user will for the most part not traverse nodes when reading or writing.

User shards makes it possible to scale horizontally, but it's not quite a slam dunk due to the messaging/social nature of the app. It brings up another important point that has been discussed a lot: Should messages be copied out to each individual user or should they be "stored once" using queries to build individual views?

User shards only make sense if you copy messages out to each individual user and I personally think this is the right approach. As far as I can tell, this is not what Twitter is doing today.

The common argument against copying is the amount of storage needed and the difficulty of updating messages (deleting them, changing access rigthts, etc). Let me address both of these:

Updating messages

Updating messages is not impossible, it's just harder when you've copied them out. I don't think this is a big deal. Messages get updated very rarely and when they do, latency can be tolerated so the update process can run at lower priority and decoupled from the front-end.

Storage

I have a hard time believe this is really a problem for the average user. To pull some numbers out of a hat, let's say the average user has 50 followers and posts 20 messages a day. A message is 140 chars, so let's say it takes 500 bytes to store it (unicode plus overhead). This means each user will produce 500kb of data each day or about or 15MB a month. I just don't see this amount of data as a problem.

Now the heavy users are a different story. Take Robert Scoble. On his top month he posted over 2000 messages or 70 a day on average and he has 26,000 followers. This results in 1GB+ a month of data if you copy all his messages out to all his followers. Is this too much storage consumption? There are two ways of looking at it. One, it's not really Robert who consumes this much storage. It's the 26,000 followers, so on a per user basis, it's not that big a deal. But for a startup like Twitter, I can certainly understand that they would not be happy with this kind of storage requirements.

So let's look at it another way. Scoble is actually an example of a different usage pattern, let's call it "broadcasting". So why not create a hybrid architecture to support each separately? For "sparse populations" (few connections, few posts) a copy scheme is used. But broadcasters" should be identified and treated differently. For example, the top 10% by number of followers could be treated as broadcasters. A broadcaster's messages are stored centrally and each user pulls them when building the view. Caching increases performance and sharding helps too. Broadcaster's messages can either be pushed to or pulled from each shard, so individual user queries do not have to cross nodes. This would only create as many copies as there are shards/nodes.

The Twitter team seems to be on top of caching and messaging. Both are crucial to ensure proper response times, but as some has pointed out, they won't solve all problems alone. Partitioning and the hybrid approach described above will help make it work.

Finally, I don't understand why Twitter hasn't imposed limits on number of followers or friends. Over time they would want to increase these limits, but meanwhile they should impose limits that help keep the system running.

Next Steps: Life after BEA

2008-06-03T14:49:00.001-07:00

Oracle completed its acquisition of BEA in the beginning of May and I decided to take the opportunity to move on and look for something new after 5 1/2 good years at BEA.

If you are interested in BPM topics and that's why you're reading this blog, then stay tuned. I may very well stay in this important and growing market and continue to cover it here.

I have not yet decided what the next step will be and I will take some time to look around before making a decision. During this time, I expect to blog more frequently, but on broader topics.

New Blog Location

2008-05-26T21:17:00.001-07:00

This site has been living a quiet existence as a "mirror" of my main blog over on dev2dev.bea.com. But I am now changing over and making this site my main blog location. To those of you who have followed me over from dev2dev, I hope you will continue to enjoy my posts. I am currently in a transition phase, so I will probably not be posting much about BPM in the coming weeks, but stay tuned. I will keep you updated on what's happening.

As always, comments are very welcome, either here or on FriendFeed.

Over 125 people in our BPM labs at Participate!

2008-05-20T16:10:00.001-07:00

Oracle|BEA Participate is over. At the event we had well over 125 people visiting our BPM labs to do hands-on exercises. This a huge percentage of the total 600+ attendees, particularly when you consider that 4 parallel tracks of really interesting sessions were taking place at the same time. Tough competition!

We are really impressed and happy with this interest and we hope that everybody enjoyed the lab exercises. We used a VMWare image to host the configuration of AquaLogic BPM Studio and the projects used in the exercises, so we cannot easily distribute this (yet). But here are the PDFs for all the exercises:

Modeling and Simulation (Business - Basic)
Business User WorkSpace and Dashboard (Business - Basic)
Your First BPM Project (Technical - Basic)
1. Your First Modification of a BPM Project
Modeling Best Practices (Technical - Basic)
Diving Deep into the WorkSpace (Technical - Basic)
Testing and Debugging Processes (Technical - Advanced)
Deployment and Configuration (Technical - Advanced)
BPM and Enterprise Service Bus (Technical - Advanced)

Finally, big kudos to the team who pulled this off: Mariano, Mateo, Nicolas, Juan, Fernando and the many others who helped during the show!

Want the real deal? Try BPM hands-on at BEA Participate BPM Labs

2008-05-08T15:27:00.001-07:00

I have almost wrapped up my homework for BEA Participate and I am fired up and ready to go.

This year, I won't be speaking in any of the sessions. Instead I have devoted my time to our Participate BPM Hands-on Labs. The lab is open Tuesday after lunch and all Wednesday and you will be able to pick between 8 lab exercises that target beginners as well as advanced users and cover both the business and IT side.

So this is not just a propeller head thing. If you are on the business side and you are wondering why in the world you should care about BPM software or you've asked yourself "How does this look like in real life?" then you should stop by the labs and find out.

Here are the list of lab exercises:

Business Focused	Modeling and Simulation	Learn how a business analyst uses AquaLogic BPM to analyze and optimize processes
Business Focused	Business User WorkSpace and Dashboard	Explore how ALBPM gives business users visibility and control over processes
Technical - Basic	Your First BPM Project	From "File -> New Project" to a running process
	Modeling Best Practices	Explore further the capabilities of process modeling
	Diving Deep into the WorkSpace	Create a complete new look and feel for your end-users application
Technical - Advanced	Testing and Debugging Processes	Step though a live engine and troubleshoot a running process
	Deployment and Configuration	Create and maintain a BPM Infrastructure using scripts
	BPM and Enterprise Service Bus integration	Integrate business processes with the SOA infrastructure.

The lab is located in the New Orleans room on the Gold Level of the Hyatt Regency, Chicago. That's right here:

Looking forward to seeing you all there!

Sneak preview of BPMN support in next BPM release

2008-04-29T10:49:00.001-07:00

Alex Toussaint just posted a sneak preview of the new Studio look and feel in our next release of the BPM Suite. Bruce Silver was looking for a bit more bling in the diagram, so here's another screen shot showing a little more rich use of BPMN specific icons:

We've kept the curved transitions to make it easier for our existing users to keep their "classic" style if they want.

Technorati Tags: BPMN, BEA, BPM

Andrew McAfee, Verizon, Proctor & Gamble at BPM and Social Computing Event, May 12-15 in Chicago. iPod Touch for all attendees.

2008-04-14T20:38:00.001-07:00

If you're a current BEA ALBPM or ALUI customer, I hope you've already heard about BEA Participate (if not, then hurry up and check it out, you definitely want to be at this event). If you're currently evaluating some of BEA's BPM and Social Computing products, then you should definitely check out this event and talk to your sales rep about participating.

Yes. The title is correct: At the low price of $1295 for registration, you will get a free iPod Touch to use at the event AND take home after if you want.

Here's what's happening:

Keynotes: Andrew McAfee on Enterprise 2.0, Procter & Gamble on portal & social computing, and Verizon on the industry’s largest BPM deployment.
Customer speakers: Novartis, McKesson, University of Pittsburgh, US Office of Secretary of Defense, JPMorgan, Boeing, Weyerhaeuser, Screwfix, United Healthcare Group and many more which is featured in the agenda.
Agenda: a packed agenda with sessions ranging from KM in modern workplace, business rules, APIs galore, integration with packaged apps and many more. There’s something for everyone whether you’re a developer, coder, LOB, architect, IT mgr, business analyst or project mgr so no one will get left behind.
iPod Touch: more social applications then you know what to do with at this conference and all on ALI 6.5. And yes, attendees get to keep the iPod Touch (unless you don’t want it).
Product Experts: more BEA experts, engineers and architects flowing around to chat with and join in the white-boarding style sessions in between breakouts and meals.
Customer Briefing Center: schedule intimate one-on-one meetings with BEA executives and product gurus
Accenture: Accenture and BEA has teamed up to create an Accenture Innovation Center for BEA based solutions. This group will be at Participate and you can schedule a meeting with them to discuss your project needs.

This will truly be a unique event. We hear it over and over again from our customers: You don't want to just listen to us. You want to talk to your peers and tap into the collective experience. The feedback from last year's event was very positive. This is the best learning and networking opportunity for the users of BEA's BPM and Social Computing products.

Technorati Tags: BEA, BPM, Social Computing

You can serialize BPMN into BPEL, but BPEL should not be the serialization standard

2008-04-14T20:24:00.001-07:00

Bruce has written another chapter (and follow-up) in the developing story of BPMN and BPEL in a response to a post by Keith Swenson. Bruce is, as always, spot on. (I just wish he would include AquaLogic BPM when he mentions products that can execute BPMN.)

As Bruce explains, BPMN is not always easy to map to BPEL and the reason is that BPMN allows you to draw a full graph while BPEL is block oriented (Bruce's examples illustrates it well). The pro-BPEL folks say that this is not a problem because you can always devise a strategy to "unfold" the graph into a block structure. That's true, you can always find a way to represent a BPMN diagram in BPEL. But the way you choose to do this is likely to be your way. It will not be standardized. Therefore, the same BPMN diagram will end up looking different in BPEL depending on your tool/engine/custom method.

In theory, when the process executes, this doesn't matter. The "unfolding" should not change the behavior specified by BPMN, so the result of the execution should be the same if two different strategies are chosen. But the problem is that the strategy is proprietary. For example, A BPMN compliant BAM tool doesn't know how to show process trace for a process executed in BPEL. A BPMN compliant simulation tool cannot take the BPEL file and turn it back into the original BPMN file because it doesn't know the strategy used to go from BPMN to BPEL.

Now you might say: Why don't we just create a standardized mapping from BPMN to BPEL?

Yup. That would work. It would solve those problems. But the thing is, then we need to create this new standard. And if we need to create a new standard, then the whole reason for trying to use BPEL as the serialization standard (reason being that the standard already exists) disappears.

To use BPEL as a serialization standard for BPMN, we will have to create another standard anyway. That standard being the mapping of BPMN to BPEL.

The other approach is to create from scratch a serialization format for BPMN, aka a meta model; basically a schema for how to represent a BPMN diagram as a data structure. I like this approach much better.

The importance of BPMN lies in the need for a familiar visualization of business processes that is maintained across repositories, execution engines, BAM tools (process auditing), modeling environments etc. I believe this requires a BPMN-centric meta model. Let me explain: Say you don't have a BPMN meta model, but you have BPEL plus a standardized serialization method. If you're building a BAM tool where you want to show alerts, queue sizes etc. rendered on the process diagram itself, you will have to take the BPEL and apply the de-serialization strategy to turn it back into BPMN (remember, no one will be familiar with the BPEL for visualization because it was not used to build the original model, it was generated from BPMN using the serialization method). Doing this will require you to invent your own meta model for BPMN so you can store and render the resulting BPMN diagram in your application. It's pretty likely that no two applications will invent the same BPMN meta model and thus the diagrams will look and/or behave different even if it wasn't intended. This is the wrong track. It is too error-prone, requires unnecessary duplication of work (many meta models) and it forces too much focus on BPEL when BPEL is really mostly relevant for execution. BPEL is not suitable for business process modeling because it's too restrictive and machine oriented. This has been acknowledged pretty widely in the industry, so I hope it's not a controversial statement.

This is not a criticism of BPEL as an execution format. But it appears much more sensible to standardize a meta model for BPMN and then let vendors build implementation specific mappings to BPEL from there.

Technorati Tags: BPMN, BPEL, BPM, BEA

Integrated BPM Suite from IBM? I am not quite sold yet...

2008-04-14T20:06:00.000-07:00

IBM has been marketing their plethora of BPM related products as one integrated BPM Suite for some time. With the acquisition of FileNet it became even more difficult to select which product to get from IBM for BPM.

Now they've announced that they're really getting serious about integrating these components. Bruce Silver has offered his perspective on this (and followed up with more). From a competitor's perspective this is a really interesting announcement because on one side it acknowledges the importance of an integrated BPMS (such as AquaLogic BPM) and on the other side it admits that IBM does not have this yet.

Now if they could only catch up quickly, it shouldn't be such a big deal right?

Right, but it's not going to be easy for IBM to bring all those pieces together. We're talking dozens of products, home-grown and acquired, built on different architectures for different use-cases and now owned by different product groups, marketed under different brands and maybe even sold by different sales teams. There are overlaps and conflicts of interests all the way up and down the stack from engineering to sales, not to mention the headache of supporting existing customers. They have to align at least 4-5 executives and their organizations to make this happen and that's a hard thing to do in any company.

I'll be looking forward to seeing some execution on this.

Technorati Tags: BEA, BPM, IBM

Amazon EC2: A bullet-proof demo environment that facilitates collaborative and iterative development

2008-03-24T17:27:00.001-07:00

Over the last couple of months, I've been spending some time on Amazon EC2. I started by installing ALBPM Enterprise with an Oracle DB and it went pretty smooth. I moved on to installing ALI, ALIC, ALBPM Enterprise and Oracle on another image and while it took a little longer to get all the parts together I've now also got that one up and running minus a few last items. I took extensive notes but most of my experience will be outdated very soon when we release ALI 6.5 and ALBPM 6.1. These new releases will radically simplify the combined setup and instead of spending the time to clean up and double check my install notes, I am going to wait for this new release and write a post on how to run it on EC2.

I promised in my earlier post that I would discuss the reasons and pros and cons of using Amazon EC2. Here are some of the use cases I see for the service in increasing order of complexity/applicability/uncertainty of value:

A bullet-proof demo environment that facilitates collaborative and iterative development of demos
A bullet-proof test environment that is highly scalable and can be heavily automated
Extra capacity for on-premise systems that are already exposed to external users as well as internal systems (but security implications must be considered)
Primary hosting vehicle for certain external and internal solutions configured and managed on premise (for internal solutions, security implications must be considered)
Hosting platform for independent service providers (highly dependent on cost structure and value added by the services of the ISV, in some ways this is just a special case of 3 and 4)

The rest of this post describes the first of these use cases.

A bullet-proof demo environment that facilitates collaborative and iterative development of demos

Building and giving demos of enterprise software can be quite a daunting task. A demo often needs to show more than just the basic features. It needs to show how a product works with other products and it can be quite complex to get all this software installed and configured properly. Moreover, often the demo itself will change the state of the system which in best case will require you to "undo" or reset it in some way, in worst case it can make the system unstable and cause future demos to fail. Now, you might say that good quality software shouldn't behave like this. But keep in mind that the software wasn't designed for demos but rather for large-scale, perpetual deployments and requirements are very different for these two scenarios. To be more accurate, software vendors actually do consider demo requirements as part of product planning, but they don't always get high priority and even if they do, there is an important use case for complex demos at the cutting edge that cannot be planned for in advance.

One approach used widely today is to use vmware. With vmware you can build a demo on a vmware image, then "snapshot" it and now you can rerun it as many times as you like without anything breaking.

The problem with the vmware image is that it requires a lot of resources on your local laptop. It can tend to run slower than you'd like and the image often takes up many gigabytes of hard drive space. With an 80-120 GB hard drive, you won't be able to store a whole lot of images, so most people have a separate external hard drive for this purpose. This is annoying in itself, but the thing most people complain about is that it's not easy to share the image with your fellow demoers (pre-sales engineers for example).

Building demos should really be a collaborative and incremental activity. You start with a v1.0, your buddy then adds Gadget X which makes it even cooler. He shares it with the team and other people come up with further improvements and so on. This is really hard if you rely on sneakernet for sharing images. If you're a pre-sales engineer, you're often on the move and it just takes too long to download multi-gigabyte images over some hotel connection. You definitely don't want to rely on it for your demo the next day, and you will rarely have the time to snapshot and upload any improvements you've made. The result is that this just doesn't work. In our SE organization I think only a few people rely on vmware images for demos and they all really wish they could collaborate more.

Amazon EC2 makes this possible. It has all the benefits of a vmware image and on top of that it is stored in the cloud and it runs in the cloud. With a little additional magic you can even build demo configurations consisting of multiple hosts that talk to each other and emulate, say an SOA. Your images are stored on and loaded from S3, Amazon's storage service. So you never have to download to / upload from your laptop. As with vmware, your image can be booted over and over again and you will always start with the exact configuration that you want, so you know your demo will run smoothly.

But the most interesting advantage is that you can share your images with other EC2 users. Say a friend on your team would like to use your demo for a meeting the next day and she wants to add a few bells and whistles to it. Here's how you do it:

You add access privileges to your image for your friend's EC2 account.The image is still stored on your S3 account and is read-only, so nothing will happen to it.
Your friend can now boot up an instance from your image
Once booted your friend adds some new cool features and bundles up the new configuration as a new image stored on her own S3 account.
Your friend demos the new stuff to a customer. Everything goes smoothly because the demo is booted from an image that was tested the night before. It can be repeated endlessly without breaking.
Now your friend can share this new image with you and everybody else on the team. Everyone can now benefit from these new enhancements with having to shuttle gigabytes of data over the wires or meet in person for some hard drive sharing.

I am not sure if this sounds like modest benefits to existing vmware users. But I am convinced that this model has the potential to dramatically increase creativity and "prototype/demo innovation" which I think is really important to competitive enterprise software sales. I am very interested in feedback from SEs and people in similar roles that may be using this approach or just agree/disagree with this use case.

Costs and limitations

Of course Amazon EC2 comes with some costs and some limitations. The costs are pretty straight forward (copied directly from the EC2 page):

Instances

$0.10 - Small Instance (Default)
1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

$0.40 - Large Instance
7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

$0.80 - Extra Large Instance
15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

Pricing is per instance-hour consumed for each instance type. Partial instance-hours consumed are billed as full hours.
One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation. See Measuring Compute Resources for a complete description of an EC2 Compute Unit.
See Amazon EC2 Instance Types for details on available instance configurations.
Data Transfer
$0.10 per GB - all data transfer in
$0.18 per GB - first 10 TB / month data transfer out
$0.16 per GB - next 40 TB / month data transfer out
$0.13 per GB - data transfer out / month over 50 TB
Data transfer "in" and "out" refers to transfer into and out of Amazon EC2.
Data transferred within the Amazon EC2 environment, or between Amazon EC2 and Amazon S3 or Amazon SimpleDB, is free of charge (i.e., $0.00 per GB), except data transferred between Amazon EC2 and Amazon S3-Europe, which will be charged at regular rates. All other Amazon S3 storage and request charges will still apply.
Amazon S3 usage is billed separately from Amazon EC2; charges for each service will be billed at the end of the month.

With respect to limitations, here's what comes to my mind:

EC2 requires network connectivity. This is potentially a severe limitation because when you're on site at a customer you may not be able to get online. But I am suspecting that it's becoming less of a problem as Internet access becomes more pervasive with corporate guest WI-FI networks and 3G mobile network access.
EC2 is not optimal for demonstrating desktop components: If you want to demo an Eclipse plug-in or a full-fledged desktop app, EC2 might not be the right choice. But it's not impossible. I've seen people putting Windows images on EC2 and you can also get X11 forwarding up and running fairly easily.
EC2 runs Linux (mostly): Like I said above, I've actually seen descriptions for how to get Windows running, but I am not sure how easy it is. Anything that runs on RedHat/Fedora is pretty easy to put on EC2. If you need Windows or Mac, EC2 may provide less benefits.

Anything I've missed?

Technorati Tags: Amazon EC2, BPM, BEA, virtualization

Learning more about fraud-detection from the Eliot Spitzer case

2008-03-24T12:11:00.001-07:00

This article in Technology Review, published by MIT has even more information about the anti-money laundering systems that caught Eliot Spitzer. We regularly get requests for what role a BPM Suite plays in anti-money laundering solutions and the article provides a good context to understand the role of BPM.

First you need to funnel an enormous stream of events from their sources (credit card transactions, account transfers, deposits, etc) to a system that applies all the sophisticated algorithms that detect unusual patterns. This part requires complex event processing as a foundation, but CEP itself is probably not enough and BPM Suite technology is most likely not relevant at all for this job.

A BPM Suite is more likely to be involved when the algorithm detects suspicious activity. This triggers a business process that routes the case to a dedicated group of people for review. The BPM Suite can enrich the record with any relevant information on the customer before it's presented for review. Reviews could be tiered or scheduled in parallel (for additional fidelity) or both. All this would be modeled as a business process and with a BPM Suite this process can be easily integrated to the upstream fraud-detection system that triggered it, the downstream systems that would further process the case (such as the FinCen) and systems in between that can provide additional data or that needs to be updated as part of the process. The human workflow controlled by the task management component of the BPMS and all activity can be monitored to measure resource utilization, guarantee internal SLAs and document regulatory compliance.

In 2006, banks filed a record 567,080 "Suspicious Activity Reports" (SARs) and 2007 is on track set a new record. These are the filtered activities after bank personnel has reviewed activity flagged by the fraud-detection algorithms. For each of these reports, the bank has reviewed many times more activities flagged by their internal systems. This is the work that the BPM Suite can automate, manage and measure.

Technorati Tags: BEA, BPM, Event Processing, Fraud Detection

BPM Breakfast seminar, Mar 27, New York

2008-03-19T11:01:00.001-07:00

As mentioned earlier, we've recently released our State of the BPM Market white paper and we are currently touring 13 cities in the US to meet with business and IT professionals interested in learning about BPM and future trends.

I will be speaking at the upcoming New York event taking place at the Ritz Carlton New York, Central Park. We kick off at 8:30am with breakfast and then I will present on BPM market trends followed by Accenture who will talk about customer success with BPM.

I hope we will get a good discussion going. If you are going and have any questions up front, please leave a comment on this post.

If you would like to go and you haven't signed up yet, you can still register here.

Event Processing and Eliot Spitzer

2008-03-12T10:24:00.000-07:00

An interesting article on zdnet blogs explains how the banks' reporting of suspicious transactions to the treasury played a part in revealing Eliot Spitzer's dealings.

Suddenly boring, dry stuff like Complex Event Processing comes to life in a new way. It's probably well-known that banks monitor and report transactions over $10,000. But it turns out they also look for patterns of transactions below the threshold that could indicate foul play. This is obviously a perfect job for a CEP engine combined with some BPM technology and it sounds like there are opportunities for technology investments by the banks:

Institutions must have processes and trained staff in place to identify when and if a CTR (currency transaction report) is required, including the ability to aggregate same-day cash transactions made by or on behalf of the same person, and to file CTRs correctly. While automation has made these tasks less difficult, most institutions reported that their processes still include “manual” steps; for example, most institutions reported that their CTRs are reviewed by branch managers or compliance officers before being sent electronically to FinCEN or by mail to IRS. Institutions we contacted were generally unable to quantify their costs for meeting CTR requirements, in large part because they use the same personnel and processes for meeting other BSA requirements or for other purposes and do not separately account for CTR-related costs.

(emphasis mine). Interesting. So it looks like there are some money to be saved here, but lack of visibility into the real costs of the manual processing prevents the banks from seeing the potential ROI.

Shameless plug: If anyone in anti-money laundering, compliance or fraud detection is reading this, go check out WebLogic Event Server and BEA AquaLogic BPM. This is the technology you need to automate these compliance requirements.

Technorati Tags: BEA, BPM, CEP, Event Processing, Anti-money laundering, fraud detection

Ray Ozzie plugs Amazon Web Services

2008-03-10T10:37:00.000-07:00

Gigaom posted an interview with Ray Ozzie containing the following question and answer (taken a bit out of context):

OM: It (mesh-oriented apps) sounds like a great idea, but in reality can we actually deliver that kind of a mesh app architecture, and how soon?

RAY OZZIE: I think that you’ll see is over the course of this year, to 18 months, you’ll see the incumbents and startups, both, do their first big volleys of services platform, apps tools, runtimes, various things. It really isn’t being taken seriously right now by anybody except Amazon. They’ve done the world a service by putting out there some fairly provocative, interesting services.

(emphasis mine)

This is exactly how I feel about Amazon's web services after having played around with EC2.

After going through my experience I brought it up in conversations with many different people inside and outside BEA. Some people knew about EC2, very few, if any, knew exactly how it worked and it took a bit of time to explain. People used to working with VM images (mostly from VMware) understand the virtual OS image part, but the whole thing about command lines that boot up virtual computers in the cloud and all the aspects that come with it take a bit of time to "get". It did for me too, but once you get it, you can't stop thinking of what kind of opportunities it offers.

I am not saying that it's the best thing since sliced bread and you're waaay behind if you haven't already built your business model on it. What fascinates me is a sense that we don't know yet how this will be used. People will build businesses on Amazon's services and we don't know if the approach will be a niche or main stream. It could greatly speed up innovation and delivery of services without ending up as a mainstream platform. Or it could be copied and perfected by another company. I think it's a wide open field right now and I don't think we get to experience a wide open field that often. So I am excited.

Technorati Tags: SaaS, Amazon EC2

BPM Excellence Awards to BEA Government Customers

2008-02-25T08:31:00.000-08:00

Congratulations to two of our government customers, Supreme Court of Louisiana and Government of Bermuda. During the BPM Awards and Technology Showcase in Nashville this week, they will be named silver winners of the Global Excellence in Business Process Management and Workflow Award.

We are naturally proud to be part of this success. We are seeing strong adoption of BPM across governments and public sector entities across the world and it's great to see some of these innovative organizations get credited for their hard work and bold bets.

We did the hard work for you. Read all about it our new State of the BPM Market report

2008-02-20T14:17:00.000-08:00

BPM has become a fast-growing market with lots of hype and lots of real trends to keep up with. It can be quite overwhelming to chew through all the research that is published in a never-ending stream.

Therefore we decided to help out. We have synthesized all the research from the last 12 months to give you an aggregate picture of where BPM as a discipline is today. And we did more than that. We also surveyed our own customer base. With over 200 responses from people who are doing BPM today, this is one of the biggest surveys of its kind. We asked them questions about ROI drivers, top product selection criteria, biggest challenges and more.

The report is available for free here.

For more overview info, read the press release

Technorati Tags: BEA, BPM

Installing AquaLogic BPM Enterprise Server on Amazon EC2

2008-02-18T21:37:00.000-08:00

This post details how to install AquaLogic BPM Enterprise on Amazon EC2. But before you get all excited let me point out the limitations up front:

You will be installing a trial version limited to 60 days and not suitable for production use.
A live instance of this installation will only keep its data until you shut it down. There is no persistent storage outside of the instance and therefore you will lose all process data in a shutdown (intended or unintended). This can actually be an advantage in some scenarios. I will discuss this in a later post.

With the disclaimer in place we can now get started. As a prerequisite you need to

Set up an account for Amazon EC2 and S3
Download EC2 tools
Generate certificate and encryption keys
acquire a basic understanding of EC2

The best way to accomplish this is by following the "Getting Started Guide" on the EC2 Developer Connection Site. From the front page, click on "Get Started" and follow the instructions in the guide. Once you have completed the guide you are ready to go.

Start with a base image

I use the basic fedora-core4 image provided by Amazon, but others will probably work too. Get the handle of this image:

ec2-describe-images -o amazon

This will return a list of images including one looking something like

IMAGE ami-XXX ec2-public-images/fedora-core4-base.manifest.xml amazon available public

The ami-XXX is the handle. Now boot this image with the command:

ec2-run-instances ami-XXX -k <keypair>

where <keypair> is the encryption key pair you created in the Getting Started Guide. Once booted (check with ec2-describe-instances), log in using ssh as you configured it in the Getting Started Guide. In the Getting Started Guide you authorized access to your default group on port 22. You will need to open up access to the following ports as well: 1521 (Oracle DB access), 8080 (Oracle web console) and 8686 (ALBPM Enterprise web applications). Use:

ec2-authorize default -p 1521
ec2-authorize default -p 8080
ec2-authorize default -p 8686

If you want, you can revoke 1521 access once you're done with configuration.

Set up hostnames

Every time you load a new EC2 instance of your image it will get a new internet address, hostname and domain name. Everything is pretty dynamic in EC2. But the apps we're installing here need to know what those names are. So we need to set something up to make this possible.

First pick a local hostname for your image. This won't change and is just an internal name. Let's pick "cloud1" for this exercise.

Now let's create a script that runs at boot to set up the local hostname and also adds our internet hostname to the hosts file. The script looks like this:

echo 127.0.0.1 `curl http://169.254.169.254/latest/meta-data/public-hostname 2> /dev/null` cloud1 localhost > /etc/hosts

hostname cloud1

The first line uses Amazon's metadata service to find the internet hostname for the instance and creates a new /etc/hosts file that contains this information. The second line sets the instance local hostname to cloud1.

Save this script in a file called "setEC2hostnames" in the /etc/rc.d/init.d directory. Then change directory to /etc/rc.d/rc4.d and create a symbolic link:

ln -s ../init.d/setEC2hostnames S79setEC2hostnames

This will make sure this script is run just before the Oracle database starts.

Download and install Oracle 10g XE

AquaLogic BPM Enterprise requires a database and we will use Oracle 10g XE. This is a free version of Oracle's database that you can download from their web site. All you have to do is create a free account. You should download Oracle 10g Express Edition (Universal) for RedHat. Here is a deep link that may break in the future.

There are several ways you can get the download package onto your EC2 instance. I used Firefox to navigate through Oracle's web site all the way until the download begins. Then I do the following:

Cancel the download
Copy the URL of the download package onto the clipboard
Use scp (secure copy) to copy the cookies.txt file from Firefox onto the EC2 instance. on Windows, this file is in C:\Documents and Settings\<user>\Application Data\Mozilla\Firefox\Profiles\<some name>\cookies.txt.
On the EC2 instance use wget to download the installer package directly with the command
> wget --load-cookies cookies.txt <paste URL>

Now the installer package will download directly to the EC2 instance which is both faster and cheaper (you pay per transfer).

Before you can run the installer, you will have to do a few other things. You need to download and install libaio which is an asynchonous IO library. I googled it and found it on rpmfind.net and installed with the command:

rpm -Uvh ftp://rpmfind.net/linux/fedora/core/4/i386/os/Fedora/RPMS/libaio-0.3.104-2.i386.rpm

Next you will need to set up more swap space. The installer requires more than the default swap space. This requires 3 steps:

Create a swap file with the dd command:
dd if=/dev/zero of=/mnt/swapfile bs=1024 count=1310720
Make it a swap file with:
mkswap /mnt/swapfile
Turn on the swap with
swapon

Now you're ready to run the installer. It's a simple RPM package followed by an additional configuration step:

rpm -Uvh oracle-xe-univ-10.2.0.1-1.0.i386.rpm
/etc/init.d/oracle-xe configure

The installation guide has additional information if necessary, but the default options in the installation script should be good enough. Make sure you configure it to start up at boot time.

Check that the local hostname ("cloud1" or whatever you picked) is the one used in the listener.ora and tnsnames.ora files in the /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/admin/ directory. It should not be "localhost" or something like "domU...". If the hostname is not cloud1, then edit the files and restart the database. You can start and stop the database with

/etc/init.d/oracle-xe start
/etc/init.d/oracle-xe stop

Next, go to the APEX admin web console on :8080/apex">:8080/apex">http://<your-EC2-hostname>:8080/apex, select "Manage HTTP Access" under the Administration drop-down. Set HTTP access to be "Available from local server and remote clients".

Download and install AquaLogic BPM Enterprise

AquaLogic BPM Enterprise (ALBPM Enterprise) is the BPM server that runs business processes as. You should already have AquaLogic BPM Studio installed on your own laptop / desktop and you should also have AquaLogic BPM Enterprise installed there because you will need it for this installation. AquaLogic BPM Studio is used to model, implement and test business processes.

Use the same trick as described above for downloading ALBPM Enterprise.

Start at the ALBPM download page,
Pick the latest version of ALBPM Enterprise, e.g. AquaLogic BPM Enterprise Server 6.0MP2 - Standalone
Pick RedHat Linux (4.x, x86) 32-bit
Log in or create a new account if you don't have one
Continue until downloading, copy URL to clipboard, cancel download and do it directly from the EC2 instance with wget like before. You will need to copy over your cookies.txt file again.

Make the downloaded package executable by executing

chmod +x ALBPMEnterpriseSA602_lin_jvm32.bin

Run it in console mode:

ALBPMEnterpriseSA602_lin_jvm32.bin -i console

Configure AquaLogic BPM Enterprise

For this next step, you will need to use your own desktop/laptop installation of ALBPM Enteprise:

Start the Enterprise Admin Center from the Start Menu
Select "Configuration"
Select "Add..." to add a new directory configuration
Leave defaults on the next two screens and click Next
In "Directory Provider Selection" select an admin password. Leave Directory Provider the default which should be "BEA's Oracle Driver..."
On the next screen"
1. Set host to your EC2 internet hostname
2. Pick a password
3. Set SID to XE
4. Leave the rest unchanged
Provide the dba user and password that you set for your database installation
Click next twice until on the "Process Engine Provider Configuration" screen. Set passwords and leave the rest unchanged.
On the next screen provide the Oracle dba system user and password again and click next.

Now the configuration tool is setting up the users, schemas and data used by ALBPM Enterprise. It is also creating a set of configuration files locally that you must now copy over to your EC2 instance. The files are:

ALBPM_HOME/enterprise/conf/default.dirxml
ALBPM_HOME/enterprise/conf/directory.xml
ALBPM_HOME/enterfuegoautostartEnterpriseprise/conf/enterprisePreferences.xml

Copy those files over to the same location on your EC2 instance. Once copied over, you will need to manually edit them and change the host name from the internet hostname you provided during configuration to "cloud1", the localhost name you provided previously.

The directory.xml file also needs to be copied into the WEB-INF folder of all the web applications. They reside in ALBPM_HOME/enterprise/webapps. Use something like

cd ALBPM_HOME/enterprise/webapps
cp ../conf/directory.xml workspace/WEB-INF/.

Repeat the cp command for every webapp.

The final step is to set up autostart of ALBPM Enterprise. Copy the autostart script into /etc/init.d (and rename it):

cp ALBPM_HOME/enterprise/bin/fuegoautostartEnterprise /etc/init.d/albpm

Edit the script and set FUEGO_HOME to point to ALBPM_HOME/enterprise and set FUEGO_SERVICE_USER to root (it should be straightforward to set up another user if you're paranoid).

Now go to /etc/rc4.d and create a link to the script:

ln -s ../init.d/albpm S81albpm

Since Oracle is S80, S81 will ensure that ALBPM Enterprise will start after the Oracle database. Also set up stop scripts in /etc/rc0.d and /etc/rc6.d:

ln -s ../init.d/albpm K04albpm

Test it

Now you should be all set. You can test the instance by rebooting it:

shutdown -r now

This will reboot the instance but will not wipe any of your changes to it (remember if you terminate the instance using EC2 tools, then everything you just did will be lost).

If everything goes well, Oracle and ALBPM Enterprise should start up, and you should be able to access the Oracle APEX interface on:

http://<your-EC2-public-hostname>:8080/apex,

the ALBPM Admin Console on:

http://<your-EC2-public-hostname>:8686/webconsole

and the ALBPM Workspace on:

http://<your-EC2-public-hostname>:8686/workspace

Create a new image

There's quite a few tricky things that can go wrong, so I am sure you will have to troubleshoot somewhere in the process. But now to the neat part.

Once you have everything working, you can create an image of your new instance and you can then bring up a cleanly installed new instance any time you want. You can even share it with your friends!

The Getting Started Guide explains how to create a new Amazon Machine Image. Just make sure you have shut down Oracle and ALBPM Enterprise and cleaned up the disk space (delete installer files) before you do it.

RSS Feed for ALBPM hot fixes

2008-02-08T16:24:00.000-08:00

Feed43 is a nice little service for scraping web pages and turning them into feeds.

I scraped the ALBPM Studio hotfix download page and turned it into a feed, so now you can get notified in your feed reader when new hot fixes become available. Get the feed here:

http://feed43.com/albpmhotfixes.xml

Six Sigma is like so tired

2008-02-05T07:06:00.000-08:00

(when it comes to BPMS)

I didn't study Six Sigma that hard, but since it's not really rocket science I'll pretend I'm a black-belt for a moment. It's about minimizing the variance of your output measured by some appropriate parameters (known bugs for example if you're in software). Six Sigma as a discipline (invented at Motorola) is about implementing this focus across the company and managing towards this kind of goal.

While this stuff appeals to me, I have not been able to figure out how it could work out in the software industry (my line of work). After all, this place is not easy to predict and put on formula. It relies heavily on ideas, trial and error, "tinkering" and other non-linear types of progress (think no progress at all most of the time, then suddenly a break-through that puts you back in black). Some level of control is obviously crucial, but will Six Sigma ever be a dominant management form in the software industry?

I don't think so.

Six Sigma assumes that deviations in a production process follow a normal distribution. But this is not really guaranteed. A normal distribution of errors typically occurs when the cause of the error is truly random. In most processes, this is not the case, but many manufacturing processes are close enough and they have a high n (units produced) which increases the certainty of observations. E.g. GM sold 3.87M vehicles in 2007. That gives them a pretty good base to measure unit production quality and variance. If you only ship 5-10 products a year (like a software company), then you can't measure your variance very well even if it was normal-distributed.

And the biggest problem here is that errors in software production processes does not follow a normal distribution. For example, what is the probability of:

1) Your software product shipping late, given that your are currently on schedule?

2) Your software product shipping 2 months late given that it's already one month behind schedule?

You will likely find that your probability of shipping 2 month late is higher when you're already 1 month late than the probability for shipping 1 month late when you're on schedule. This is contrary to normal error distributions.

This makes it hard (read: not really helpful) to apply Six Sigma and related principles to software production.

This is important beyond just the software industry. Software production is an example of a larger (and growing) class of knowledge-intensive production processes. Any knowledge-intensive business will exhibit similar characteristics and hence Six Sigma will apply poorly to this whole class. Tom Davenport wrote an interesting article, "Why Six Sigma Is on the Downslope" that touches on this issue.

This is significant because in the developed world, knowledge-intensive businesses are growing much faster than the rest. McKinsey showed in 2005 [requires for-pay subscription] that the US economy created 4.5M new knowledge-intensive jobs (what McKinsey called "tacit interaction" jobs) from 1998 to 2005. During the same time, "transactional" jobs (think routine office work) grew by 1.9M and "transformational" jobs (think factory worker) declined by 2.9M.

In my line of work (peddling BPM software) this has additional significance. BPM technologies (such as AquaLogic BPM) is primarily applied to processes that hasn't already been automated. While the manufacturing sector has done a pretty good job at automating its processes, the services sectors such as banking, insurance and government are only now beginning to optimize processes and these sectors are big users of BPM. They also happen to have much more knowledge-intensive processes than the manufacturing sector.

So, to make a long story short. Because Six Sigma doesn't apply well to knowledge-intensive processes and because these are exactly the types of processes that are being automated using BPMS technology, Six Sigma is rarely a driving initiative behind BPMS adoption.

Technorati Tags: BPM, Six Sigma

At Gartner BPM Summit, Las Vegas

2008-02-05T06:50:00.000-08:00

The Gartner BPM Summit, Las Vegas 2008 has been kicked off and I will be here for the whole show. You can find me and the rest of the BEA crew at our booth during booth hours.

If you want to read about the sessions at the show, Sandy Kemsley usually does an excellent job at covering the conference on her blog. Sandy will be happy to find out that there is wireless here. It is working in the ballroom I am in right now. Hopefully it will be consistent throughout the area.

Technorati Tags: BPM, BEA, Gartner