ProcessWorks Consulting

Good Design Documentation

Scott Altham — Sat, 14 Apr 2012 12:50:47 +0000

Ok, that’s a relatively general title. What I mean specifically and what I want to talk about here is the importance of a good enough technical design specification and how the quality of said document really does impact the resulting solution. I want to share some recent experiences where documentation really did let the side down.

I’ve not long finished a 6 month placement at a company working on Cordys C3 based bug fixes and change requests. It was challenging in 3 respects.

Firstly, the work was specific to Business Processes and XForms based user interfaces and both of which where not exactly organized within Cordys in a very intuitive and structured manner that provided any contextual reference to how the business solutions worked together. Second, the new change requests being received really required a good knowledge of the existing 300+ processes and forms that made up the applications that ran on Cordys. This was because design documentation was less than adequate and in six months, you really can’t get a detailed enough grasp of such a large number of processes, especially non-standard business processes in the world of media management and digital content distribution. Finally, the loaded web service interfaces (and so the services themselves) included operation parameters such as arg0, arg1 and arg2, which as you’ll have no doubt evaluated, is unbelievably unhelpful in determining what the service operation actually does and what data should be provided in the messages.

OK, these issues aside for a moment, the real issue I want to discuss was how the design documentation should go some way in explaining which components, services and forms should be used or designed for the given solution, what dependencies are outstanding at the time of writing and what risks may be involved. I worked on a couple of CR’s and the documentation was poor. This was through no fault of the architect however, who knew the systems inside out and was stretched so thinly, that their desk was only visited once every 2 days when meetings were cancelled. Poor and constantly changing requirements where no help either.

In order for developers to attach time estimates to tasks detailed in a design specification document, there must be enough detail so that the estimate confidence level of such tasks can be as high as possible and thus the business is kept more realistically informed of how long the solution might take. The problems I found with the documentation in this instance where as follows:

- Very brief document objective
- Zero mention of any outstanding dependencies at the time of writing (i.e. are services required by this functionality written yet? Some where not)
- ‘To Be’ process illustrations made up the bulk of the document, with very little supporting written description.
- Message interface definitions where non-existent
- Not all of the requirements were met in the design document
- No exception handling or compensatory actions where detailed in the event of errors
- No architectural overview was presented and minimal system interactions (i.e. sequence diagrams) where present

In short, the design specification put the responsibility on the developer to fill in the gaps in detail. Whilst this free’s up the architects time, this really is no good for a few reasons:

1) Not all of the design is documented and therefore cannot be referenced in the future if the client questions functionality or attempts to request amendments outside of a new CR
2) New developers to the team (myself in this instance) are left with gaps in their knowledge that requires time consuming investigation with potentially mutliple teams (and as such consumes estimated time but no development is done)
3) Gaps in design specifics can lead to incorrect assumptions about how the solution should operate
4) Inadequate design detail leaves room for (mis)interpretation of the design and can mean solutions move away from company design standards and architectural rules. This leaves a messy set of solutions that operate differently, don’t really utilize re-use and only further confuse developers.

In this case, clearly the company may not have the resource to focus more on detailed documentation or maybe they believe it’s just not as important as I do. The bottom line however is that if you are going to develop a solution that’s more complex than ‘Hello World’ you should really think about documenting the following (and I apologize in advance if you are great at your design specifications) :

- Start with a document summary. This should includes, author, distribution list, document approvers and release history.
- Basic I know, but include a ‘contents’ section that logically breaks up the design into layers (data access, service, process, ui).
- Provide a detailed overview of the solution. Detailed being the key word here. Copying chunks of text from the functional specification is not cheating. The overview should include how the solution will improve any existing solutions (i.e. improve stability, boost usability, provide a greater level of system managability)
- If necessary, provide details of the runtime environment and any antipicated configuration changes
- Make reference to any other materials such as the functional specification and use case documents
- Include design considerations (company or otherwise)
- Detail any known risks, issues or constraints
- Detail any dendancies at the time of writing. This should include any work being performed by other teams that the solution being detailed requires in order to operate successfully.
- Provide a top level architectural diagram, even if the solution is basic. Diving into the detail without giving the developers a 1000 foot view of where the solution fit’s in to the wider solutions architecture to me is just wrong. Support the diagram with a sentance or two.
- List the components that will change as part of the design
- List any new components
- Diagram component interactions (sequence diagrams)
- ‘To be’ process designs should include annotation, even if it’s assumed current knowledgable developers would know this information. You will not always have the same people doing the work.
- For UI’s, detail the style and appearence ensuring it’s in line with company accessibility policies. That may require detailing fonts and hexidecimal colour references [#FFFFFF].
- Detail what programming may be required. Server side, client side. What functionality should this code provide. What coding best practices should be honoured.
- Keep a glossary of terms to the back of the document

Finally and most importantly, even if you are the architect and the master of all knowledge when it comes to your solution domain… Distribute the document and enable change tracking. Send to the subject experts for clarification and the business stakeholders, even if they don’t understand the content.

Most of us do and that’s great. Design specification templates can be found online so there’s really no excuse.

SOA. Just the basic facts. In 5 minutes.

Scott Altham — Sat, 14 Apr 2012 10:16:52 +0000

What a SOA is not

SOA does not mean using web services.
SOA is not just a marketing term.
SOA also does not just mean ‘using distributed services’.

What a SOA is

SOA is an architecture of business services (usually distributed and sometimes ‘connected’ by means of a service bus) that operate independently of each other, advertise what services they offer through well-defined interfaces and can be heavily re-used not only to aid development productivity of the IT department but to enable use of existing IT assets/systems. Ultimately, this means quicker turn around of IT projects, improving how IT serves the business and thus improves business agility.

‘Service’ orientation means that logic is provided via a suite of services which should be seen as ‘black boxes’. They do stuff, but you or the services consuming them don’t need to know what’s going on under the hood, only what messages they require as input (usually SOAP messages) and what the service will do for / return to you. A black box service doesn’t have to be a web service, though they are the most commonly implemented type of service for maximum distribution and cross-platform compatibility.

So whilst that goes some way in explaining what SOA is on a general level using these developer written ‘services’… What SOA really is, is A FUNDAMENTAL CHANGE IN THE WAY YOU DO BUSINESS via a top down transformation requiring real commitment from the business, not just IT. That requires a change in mind-set of the top people.

Characteristics of a black box ‘service’ in a SOA

- Loosely coupled (minimizes service dependency)
- Contractual (adherence to well-defined service interface contracts… ‘if you wanna do business you need to abide by my interface contract’)
- Abstract (service is a black box, internal logic is hidden from service consumers)
- Reusable (divide and conquer! – We divide up business logic to basic reusable services)
- Composable (can be used as a building block to build further composite services)
- Stateless (retains little to no information about who it interacts with)
- Discoverable (describes itself, a bit like a CV so that the service can be found and assessed ‘hello I’m a service, here is my CV’)

What can these services do?

Whatever you need them to do in order to satisfy business change needs / requirements. Common functions of services include:

- Perform business logic
- Transform data
- Route messages
- Query data sources
- Apply business policy
- Handle business exceptions
- Prepare information for use by a user interface
- Generate reports
- Orchestrate conversations between other services

The business benefits of implementing a SOA strategy

- Open standards based
- Vendor neutral
- Promotes discovery
- Fosters re-usability
- Emphasizes extensibility
- Promotes organizational agility
- Supports incremental implementation (bit by bit development)

What a SOA might look like

The below shows a business application based on a SOA.

The lowest level of operation consists of application logic, including existing API’s, DAL code and legacy systems. This may include ‘application connectors’ that are middle men that interface between a simple exposed API and large systems like ERP, MRP etc).
This low-level application logic is then exposed as basic level services (application orientated services as they are a wrapper for parts of the application logic).
These basic level services form the building blocks of composite level services. More application aligned services are combined together to form services that are more aligned with the business, thus are more business orientated services. This can include exposing a business process as an independent business service.
Basic (application orientated) and composite (more business orientated) services can then be orchestrated by business processes.
These business processes may include human interaction points where user interfaces are required. Processes can also be initiated via user interfaces (requests / orders / applications etc).

Steps in Implementing a SOA with web services

1) Creating and exposing services (development team creating component services)
2) Registration of services (SOA isn’t truly in place when you just have random web services sitting on different web servers exposing WSDL. Where services are just consumed based on word of mouth and passing WSDL documents around. A SOA requires a directory of its available services where all available services can be registered. UDDI 3.0 being the standard when using web services. This directory is the yellow pages for the services).
3) Address security. Exposing business logic as services over large networks opens up a serious set of security challenges. Security standards must be implemented for the services so that consumers of services can meet the security requirements.
4) Ensure reliability. Services must be monitored to make sure they are always up (high availability) and performance must be monitored to ensure reliability.
5) Concentrate on governance. How are all of these steps governed? What are the policies that should be enforced at runtime. Compliance is also important.

SERVICES THAT ARE EXPOSED, REGISTERED, SECURE AND PERFORM WELL FORM A SOLID SOA FOUNDATION.

That’s all for now. Hopefully that paints a very top-level picture of what SOA is, what it is not and how you should go about implmenting it (with the all important business buy in).

Cordys BPM Constructs 101a

Scott Altham — Thu, 29 Dec 2011 11:00:35 +0000

I’m now really quite comfortable with Cordys process modelling and the models component constructs. There are simple differences between certain constructs that I’ve used in other products such as MS Biztalk so I wanted to absorb the Cordys documentation and present a sort of quick and dirty intro to each construct and how Cordys uses them to execute your processes. As there are several constructs and I wanted to go into some detail, I’ll present some of them here and the rest in a subsequent article.

If you don’t already have a copy of the Cordys community edition VM that runs C3 or BOP4 on a CentOS virtual hard disk, I’d suggest getting hold of it so you can ‘play’.

/*Note that most of the constructs can only be used if a user is granted the Business Analyst role, from the Cordys Classic Studio ISV package.*/

Start Event – Only one per process. Can have several trigger types:

- Message (a defined process specific message)
- Timer (by setting a time unit (e.g. days) and no of occurrence
times)
- No Message or Timer

End Event – Process can have multiple end events. Error, Message and Rollback. You define an output message by dragging a ‘Document Type’ from the BPM Components repository to the end event. This then invokes a web service as the process ends. You can specify custom error XML in the ‘error details’ tab of the end event.

Activity - The activity is the construct that represents a generic step in the process and should be configured with an activity ‘type’. It can be of type ‘Application’ (web service calls) or ‘XForm’. When of type XForm, an additional XForm tab becomes available in the activity properties. An activity can also be set to be a ‘dummy activity’ meaning it does nothing. Each
activity can be conditionally executed based on conditional evaluation returning a boolean result.

Decision – The decision is a basic construct that allows multiple mutually exclusive outputs. It’s actual conditions are defined in the connectors that exit the decision construct and Xpath evaluations can use message map data for data driven decisions.

Intermediate Message Event - The intermediate message should have a process specific message defined and is invoked when a message is received to the process. Normally this is used to ‘wake’ a waiting process instance. The gateway inmessage based on the process instance id (ensuring message correlation). For asynchronous web service calls, an intermediate message can be used to receive the response message from a service operation call. Dragging the output message from message map will set this. Therefore it is useful to know whether
the operations you are calling via your activities are sync or async.

Compensate Event - Used to rollback any effects of an activity or context or activities when an exception is caught. A compensate event can be defined for an activity, sub-process,
context (embedded sub-process), for each, while and until loops. An activity or sub-process can only have one compensate event associated. Compensate is represented as two left pointing arrows, side by side in a circle.

Delay Event – An intermediate event that stalls the process for a configurable amount of waiting time. Delays can be of type:
- Fixed delay. That is a set amount days, hours, minutes or seconds.
- Message read delay. That is a delay value read from a process specific message using an Xpath reference. As with a standard activity, a delay may be conditionally executed based on a boolean return evaluation.

Exception Event – An exception is an event which is fired when a Cordys error occurs (e.g. a soap fault is detected via external service responses or via SOA grid messages).
As you would in programming code, you can specify a specific exception name to catch or just catch any exception. Once an exception is thrown, process execution is diverted to the connected exception event where subsequent exception handling activities should follow. There are several error types which can be caught and which are configured on the exception event construct:
- All. All exceptions are caught, regardless of the error code.
- Custom Error. You can set an error code you may expect from a service response indicating an error has occurred (e.g. a SOAP fault
code).
- System Error. Simply put, a Cordys internal error where some part of Cordys may not be operational (a soap processor may be down for example) or syntax rules are broken.
- Communication’s Error. These errors are specific to SOAP faults returned from external service calls. Error code 11 is returned for comm’s failures.

There are error codes that are only raised when a sub-process is being invoked from a parent. These are:
- Process Loading Error
- Process Instantiation Error
- Process Model Not Found Error

… more to follow soon!

Metastorm BPM : It’s not an application development tool

Scott Altham — Sat, 26 Nov 2011 10:47:23 +0000

After 2 years of designing a large operational system using Metastorm v7.6, I wanted to reflect on why it’s a bad idea to use Metastorm BPM to build big workflow based systems.

The problem with the finished system is not that it doesn’t satisfy the requirements or doesn’t perform well enough (considering), it’s that it is a maintenance nightmare. I wrote an article this time last year whilst travelling back home to Holland from being snowed in and which concerned why maintainability is the most important design factor (over and above scalability and extensibility). Coupled with a ‘big bang’ design approach (over an agile dev approach) and consistent requirement changes, it’s a surprise the system runs in its operational state.

I don’t wish to run the product down, because for small to medium workflow driven applications, it does the job. But, it’s clear lack of object orientation is the biggest single product flaw and when building a big system with Metastorm this cripples maintainability. A solid design architecture is obviously of major importance. Basic application architecture fundamentals such as breaking a system design down into cohesive functional ‘components’ that represent ‘concern’ area’s for the application can be difficult to implement. This is down to the fact that process data is stored in the database per process and passing data between processes using flags can become messy, especially when certain characters are passed using those flags (that Metastorm classes as delimiters). Sub-processes are then an option, but these also have inherent flaws.

Forms, which again are application components are process specific, so re-use is again suffering and so replication of forms has to be done, further disabling the idea of good maintainability.

Having data repeated in processes and having no code dependency features is bad enough, but because you have to remember where you have used process variables and keep in mind when and where values may change, the tool puts all the responsibility on the developer. Once the system get’s very large, the event code panels (the ‘on start’, ‘on complete’ etc) get very complicated and tracking variables and when they may change etc becomes a struggle in itself. Changing a variable value in a large process has the risk of making other parts of the process not work quite right because ‘you’ve forgotten that the variable is used in a conditional expression later on’.

This then begs the question, should you even use the Metastorm event/do this panels for ANY business logic. I’d say no. Only UI or process specific logic should be used and you should push ALL of your business logic to server-side scripts and a suite of external .NET assemblies. You can then at least implement a fully swappable business logic layer.

So along comes v9. This product is a great move towards stronger application architectures. OOP design and ability to debug alone save a whole lot of system maintenance time. So although this version takes us closer to being able to create solid, maintainable operational applications, it was released too early. It is slow (halving development productivity against version 7), it had many broken features and grids, one of the most used visual components, especially for data driven applications (which is most business apps) were just terrible. They continue to perform almost independently from the rest of the system and patch 9.1.1 is still addressing grid shortfalls. Obvious shortfalls which should have been picked up by a thorough QC team @ (OpenText) Metastorm.

The new OOP approach means that designers and developers no longer have to use the ‘Line by line interpreted’ syntax of v7 and can design re-usable components. So there is a greater case for using Metastorm BPM as an application development tool for fair-sized applications but whilst development productivity is still poor and the designer is still very buggy, it’s not quite there yet.

Create PDF from HTML

Scott Altham — Fri, 21 Oct 2011 14:45:18 +0000

I’m working on a project at the moment that needs to generate a PDF document from data on the SQL Server. The following steps illustrates the approach I’m using but in this article I want to focus on the HTML to PDF conversion using the C# port of iText, iTextSharp.

Select the required data into a .net DataSet
Create a new XmlDataDocument(DataSet) so that we have our data in XML
Create a new XSLT converter using XslCompiledTransform, load the XSLT file, then transform the XML to the output HTML we need

So the HTML I have is structured correctly and links to images that are required to be included in the PDF output. In order to generate the PDF i went searching for a free PDF API and iTextSharp seems to be the commonly used from what I could tell. I’ve had a quick read through their API documentation and put together a quick example of how to generate a PDF from an existing HTML file (so this doesn’t use the iTextSharp elements). I hope this is useful (click on the image to enlarge):

To invoke the method, you simple pass in the source HTML file location (use MapPath() if on a web server) and where you want your output PDF to be created. Simples.

Cordys BOP4 : Messaging on the SOA Grid

Scott Altham — Sat, 08 Oct 2011 17:55:28 +0000

I’ve spent a few weeks getting used to Cordys BOP 4 and as I usually try and do with a new product, I wanted to know more about what’s going on under the bonnet with it. The central coordinating component of Cordys is its SOA grid, which takes care of messaging between all of the core Cordys services and other web services. Based on the information provided in the Cordys offline documentation and because I’m a visual learner, I’ve drawn up the following image that should hopefully shed some light on how Cordys organises its internal services and how they communicate via the SOA grid. Click on the image to zoom to actual size.

What I’m trying to show here is how Cordys deals with an inbound service request. The dark line represents the path of the message along the service bus.

To illustrate an example of how the above image can be used to understand what Cordys does is the request of an XForm from the Cordys client. The client sends a request to display an XForm so sends a HTTP request to the web server for a resource of .caf file extension. The web server, based on the .caf file extension hands the request over to the Cordys web gateway. The web gateway contacts the LDAP service container and checks for the location of the XForms service container (the LDAP service must always be up and running for proper SOA grid functioning). The LDAP service container has an LDAP application connector which talks to CARS. Next the SOAP request is sent to the XForms service container and the XForms engine takes care of rendering the HTML response. Not only that, but the XForms engine also validates controls against schemas and automatically contacts other web services required whilst rendering. Once the HTML is generated, it is returned via the SOA grid to the Cordys web gateway, then back to the calling client.

I should mention at this point that web services on the SOA grid are called based on the service operation name and namespace in the SOAP request.

This is very high level and it’s always a good idea to read further into the Cordys documentation, but I hope this graphic helps to illustrate the architecture of services, service containers and service groups on the Cordys SOA Grid.

Understanding Multi-tenancy

Scott Altham — Sun, 11 Sep 2011 00:09:13 +0000

I’m doing a lot of research and practical ‘playing’ with the Cordys BOP 4 environment at the moment. It’s a relatively young product from a young company (founded in 2001) but from what I have understood about the product and its architecture it is a strong, versatile collaborative tool that really supports rapid application development and maintenance for business change thus reducing general cost of ownership. I don’t want to talk about the product itself, I will be doing that soon enough, but I wanted to cover one of the software’s best features in my opinion and that is its ability to operate within the enterprise and in the cloud utilizing a fully multi-tenant architecture.

Multi-tenancy is an architectural principle which says that a single install of a software program / server application can service multiple client ‘organizations’, providing a customized isolated application environment with its own data. This is in complete contrast to the idea of multiple instances for each organization (i.e. installing an instance of some server software and underlying database to serve one organization and store only that organizations data). Each ‘organization’ (in the general sense of the word) is classed as a tenant, using the application, therefore if one single installed application can serve multiple tenants their customized view of the application, then it is said to have multi-tenancy (supports multiple tenants). Google apps is a perfect example of a multi-tenant architecture. Multi-tenancy is the foundation architecture for most if not all Software as a Service applications, thus cloud applications support multi-tenancy.

How multi-tenancy is implemented in an application can vary from application to application. The general principle however is to have the application virtually partition portions of its data and services to different consuming organizations. A very simple example at the data level would be to have the application generate a set of related data tables under new database schemas as new organizations are set-up to use the application (so a schema per organization). This separates off the data into logical groups that have their own security context on the database. There are other ways to partition the data, but this is just to illustrate one potential method.

So multi-tenancy is a software architecture and one that is prevalent in cloud applications. Cordys BOP 4 does this very well and I’m looking forward to investigating this product and its virtualization capabilities further.

Metastorm 9 : Ignores ‘Client Paging’ property

Scott Altham — Wed, 07 Sep 2011 00:05:46 +0000

Grids were one of the biggest reasons I upgraded to 9.1 from an application written in 9.0.0.1. Mainly because the ability to wrap text in grids was removed in v9.0 and so columns with large amounts of text to display became unreadable. Now although this has been fixed in 9.1, grids are still throwing up issues. The first is noted in my previous post where you cannot assign a business object field to more than one column of a grid (worked in v9). Second, the grid page size on many (if not all grids) appeared to reset to 50, regardless of what was previously set. Lastly, it appears on deployment that 9.1 is completely ignoring the fact that I’ve unchecked the client paging check-box as I don’t want to deal with pages of data in the grid. The below example shows what is set in the designer and the resulting form after deployment….

It is clear that the client paging property is not checked for this grid.

You can see from the above however that paging is still active on the grid (the pages are shown to the left of the screen grab, but the white space represents the paging bar). Clearing the browser cache doesn’t help in any way. It really makes me wonder how much in-house quality control testing is done at Metastorm.

Metastorm 9.0 to 9.1 – Same key has been added

Scott Altham — Tue, 06 Sep 2011 22:38:26 +0000

So I performed the relatively simple Metastorm 9.0 to 9.1 upgrade this week and for the most part, functionality for existing applications has remained. There are however a few issues I experienced, the biggest being a new message appearing on a few of my forms. ‘An item with the same key has already been added’.

The error kept showing its face on grid heavy forms, especially when you select a row in a grid or where there are buttons on a form and the grids are dependant on other fields. After some trial and error (and guess-work) it became apparent which grid was causing the error to occur and further to that I noticed that the grid had the same business object item assigned to 2 columns. I killed one of these columns, redeployed and all was well. The annoying part is that this worked in v9.0 with no issues. So it seems that in 9.1 you can only assign business object items to a single grid column only.

Metastorm announcing BPM 9.1

Scott Altham — Mon, 11 Jul 2011 09:00:14 +0000

This morning at US Eastern Daylight Time, Metastorm are set to make some product launch announcements including Metastorm BPM 9.1. The other product announcements include Smart Business Workspace 9.0 and Metastorm Business Performance Intelligence Dashboards and the predefined dashboards for Metastorm Business Process Management, Metastorm Integration Manager and Metastorm Knowledge Exchange. These boosting the product(s) visibility to business users.

For me personally, the Metastorm BPM 9.1 release is the most welcome news. I work with this product every day and if you’ve read my recent posts, I’ve shared my concerns about its general stability and poor performance. It sounds like many new features in this release surround the client UI and building of forms with this version (hopefully!) introducing Panels (collapsible groupings of form controls), Field Anchoring (spacing your form content when window maximized), improved field visibility options, dynamically visible actions and I’m sure many others.

Many Metastorm BPM developers are looking for big improvements in this release and are hoping that Metastorm has made the development experience more efficient. It has taken some getting used to the new way of design and it can be tedious at times, so any new features that boost development efficiency I can see being received with applause by the development community.

Let’s wait and see what comes to pass.