The Enterprise Architect

10 things you should know about Model Driven Development

Mon, 09 Nov 2009 20:04:00 +0100

Last saturday I gave a talk at the Devnology community day about Model Driven Development (MDD). I have talked about ten things you should know before you start with MDD. It was an introduction to MDD with some highlights of more advanced topics. In this article I share the slides of my presentation including a short explanation of each of the 10 points.

1. Differs from model-based development

Using a model in your software development approach does not mean you are doing Model-Driven Development. The key concepts of MDD are abstraction and automation. The model of a software application is defined on a higher abstraction level (than 3GL languages) and this model is converted into a working application using automated transformation or interpretation.

2. Is still a software development approach

When using MDD do not forget you are still building software. See 8 reasons why MDD is dangerous for a more detailed explanation of the points. One additional point I mentioned is the law of leaky abstractions. This law coined by Joel Spolsky states: All non-trivial abstractions, to some degree, are leaky. I do not fully agree with the full article about this law, but it is true that if you use MDD and model on a higher abstraction level, sometimes you will need knowledge about the details 'behind' the model. For example to optimize the performance of the resulting system.

3. Asks for agile development

To get all advantages of MDD you need to use it in an agile way. Involve all stakeholders in the development process, use short iterations, and do not forget that YAGNI also holds for modeling. Because it is so easy to add functionality when using MDD, you will not be the first one ending up with a 'spaghetti-model'. MDD, on the other hand, also enables agile development. MDD enables short iterations and makes it possible to show the results of a model change almost directly in the working application.

4. Needs a standardized architecture

MDD needs a standardize architecture. The functional principles of the architecture are enclosed in the Domain-Specific Languages (DSLs) and the associated model validations. The constructional principles are reflected in the code generator or interpreter. On the other hand, MDD really enforces the architecture. Each architecture principle is guaranteed to be followed by the resulting applications.

5. Asks for different tools

See Model Driven Engineering tools compared on user activities for a detailed explanation of this point.

6. Leads to a change in roles

See roles in Model Driven Engineering for a detailed explanation of this point.

7. Can lead to business-IT alignment

By using domain-specific concepts, involving domain experts in the development process, and by enabling shorter iterations MDD can lead to more business-IT alignment. This alignment can possibly become more explicit by using a framework for Model-Driven SOA.

8. Needs another business model

MDD needs another business model than traditional software development approaches. One example is that companies cannot keep selling development hours. They can of course keep doing it, but in most cases you will need less hours to develop a solution...

9. Is not enough

We need more than MDD! We need to integrate modeling and programming. See this presentation for an excellent view on this topic. We also need to focus more on changing applications. Just building applications very fast does not solve our problems. We need to be able to adapt applications to changing business requirements with preservation of runtime data, long running processes, etc. Another point of attention is the runtime flexibility of applications. Due to the fact that MDD enable fast changes in applications a lot of the resulting applications are not flexible themselves. A change in your model means however that you need to test everything again. What we still need is the ability to change an application at runtime in a controlled way. For example: changing security rules, GUI elements, rules, etc. One solution direction is to use adaptive modeling (I will write more on this topic in a future article).

10. Leads to resistance

MDD leads to changes in roles, tools, and business models. Not everybody likes change...

The slides:

10 Things You Should Know About MDD

View more documents from Johan den Haan.

Architecture and Engineering in Business Engineering

Tue, 27 Oct 2009 20:47:00 +0100

It has struck me that many discussions about business-IT alignment and enabling the involvement of the business in software development still only talk about solution domain concepts: SOA, WOA, REST, web services, cloud computing, etc.

I think the question should not be what technology to use, but how we can create an IT landscape truly supporting the business part of an organization. This is not a trivial question; because it is questionable whether enterprises can actually maintain a focused strategy long enough to align their core business processes with IT. The current dynamic business environments do not give enterprises that time.

Article highlights

We can create more business-IT alignment and agility by moving from software engineering to business engineering.
Business engineering: software built by people who understand the business and who are able to express that knowledge in formal models.
Necessary elements for business engineering: MDE with model execution and SOBA.
Overview of a tool supporting business engineering.
Future developments: integrating engineering models and architecture models.

Business Engineering

Engineering a software application has two different perspectives: the perspective of the function, or what the application is supposed to do, and the perspective of the construction, or how the application is being made. The first perspective is the one of the user, e.g. the driver of a car. The car driver does not need to know how the engine works, he just needs to know how to handle the car in order to travel; he knows everything about where he wants to go to. The second perspective, however, is the one of the engineer, e.g. the car repairman. He needs to know how to construct or reconstruct a broken car engine. This is quite a fundamental distinction that can be used to define architecture, design, and engineering in a formal way as Jan Dietz has done in a scientifical setting.

Most software is made from a constructional perspective. Software engineering as such can be described as creating a low-level model of a software system, expressed in a programming language (e.g. Java, C#, etc.). In essence, a programmer specifies the construction of an application; he describes how the application should work. He is usually doing that on the basis of a "functional" design which describes the what or function of the application. Functional designs, however, are usually not very precise, do not contain formal models, and leave many functional decisions to the programmer.

The idea of business engineering is to have the business in the lead of application development. Instead of creating models that describe how an application should work, we should create precise and formal models that describe the function of an application. These models should describe what the application should do. The construction should follow logically out of these functional models.

The idea of business engineering is not new. OMG's Model-Driven Architecture (MDA) also describes models focused on the function of a system without specifying technical details. However, the MDA defines that these models should be transformed / refined into lower level models until a model is produced from which code can be generated. The idea of business engineering is to have engines directly executing the high-level models, without any further refinements. So, the model is the solution. Whereas MDA is automating the model transformation process, Business Engineering automates the interpretation of the functional models.

It is important to note that business engineering leads to a change in roles in the software development process. Software engineers have to switch to working on the tools and platforms to enable business engineering, while business solutions are built by so-called business engineers (i.e. non-programmer domain experts). Business engineers are people who understand the business and who are able to express that knowledge in formal models.

Business engineering has the following advantages:

Due to the use of declarative models (i.e. models as much as possible focused on the function of a system, for example business rule models), business engineers can be involved in the development process.
No complex transformation from model to code.
Models and code cannot be out-of-sync.
Changing an application is just changing the model.
Understanding the behavior of an application just asks for reading the models (instead of source code).
Debugging an application means debugging the models (i.e. debugging in terms of business models instead of source code).

Because of these advantages business engineering can bridge the gap between business and IT: applications can be built faster, applications are easier to check for compliance with requirements, and applications are easier to change.

I hope you understand the advantages of business engineering, but I will not be surprised if you think this is just a nice dream which is not possible in practice.

MDE and SOBA, necessary assets for Business Engineering

The first thing we need to enable business engineering is a Model Driven Software Development approach focusing on model interpretation, i.e. directly executing high-level models. To make models both high-level (i.e. no technical details) and directly executable we need Domain-Specific Languages (DSL).

The idea of a DSL is to have a small language with a limited number of concepts for a specific domain. This domain can be a problem domain (e.g. insurance, healthcare, transportation) or a system aspect (e.g. data, presentation, business logic, workflow). If system aspect DSLs can be made high-level enough, they are also usable by non-programmer domain experts. To make them executable we need engines directly executing models expressed in a DSL.

The second necessary asset for business engineering stems from the need for easy to change applications delivering business services. The definition of ‘application' is rapidly changing nowadays. As the industry is moving from using an application-centric architecture to a Service-Oriented Architecture (SOA), the focus for building functionality is moving to Service-Oriented Business Applications (SOBA). This means that applications are a set of services and components working together in fulfilling a certain business need. An example of this movement is the upcoming Service Component Architecture (SCA) standard.

With SOBAs you are not only flexible in changing the high-level orchestrations, but also the lower level service orchestration within ‘applications'. Business engineering needs SOBAs because it enables to change applications in an easy way. Services can be changed independent from each other, as long as the interface stays the same. Furthermore, we can define a service type for each DSL. In that way the engines executing the DSLs can be fully separated and can make use of different technologies. For example, a Business Rule DSL can be made executable with a Business Rules Engine, publishing decision services. Another example: a Reporting DSL can be made executable with a Report engine, publishing reporting services. In this way we have the same separation of concerns at runtime (with services) as we have at design time (with DSLs).

Summarizing we can say that for enabling business engineering we need an approach to Model-Driven SOA. Model-Driven SOA enables the creation of an IT landscape truly supporting the business part of an organization. Due to the use of a Model-Driven approach based on model interpretation, this landscape can be changed fast enough to catch up with the current dynamic business environments.

Business Engineering in practice

We of course need tools to support business engineering. In general a separation can be made between DSL tools and Model-Driven Software Factories. DSL tools are used to define the needed domain-specific languages, with which a Model-Driven Software Factory can be constructed. A Model-Driven Software Factory is a complete platform consisting of a set of DSLs to model a business solution and the interpreters needed to execute the models.

Mendix is an example of a Model-Driven Software Factory for creating Service-Oriented Business Applications (SOBAs). This factory consists of a Business Modeler, a Business Server (the runtime environment), and a rich web 2.0 client (providing the user interface). With the Business Modeler non-programmer domain experts can model a SOBA. They can make use of the following system aspect DSLs:

Business Object DSL
Rich Internet Forms DSL
Microflow DSL
Business Rules DSL
Report DSL
Services DSL
Security DSL
Mapping DSL
Internationalization DSL

Because each system aspect is covered by its own DSL, these languages can be very simple, and thus easy to learn. Each DSL is at least readable for non-programmer domain experts. Most of them can also be used by business engineers (i.e. non-programmer domain experts) to model parts of the solution they are building.

The modeling environment guards the consistency of all the models. If the models are consistent they can be deployed to the runtime environment (‘one-click-deploy'). No further refinements or transformations are needed and no code is generated. For more details, including an overview of the architecture and how Mendix fits in an existing application landscape, see the whitepaper "Modeling the Agile Enterprise".

Future Developments

This toolset is a huge first step in the direction of business engineering. However, we need to go further. As Peter Bakker stated in his article on Via Nova Architecture: architects and engineers have to work together to understand complex systems. They both have to use models. As shown in this article the (business) engineering part of software development is possible right now. An open future question is what role (business) architects can (and should) play in business engineering. I think architecture models and engineering models should be put in a central repository, thereby enabling all kind of automatic consistency checking between them. For example, a change impact analysis on implementation models when the process documentation changes. Another possibility would be to define architecture principles in formal way, enabling automatic enforcement of these principles on engineering models.

Mendix and Bizzdesign have started the integration between Bizzdesigner and the Mendix Business Modeler, aimed at bridging the gap between business process documentation and business process implementation. The most important advantages of this integration are the possibility of tracing between process documentation and implementation elements and the possibility to get an automatic change impact analysis, i.e. when the process documentation changes the implementation elements affected by the changes are reported.

To finish, some questions for you:

What do you think of Business Engineering?
How do you see the role of the architect in Business Engineering?
How should the architect be supported with tooling?

Photo by dlemieux

An Enterprise Ontology based approach to Model-Driven Engineering

Thu, 15 Oct 2009 20:55:00 +0100

Today I successfully presented the results of my thesis at the Delft University of Technology. The goal of my research was:

Design an MDEE approach based on a sound theoretical foundation, providing end-to-end guidance to refine and transform an organization model into an IT system supporting that organization.

MDEE is the abbreviation of Model-Driven Enterprise Engineering, which is the name of the Model-Driven Engineering approach resulting from my research. On this blog I mostly call such an approach Model-Driven SOA. See for example my posts: SOA is dead; long live Model-Driven SOA and A Framework for Model-Driven SOA.

In this article I want to give you a short overview of the results of my research. I have also added the slides of my presentation.

Why another research on Model-Driven Engineering?

Because of the lack of a significant increase of productivity in the last 20 years we are still in a huge need for increasing the return a company derives from its software development effort. Model-Driven Engineering (MDE) aims to raise the level of abstraction in application modeling and increase automation in application development, thereby increasing the productivity in software development.

The Model-Driven Architecture tries to define an MDE approach, but is mainly focused on technical variability and lacks formalism. Other approaches, using Domain-Specific Languages, do not define a process or framework at all. Research in this area is focused on language engineering and multi-modeling. Literature on formalizing MDE is focused on defining the concepts and assets needed to construct an MDE framework. We only know one attempt on formulating an end-to-end (from business model to IT implementation) MDE approach, but the resulting framework does not have a theoretical foundation (see the science of model-driven SOA).

Concluding we can state that no end-to-end MDE approach exists describing abstraction layers, models, modeling languages, and transformations, based on a formal, theoretical foundation. My thesis presents an MDE approach based on a sound theoretical foundation, providing end-to-end guidance to refine and transform an organization model into an IT system supporting that organization.

The Model-Driven Enterprise Engineering approach

The theory of Enterprise Ontology constitutes the basis of my Model-Driven Enterprise Engineering (MDEE) approach. MDEE is also based on the Generic System Development Process which states that a system development process consists of five steps: reverse engineering, function design, construction design, engineering, and implementation. Figure 1 gives an overview of MDEE.

Figure 1 - The Model-Driven Enterprise Engineering approach

The first step in our MDEE approach starts with interviews and the documentation of an organization and results in an organization model. We call this reverse engineering.
Using service identification a set of services is identified which are needed to support this organization. These services are specified in the service specification step, resulting in a service specification model. Service identification and service specification together form the function design step of our MDEE approach.

The next step in our MDEE approach is construction design. As shown in Figure 1 this steps consists of two parallel tracks. Service orchestrations are derived from the previous models based on a set of derivation rules. Another set of derivation rules defines how to derive BCI3D input tables from the previous models. The component identification step, implemented with BCI3D, identifies a set of components based on these tables. The identified components are specified in the component specification step. The model created in the construction design step is a business component construction model and consists of service orchestrations and a component diagram.

In the engineering step the business component construction model is transformed and refined into a business component implementation model. As exhibited in Figure 1 this is done by defining an implementation for the orchestrations, human services, IT services, and data services. These implementations are combined into SCA composites which form the business component implementation model. This final model can be implemented by executing it on engines, resulting in a working IT system.

Conclusions

As enterprise ontology is a way of thinking, different modelers create the same model for the same organization. This means our MDEE approach has a stable basis. While our MDEE approach is also model-driven and consists of steps which are partly automatable the quality of the resulting IT system can be very high depending on the quality of the MDEE approach itself. In other words: as the IT system is derived automatically (for a big part) from the organization model and the organization model is stable because of the underlying theory, the quality of the IT system can be ensured by improving the quality of the MDEE approach itself.

In our definition of MDE the automation of transformations is an important aspect. Hence, the question is to what extend the steps in our MDEE approach are automatable. The steps in our MDEE approach consist of two parts: a transformation part and a refinement part. The transformation part is defined with derivation rules and specifies what elements of the output model can be directly derived from the input model. This part can be fully automated with model transformations. The refinement part describes what additional elements have to be added to produce the output model. In this part decisions have to be made about what elements to add and how they are modeled. These decisions can be 'hard-coded' in transformations, thereby also automating this part of an MDEE step. However, a part of the decisions will always stay human decisions. The more an MDEE approach is tailored to a specific domain (e.g. the approach is only applicable to insurance companies), the more decisions can be taken beforehand and thus 'hard-coded' in automated transformations.

Because a significant part of the steps in our MDEE approach can be automated and the implementation model (which is directly executable) is defined in a much higher level language than currently existing programming languages, we can conclude that our approach fully complies to the definition of MDE. Our MDEE approach does raise the level of abstraction in program specification and it increases the automation in program development. Hence, our MDEE approach can finally deliver the increase in productivity we are waiting for quite some years.

Concluding we can state that we succeeded in designing an MDEE approach based on a sound theoretical foundation, providing end-to-end guidance to refine and transform an organization model into an IT system supporting that organization. As we have shown with our running example, starring the Protector case, our MDEE approach is applicable to real-life cases. However, due to the limited scope of our research there are some research limitations.

More information

You can find the slides of the presentation of the research results below. They give some more details about the MDEE approach and the used languages and techniques.

An Enterprise Ontology based approach to Model-Driven Engineering

View more documents from Johan den Haan.

In the near future I will post more information about MDEE. I am currently working on some scientific papers based on my research.

What do you think about MDEE?

Do you think extending model driven approaches with models of the organization does have advantages?

MDEE is quite scientific; do you see parts which can be useful in practice?

Modeling an organization using Enterprise Ontology

Sat, 10 Oct 2009 12:05:00 +0100

Business-IT alignment is hot, and that's not new. As regular readers know, my suggestion to attack the business-IT alignment problem is a model-driven approach. In a previous article I presented a framework for Model-Driven SOA in which we have seen that the starting point for a Model-Driven SOA approach is an organization model. While the organization model is the starting point of a model-driven process in which each model is as much as possible is automatically derived, it is important that this model is coherent, consistent, and concise.

In this article I want to explain the theory of Enterprise Ontology which describes a well-founded method to model the essence of an organization in a coherent, comprehensive, consistent, and concise way. The theory presented in this article is relatively new and differs quite a bit from existing organization modeling methods. I think Enterprise Ontology can offer advantages in understanding the essence of an organization and in using organization models as a starting point for building software supporting organizations.

Why Enterprise Ontology?

Enterprise ontology is focused on the essence of the operation of an organization, meaning that it is fully independent of the (current) realization and implementation of the organization. The theory that underlies the notion of enterprise ontology as presented by Dietz [1] is called the PSI-theory [9]. Dietz uses this theory to construct a methodology providing an ontological model of an organization, i.e. a model that is coherent, comprehensive, consistent, and concise, and that only shows the essence of the operation of an organization model. This methodology is called Design and Engineering Methodology for Organizations (DEMO).

Compared to its implementation model, the ontological model of an enterprise offers a reduction of complexity of over 90% [1]. This reduction of complexity makes an organization for a manager intellectual manageable and transparent. It also shows the coherence between all fields within the enterprise, like business processes, workflow, organization structure, etc.

DEMO has been widely accepted in both scientific research and practical appliance. It has been used as a base for formalizing enterprise architecture and governance [2, 3, 4] and for formalizing the splitting and allying of enterprises [5]. Further research has extended DEMO by constructing an ontological model of an information system supporting the enterprise model (the BCI-3D method) [6]. More recently enterprise ontology has been used to construct a formal framework for service specification [7]. An extensive ten year study executed with 28 projects concluded that DEMO is a good method for the fast (re)design of organizations [8].

The four axioms of Enterprise Ontology

The overall goal of the PSI-theory (the theory behind the notion of Enterprise Ontology) is to extract the essence of an organization from its actual appearance. It presents four axioms that help to achieve this goal [1].

The operation axiom tells us that the implementation independent essence of an organization is that it consists of subjects fulfilling actor roles. A subject fulfilling a certain actor role is called an actor. Actors constitute the operation of an organization by performing two kinds of acts: production acts and coordination acts.

By performing production acts (P-acts for short) the subjects contribute to bringing about the goods and/or services that are delivered to the environment of the organization. The results of P-acts are production facts (P-facts for short), which can be divided into material (something is manufactured, stored or transported) and immaterial (decisions or judgments) facts.

By performing coordination acts (C-acts for short) subjects enter into and comply with commitments towards each other regarding the performance of production acts. A C-act is performed by one actor, the performer, and directed to another actor, the addressee. C-acts consist of an intention (e.g. request, promise, question, assertion) and a proposition (the performer proclaims the fact and the associated time the intention is about) and result in coordination facts (C-facts for short).

Figure 1: Graphical representation of the operation axiom [1]

A graphical representation of the operation axiom is presented in Figure 1. As visualized, P-acts have effect in the so-called production world (P-world for short), C-acts have effect in the coordination world (C-world for short). The state of a world at a particular point in time is the set of facts created up to that point of time. The creation of a fact of some type is a state transition in one of the two worlds.

The transaction axiom tells us that C-acts are performed as steps in universal patterns, called transactions. This axiom reveals universal socionomic patterns of coordination that hold for all organizations. The standard pattern of a transaction is shown in Figure 2. A white box represents a C-act type and a white disk represents a C-fact type. A gray box represents a P-act type and a gray diamond a P-fact type.

Figure 2: The standard pattern of a transaction [1]

Two actors are involved in a transaction, the initiator and the executor. A transaction evolves in three phases: the order phase (O-phase for short), the execution phase (E-phase for short), and the result phase (R-phase for short).

In the O-phase the initiator and executor work to reach an agreement about the intended result of the transaction, i.e., the P-fact the executor is going to create as well as the intended time of creation. As shown in Figure 2 the initiator starts the transaction by requesting something from the executor. The C-fact 'rq' is created and the executor can either decline the request (after which the initiator can request again or quit the transaction) or accept the request by promising to deliver the requested P-fact (the C-fact 'pm' is created).

In the E-phase, the P-fact agreed upon in the O-phase is actually brought about by the executor. The gray box and diamond in Figure 2 reflect this phase.

A transaction ends with the R-phase in which the initiator and the executor work to reach an agreement about the P-fact that is actually produced, as well as the actual time of creation (which may both differ from what was agreed upon in the O-phase). Only if this agreement is reached the P-fact will come into existence. In figure 2 this phase starts when the executor states that the delivery of the P-fact has been done (C-fact 'st' is created). The initiator can either reject (after which the executor can state again or stop the transaction) or accept (thereby ending the transaction). The result of a successful transaction is the creation of a P-fact.

The composition axiom tells us how P-facts are interrelated. It states that every transaction is enclosed in some other transaction, or is a customer transaction of the organization under consideration, or is a self-activation transaction. According to Dietz this axiom provides the basis for a well-founded definition of the notion of business process. Which states [1]:

A business process is a collection of causally related transaction types, such that the starting step is either a request performed by an actor role in the environment (external activation) or a request by an internal actor role to itself (self-activation).

The distinction axiom tells us that actors exert three basic human abilities: performa, informa, and forma. Through the distinction axiom a substantial reduction of complexity and diversity is achieved, regarding both the coordination and the production in an organization.

Figure 3: Summary of the distinction axiom [1]

The forma ability concerns the form aspects of communication and information. As shown in Figure 3 communicative acts at the forma level are about uttering (speaking, writing) and perceiving (listening, reading). Production acts at the forma level are datalogical in nature, meaning that they store, transmit, copy, destroy, etc. information.

The informa ability concerns the content aspects of communication and information. Communicative acts at the informa level express thoughts (formulating) or educe thought (interpreting). Production acts at the forma level are infological in nature, meaning that they reproduce, deduce, reason, compute, etc. information. Note that on this level we deal with communication and information while fully abstracting from the form aspects. Figure 3 summarizes the different acts on this level.

The performa ability concerns the bringing about of new, original things, directly or indirectly by communication. As can be seen in Figure 3 communicative acts at the performa level are about exposing or evoking commitment. Production acts at this level are ontological in nature, meaning that they decide or judge about something. The performa ability is considered as the essential human ability for doing business of any kind. This additional layer is important as it explains and clarifies the organizational notions of collaboration and cooperation, as well as notions like authority and responsibility.

Figure 4: The process of performing a coordination act [1]

In order to successfully perform a C-act (as presented before in the operation axiom) the communicative acts at all three levels have to be performed, as visualized in Figure 4. The performer (P) has to expose his commitment to the addressee (A) to perform the C-act, i.e., P and A have to come to a social understanding. The only way to gain that understanding is by reaching an intellectual (or semantic) understanding. P has to express his thoughts, while A has to educe this thoughts and to confirm that she has (intellectually) understood P's information. P can only express a thought by formulating it in a sentence in some language, and by uttering it in some form, such as speaking or writing. A, on the other hand, can only educe the thought by listening or reading it. So, another level of understanding is needed, called significational (or syntactic) understanding. Lastly, we of course need some physical exchange of data, i.e., the field of (tele)communication technology. Dietz assumes the condition of correct physical transmission to be included in the forma condition [1].

We can make the similar distinction between the three levels of abstraction on the production side. Examples of acts on each level are mentioned before and summarized in Figure 3. Transactions having an ontological P-act as result are called ontological transactions. Infological transactions, i.e., transactions having an infological P-act as result, are considered to be a matter of realization of the ontological transactions. Infological transactions only reproduce or derive knowledge that is needed by the ontological actors. Datalogical transaction, i.e., transactions having an datalogical P-act as result, are in their turn, considered to be a matter of realization of the infological transactions. They only care for the storage and retrieval, the copying, and the transmission of documents containing information that is needed by the infological actors.

The Organization Theorem

In the previous section I have presented the four axioms of enterprise ontology. As stated before they all serve the overall goal of the PSI-theory to extract the essence of an organization from its actual appearance. The operation axiom abstracts the implementation independent essence of an organization to actors, acts and facts. The transaction axiom brings another reduce in complexity by combining acts in universal patterns holding for each organization. The composition axiom brings us the notion of business process and lastly the distinction axiom reduces complexity even more by providing a distinction between acts on different layers of human ability.

The organization theorem combines the benefits of these axioms into one concise, comprehensive, coherent, and consistent notion of enterprise [1]. This theorem states that the organization of an enterprise is a heterogeneous system that is constituted as the layered integration of three homogeneous systems: the B-organization (from Business), the I-organization (from Intellect), and the D-organization (from Document). We call these three systems the aspect systems of the (total) organization. Figure 5 visualizes the organization theorem and shows us that the D-organization supports the I-organization, which supports the B-organization. The coordination parts of these three systems are similar, they only differ in the kind of production: the production of the B-organization is ontological, the production in the I-organization is infological, and the production in the D-organization is datalogical.

Figure 5: Representation of the organization theorem [1]

Dietz draws a clear distinction between the realization and the implementation of an organization. By the realization of an organization he means the thorough integration of the three aspect organizations. The implementation of an organization is the making operational of the organization's realization by means of technology [1].

The integration of the three aspect organizations is established through the layered nesting of them. As said before a system in some layer supports the system in the next higher layer, e.g., the D-organization supports the I-organization, which supports the B-organization. Controversily, a system in some layer uses the system in the next lower layer. When a layer supports another layer its function supports the construction of the other layer, e.g., the function of the I-organization supports the construction of the B-organization. The distinction of the function perspective (F) and the construction perspective (C) provides a more elaborated view on the layered integration of organizations and is visualized in Figure 6.

Figure 6: The layered integration of an organization [1]

As said before, Dietz defines the implementation of an organization as the making operational of the organization's realization by means of technology. He refers to the possible ways of implementation of actor roles, C-acts, and P-acts as actor role technology, coordination technology, and production technology [1].

Conclusion

We have seen that enterprise ontology as described by Dietz gives a well-founded theory about enterprises. The DEMO methodology is based on this theory and describes a set of models to model an organization and how to create these models. We leave the explanation of DEMO for a future article. If you are curious you can read one of the references (listed below) or you can take a look at the homepage of the DEMO knowledge centre.

What I like about enterprise ontology is that it is focused on collaboration and social interaction among actors. The focus on transactions between social actors gives a better insight in the construction of an organization. It also can lead to a more dynamic and flexible view on organizations because you can look at it as a collection of autonomous actors dealing with incoming events and performing actions based on these events and a set of rules.

I especially like the distinction axiom which delivers a clear, well-founded layering of organizations. This axiom reduces the complexity of organization models by abstracting away from implementation details. However, in contrast with other approaches it is well-defined what elements should be modeled on what abstraction layer. I think the distinction axiom can play an important role in defining model-driven approaches using an organization model as starting point.

In other words: enterprise ontology can deliver us a theoretical foundation for defining the needed models of a Model-Driven SOA methodology.

--------------------------------------

[1] J. L. G. Dietz. Enterprise Ontology. Springer-Verlag, Berlin Heidelberg, 2006.

[2] J. A. P. Hoogervorst. Enterprise Architecture: Enabling Integration, Agility and Change. International Journal of Cooperative Information Systems, 13(3):213-233, 2004.

[3] J. A. P. Hoogervorst. Enterprise Governance & Architectuur - Corporate, IT en enterprise governance in samenhangend perspectief. Sdu Uitgevers bv, Den Haag, 2007.

[4] J. Dietz and J. Hoogervorst. Enterprise Ontology and Enterprise Architecture - how to let them evolve into effective complementary notions. GEAO Journal of Enterprise Architecture, 1, 2007.

[5] M. Op 't Land. Applying Architecture and Ontology to the Splitting and Allying of Enterprises. PhD thesis, Delft University of Technology, Schildmos 13, 3994 LS Houten, Netherlands, June 2008.

[6] A. Albani and J. Dietz. The benefit of enterprise ontology in identifying business components. In WCC '06: Proceedings of the IFIP World Computer Congress, Santiago de Chile, Chile, 2006.

[7] G. Hardjosumarto. An Enterprise Ontology based Approach to Service Specification. Master's thesis, Delft University of Technology, Delft, the Netherlands, October 2008.

[8] J. Mulder. Rapid Enterprise Design. PhD thesis, Delf University of Technology, 2006.

[9] J. Dietz. Enterprise ontology - understanding the essence of organizational operation. Enterprise Information Systems VII, pages 19-30, 2006.

Photo by mr_walker

From Process Design to Process Automation

Wed, 23 Sep 2009 19:13:00 +0100

Today I gave a talk at the BPM2009 conference in Garderen. A lot of talks were given, all centered around Business Process Management (BPM). Although a lot of people associate BPM with technical IT stuff, most of the talks were focused on process design, change management, emotions, and people. My talk, however, was a bit more technical as I focused on translating a process design into a Service-Oriented Business Application (SOBA) in a model-driven way. You can find the slides and a short overview below.

From Process Design to Process Automation

View more documents from Johan den Haan.

I started my presentation with my favourite metaphor to explain Model-Driven Development. After introducing Mendix, the company I work for, I have argued that we need to move from software engineering to business engineering. Software should be build by using declarative business models which describe the functionality of an application and by interpreting these models with appropriate engines. In a future article I will explain in more detail why I think direct model execution is necessary as successor of code generation.

I briefly touched the subject of a Model-Driven SOA framework, but I mainly focused on explaining the Mendix Modeling Method and how this method is supported with tools.

I hope you enjoy the slides. Please don't hesitate to post your comments and/or questions.

A metaphor for Model Driven Engineering

Wed, 05 Aug 2009 22:38:00 +0100

A couple of weeks ago I wrote an article introducing a framework for Model-Driven SOA. This article doesn't stand on its own, it's based on earlier pieces on Model-Driven SOA and observations that Model-Driven Engineering should focus on the problem domain. However, I'm noticing that these articles can be hard to grasp if you have no background knowledge. That's why I want to explain Model-Driven Engineering using a simple metaphor I often use in my presentations.

Figure 1 shows the metaphor which compares building a house with building software. ‘Traditional' programming can be compared with building a house manually. Someone studies the design and instructs people how to build the house (the upper half of the picture).

Figure 1 - A metaphor for Model Driven Engineering (MDE)

Model Driven Engineering (MDE) can be compared with the lower half of the picture. Based on some ideas on paper a model of a house is constructed. Once the model is complete it is put in a generator, et voila, the end result can be generated automatically.

When a Model Driven Engineering approach is used for developing software the main effort moves from programming to modeling. Based on functional requirements a model has to be developed. This model can be automatically converted into working software with a tool generating code from the model or by interpreting the model with a runtime environment (a.k.a. engine, virtual machine).

Based on this simple metaphor we can create an overview of the elements of Model Driven Engineering. We need:

an executable model. This can be accomplished by using Domain-Specific Languages (DSL) to specify the model. DSLs can vary in structure and in most cases we need more than one DSL to keep them small and specific.
a methodology to come from functional requirements, documentation, or something like that to the model. A first step into the direction of a methodology is this description of a framework for Model-Driven SOA.
tools to define the DSLs and to specify the model. We also need tools to ensure the quality of the model.
a generator or interpreter for making the model executable.
an architecture describing the type of applications delivered by the model driven approach. There is quite a difference between a house and a software application, but also within software there are lots of differences.

What is your favorite metaphor for explaining Model Driven Engineering?

6 questions to Maarten Steen, experienced researcher in the field of Model Driven Engineering

Wed, 08 Jul 2009 10:27:00 +0100

Recently I visited the second Dutch Model Driven Experience symposium, organized by Maarten Steen. It was interesting to meet other people in the field and to share experiences in the field of Model-Driven Architecture, Model-Driven Engineering, Domain-Specific Language design, Model-Driven Software Factories, and most important experiences with Model-Driven approaches in real-life projects. I thought it would be interesting to ask Maarten Steen to share some of his views and years of experience in the field. So, without further ado, six question to Maarten Steen:

Can you please introduce yourself?

Of course and thanks for the opportunity to present myself to your audience. You have a very interesting blog that I've been following for about a year now.

Like yourself I'm Dutch and based in the Netherlands. However, I have also lived in England and France, and I do not exclude the possibility to eventually settle in Portugal. I am married and have two small children. We are soon to set off again in our mobile home to travel through Europe like nomads as we do every summer.

I work as a researcher and consultant at Novay, the former Telematics Institute. Novay is an independent, non-profit institute for ICT-based innovation. We help organisations to innovate their products and services or the way they work by using ICT as an enabler. As a researcher and consultant I get the best of both worlds: I can both work together with researchers at universities to advance the state of the art, and I can directly work with enterprises to advise them on how to employ these novel ideas in their organisation.

In addition I try to be active in both the national and international MDD communities. At a national level I organise the annual Model Driven Experience symposium and I moderate the associated online community of practice. At an international level I am on the steering committee of the International IEEE Enterprise Computing Conference (EDOC) and have hosted this conference once in the past.

For more details, see:

Can you tell something about your research in the field of Model Driven Development?

I first encountered UML, OCL and the idea of model transformation when I was doing my PhD in England at the University of Kent at Canterbury. My own PhD research was about reconciling different formal specifications of distributed systems, which required a kind of horizontal transformation. In addition, I had the opportunity to discuss modelling issues with David Akehurst (developer of the Kent Modelling Framework and SiTra, a Java framework for Simple Transformations) and Stuart Kent (currently responsible for the Microsoft DSL tools). Mind you, this was still before the MDA initiative was launched by OMG!

Then when I first joined Novay in 1999, I worked mainly on methods, techniques and tools for conceptual modelling in various domains, such as networked enterprises, inter-organizational business processes, enterprise architecture and service-oriented architectures (SOA). So, I guess you could say that I have been researching some form of model-driven engineering (MDE) for over 10 years now.

The most concrete steps I have been able to make in de recently finished A-Muse project, where we developed a method, several DSLs and supporting tools for model-driven development of context-aware mobile services. Amongst many other lines of research, we test-drove the idea of platform-independence: is it really possible to generate PSMs and code for two different platforms - J2ME and .NET Compact Framework in our case - from a single PIM? The answer was: yes, but it ain't easy. As a side-effect we gained a lot of experience with QVT, the new model-to-model transformation standard, which we published as a kind of design patterns for model transformations. Another interesting experiment was to develop a graphical modeller and code generator for Google's Android platform using Eclipse EMF, GMF and xPand. Through this work, I got very excited about recent developments in Eclipse-based tooling for MDD (EMF, GMF, M2M, M2T). It's getting so much easier now to create the tools that you need.

My current research is focussing on applying MD-techniques in specific domains. In the VESTA project, for example, we are investigating how to model, analyse, simulate and visualise processes and scenarios for public order and safety. The idea is to use model transformations to link the various modelling, analysis and visualization tools used in the project.

What future plans does Novay have in the field of Model Driven Development?

Novay stands for networked innovation. We enable organisations to join forces and innovate together using ICT. The Model Driven Experience symposium is one example where we bring together organisations to exchange their experiences with MDD in order to inspire each other. And we intend to continue this highly successful symposium series in future years.

In addition, we are working on new projects where MD technology plays an enabling role. For example, we are currently investigating the feasibility of a project on combining business rules, business processes, services and objects in the development of enterprise systems.

What do you see as the most important research gaps in the field of Model Driven Development?

First of all, there is still a considerable gap between the state of the art and the state of practice in MDD. Practitioners rarely go beyond straightforward code generation from UML or DSL models. As a consequence these models are rarely platform-independent. MDD is mainly positioned as a method for increasing developer productivity, whereas it can be used also to get rid of the always problematic business - IT gap. Our efforts should be focussed on raising the level of abstraction at which models are made and used.

For me the most important research issues are related to this question: how to make MDE more practical? What should an MD development process look like? Which models should be made? In which order? How do we know they are consistent? How do we deal with changes? How do we manage and version models? And one issue I am particularly interested in myself is how to employ model transformation technology to relate and synchronize models from different viewpoints.

You recently organized the second Model Driven Experience conference, did you see emerging trends?

There was a lot of consensus on the advantages and pitfalls of MDD. Almost everyone agrees that MDD reduces development time and costs, increases software quality, increases maintainability and improves communication between stakeholders. On the other hand, it is generally acknowledged that MDD requires changes in culture, process, roles and skills, and that these changes cause resistance. Some also mention that MDD reduces flexibility and introduces new integration problems.

When we organised the symposium for the first time last year, most of the presentations were on experiments and small pilots with MDD. This year the presented initiatives seemed to be much more mature. Several Dutch companies, such as Sligro Food Group, ASR, Ordina, and Capgemini BAS, have a operational model-driven software factory, and others, such as ASML and Philips Healthcare, are using MDD in the development of critical software components.

Another noticeable development is that UML is hardly used anymore for modelling; almost all MDD initiatives now use some kind of DSL.

Do you have some advice for the community about how to spread the message of Model Driven Development?

One of the interesting conclusions of the discussion panel at the end of the Model Driven Experience 2009 was that most resistance against MDD comes from within the IT departments, not from the business. Business managers are easily convinced by the usual arguments of lower development and maintenance costs, better and more consistent software quality and reduced time-to-market. The software developers themselves however are often sceptic that their work can be automated. They also may feel that they loose control over the development process and the produced code.

Organisations that adopt MDD should address these fears. Switching to MDD requires a cultural change both at the business and the IT side. The business people should start feeling responsible for modelling the application domain and the required functionality. They also should learn how to work according to a structured development process. The IT people, on the other hand, should focus on generic solutions, reference architectures and the transformation to specific platform technology, and no longer feel responsible for the business analysis and application design. As with any cultural change, this won't happen overnight.

As a community we should collect and disseminate the increasing number of success stories to make people more aware of the advantages and implications of model-driven architecture and development.

8 reasons why Model-Driven Development is dangerous

Thu, 25 Jun 2009 22:44:00 +0100

Last week I gave a talk at the Hogeschool Arnhem Nijmegen as part of the conference "Information Systems the Next Generation". This article is inspired by a talk titled "Model Based Development - How to organize and architect survival of MD*" by Wiebe Wiersema given at the same conference. It was a well-balanced, realistic talk about the do's and don'ts of Model Driven Development.

While Model-Driven Development (MDD) is getting more and more attention by both tool vendors and developers, I think it's time to look at 8 reasons why MDD is dangerous. If you recognize one of these points from your practical experiences or if you think you have a solution for (almost) all points, please let me know it in the comments! Although model-driven approaches can differ a lot, I think these 8 points can be applied to almost all of them.

1. MDD actually introduces a lot of rigidity

If you're used to programming everything by hand, MDD can be quite rigid. The goal of MDD is to ‘program' on a higher level of abstraction. This means that you have to specify less and generate more. However, this also means that you can't change every little detail you want. It is, for example, often the case that generated graphical user interface are inflexible and they all look like each other.

2. Models are only flexible where flexibility has been designed

The problem with using models to directly drive the engineering of software is that they are far from flexible. First, you are limited by the kind of Model Driven Engineering tool you use. Second, you're only flexible in the parts of the solution covered by the used Domain-Specific Languages. The higher level the DSL the more commonalities are ‘hard-coded' in the framework or engine you use. Third, sometimes models are made flexible at predefined points by lower level expressions or languages. That's nice, but hopefully it is exactly done at the points you'll need it.

3. The roles of project members are quite different

You need to be aware of the fact that the roles in Model-Driven Engineering are different from the roles in traditional development. The real technical or programming part is mostly moved to the ‘meta-team' building the model-driven software factory. Building the solution is done by so-called business engineers instead of programmers. These people need to understand the business (like e.g. traditional functional designers) but they also need to express the knowledge they gather in a formal model. People able to do both these things are quite rare.

4. The modeling environment doesn't always support version control

If you have experience with building (enterprise) software applications I guess you're using a version control system. If not you should go and read this nice explanation. Existing version control systems work very well for textual programming languages. However, versioning of graphical models is a far less researched field. Most existing modeling and model-driven tools don't include a full featured versioning system. That's a pain when working with large teams.

5. The modeling tool/approach is "almost" finished at project start

The famous word "almost" is pitifully used a lot when talking about how far the model-driven software factory is finished at the start of a project. People building a model-driven software factory using DSL tools will have a lot of fun during the process. However, if the factory isn't finished and tested in practice before you are going to start a big project you will have a huge risk! If you're not scared by this article to use MDD, at least use a proven tool/approach!

6. The requirements team needs to understand what is allowed and what not

In point 1 and 2 I pointed at the rigidity and inflexibility of MDD. This is mostly dangerous from a technical perspective. However, it is even more dangerous when the requirements team, i.e. the people talking with the customer, don't understand the limitations of the used tool/approach. In the ideal situation, the people talking with the customer directly translate their knowledge in executable models. However, in big projects you often see a separation in two roles. As in traditional development this can lead to a gap. This gap can become a nasty problem if the requirements team doesn't understand the limitations of the tool/approach. For example, the requirements team reaches a consensus with the customer on a certain part of the graphical user interface (let's say a form), while it isn't possible to build that form with the model-driven tool without a lot of custom handiwork.

7. MDD is sold to the customer, but the team has no experience

While MDD is becoming more popular, customers start to ask for model-driven solutions. Actually I think MDD will become a hype (again!) very soon, and as with each hype everybody will start sticking the MDD label on their products. Doing a project in a model-driven way with an inexperienced team can lead to at least two problematic situations. First, the solution will not be build on time and it will not fulfill the customers' expectations due to most of the issues mentioned in the previous points. Second, big chance the end product isn't as easy to change as you expect from a solution built using a model-driven approach, due to misuse of the tool, not knowing its limitations, or due to using immature tools.

8. Innovation distracts

Last but not least, MDD is dangerous because innovation distracts. People tend to focus on the new tool they are using with all its cool features. Even more, people will start bragging about the great advantages they have with the new technology, in the end forgetting the projects objectives. Experiments have been done actually showing this behavior of project teams starting to use new innovative tools. The important lesson: don't forget your project management, your process, and all other things you need beside the tool or technical approach you use!

Photo by netwalker.

A Framework for Model-Driven SOA

Wed, 03 Jun 2009 17:36:00 +0100

A couple of weeks ago I gave an overview of two scientific publications researching a framework (or methodology) providing end-to-end guidance and assistance to refine and transform business models created by business experts into IT models and then to IT implementation. I tend to refer to these frameworks as Model-Driven SOA, because they focus on using models to construct an IT implementation complying to SOA principles.

However, I didn't elaborate on the practical applicability of these studies. I asked you for your opinion. From the comments on the article and the discussion in the LinkedIn MDA group triggered by this article, it became clear to me that there's a need to describe Model-Driven SOA in more detail before we can sufficiently compare different approaches and discuss the needed models, languages, and transformations.

In this article I present a generic framework describing Model-Driven SOA. I first describe the common terms in system development, like design, engineering, and architecture. After that we can apply these terms to a Service-Oriented Architecture (SOA) based development process. Finally, I want to define the requirements making this SOA system development process a Model-Driven Engineering approach.

System Development

System development is the bringing about of a new system or of changes to an existing system. The system development process comprises all activities that have to be performed in order to arrive at an implemented system. Such a development process can be described by the Generic System Development Process (GSDP) as introduced in [1] and recently described in more detail in [2]. Figure 1 exhibits the GSDP. At first sight it can looks a bit abstract, but give me a moment to explain it and I hope you'll see that it's in fact an easy and powerful way to explain the common terms in software development.

Figure 1 - The Generic System Development Process [2]

I have explained this process before in a post on the definition of architecture. So, for details I refer to that post. However, the process has slightly changed over the years as did my view on it. I will therefore briefly explain this generic system development process again.

The GSDP always concerns two systems, the object system (i.e. the system which is going to be developed) and the using system (i.e. the system that is going to use the functionality of the object system). If we, for example, want to build a human resource management (HRM) information system, this HRM information system is the object system. The using system is the HRM department. People are often referring to the using system as the problem domain, while they call the object system the solution domain.

While in a lot of cases no construction model of the using system exists, we need to reconstruct the higher level models from the implementation. This process is known as reverse engineering. Let's look again at our HRM system example. Before we can build an HRM information system supporting a certain HRM department we need to know what to support. We need to describe the using system and the only way to come to this description is to use reverse engineering. In case of the HRM department reverse engineering means talking with the people (the "implementation" of the using system) to learn how their work is performed.

The design process is split into function design and construction design. Function design starts from the construction of the using system and ends with a functional (black-box) model of the object system. This functional model describes the external behavior of the object system in terms of the using system, it doesn't talk about the construction of the object system. The functional model of an HRM information system can for example be describe with use cases.

Construction design starts with the functional model and ends with a construction (white-box) model of the object system. As shown in Figure 1 multiple construction models can exist on different abstraction levels.

The engineering of a system is the activity in which the highest level constructional model is converted into lower level models until the implementation model is constructed. Dietz defines the highest level constructional model as the ontology of a system, i.e. a model of the construction of a system that is completely independent of the way in which it is implemented [2]. This can be compared to the Platform Independent Model (PIM) in the Model Driven Architecture (MDA). In practice it is often the case that not all constructional models are defined, in most cases only the implementation model (the lowest level constructional model) is defined in a programming language (e.g. Java, C#, C++).

Implementing a system means assigning technological means to the constructional elements in the implementation model [2]. In case of an implementation model expressed in a programming language, the implementation is the compilation of that model to a specific platform. Once a system is implemented it can be put into operation.

The last part of the generic system development process is "architecture". Dietz uses the prescriptive notion of architecture, in contrast to the descriptive notion of architecture. The descriptive definition defines architecture as 'blue-prints', but the GSDP already refers to this by "constructional model" or "ontology". The prescriptive notion of architecture sees architecture as a consistent and coherent set of design principles. In the generic system development process, architecture is split into functional principles guiding the function design and constructional principles guiding the construction design.

Service-Oriented Software Development

It's time to tailor the generic system development process to a service-oriented development process. Lot's of SOA definitions exist, but I prefer to see SOA as an architectural style. Figure 2 shows the service-oriented development process, based on the generic system development process.

Figure 2 - The Service-Oriented Development Process

As shown in Figure 2 the using system is an organization (or part of an organization) which can be modeled with an organization model. Such a model can consist of process models or just textual descriptions. The organization model needs to be reverse engineered from the organization's implementation, which is in most cases a group of people and optionally already existing software.

The object system is an ICT system, to be designed and implemented to support the organization. The design, consisting of function and construction design, is guided by architecture principles. In case of a service-oriented development process the design is guided by SOA principles. A good overview of SOA principles is given by Thomas Erl in his book "Principles of Service Design" [3].

While the functional model of the software to build is defined with a set of services, the function design step consists of service identification and service specification. First the set of needed services is identified based on the organization model, next each service is specified in more detail (preferably using a standardized template). I'm just giving the outline of a generic service-oriented development process, so the details of question like "what exactly is a service?" and "how to specify a service?" are left for future ramblings.

The next step in the process is to take the functional specification consisting of service specifications and to design the construction of the needed ICT system which is going to provide these services. Following the Service Component Architecture (SCA) services can be grouped in components, which in their turn can be implemented with other components. Hence, service-oriented construction design consists of component identification. The highest level constructional model can for example be described by an SCA-based model.

The SCA also specifies how components can be implemented. Example component implementation possibilities are Java, .NET, and BPEL. So, the engineering step consists of specifying component implementations using Java, .NET, or BPEL, resulting in an implementation model consisting of artifacts described in these languages. This implementation model can be "implemented" on technology thereby resulting in the final software application, i.e. a Service-Oriented Business Application. Implementation of the implementation model means deploying all constructed artifacts to an SCA container.

Model-Driven Service-Oriented Software Development

Now what distinguishes SOA from Model-Driven SOA? I've already stated the term "model" a couple of times in the previous section and in Figure 2, but does that make the described service-oriented development process model driven? Model-driven engineering is based on formal models which are transformed into other models until the lowest level model can be executed with an engine or until code can be generated from that model (for more explanation read this introduction into model-driven software development). As we saw in the previous section the models are not necessarily formal and the implementation model is directly specified in a programming language. So, what are the requirements for Model-Driven SOA?

First, each model needs to be formal, i.e. it needs to be specified in a well-defined language. As we know from the previous the following models are needed:

Organization model - for example specified in BPMN.
Service model - mostly consisting of service specifications which are represented with a table listing some properties.
Component model - for example specified using the SCA assembly specification.
Component implementation model - each component can be specified with a set of connected Domain-Specific Languages (DSLs), meaning that a component implementation model can be considered as a multi-model.

Second, each model (except for the first model) is linked to the previous model, derived from the previous model, or a refinement of the previous model. The following steps are part of a Model-Driven SOA process:

Reverse engineering - an organization model is manually created.
Function design - a service model is derived from the organization model, whether this can be automated or not depends on how formal the organization model is. In most cases this is handiwork. Elements of the service model are linked to the organization model to enable tracing.
Construction design - a component model is derived from the service model. This is again a manual job.
Engineering - a component implementation model is a refinement of the component model. Implementation elements are specified in detail using DSLs.
Implementation - the component implementation model is implemented on technology, i.e. the models specified using DSLs are executed on high-level engines. The result is a Service-Oriented Business Application.

Multiple roles are involved in such a process using different tools. An important aspect for the success of a Model-Driven SOA approach is the ability to specify DSLs which are both high-level and directly executable. Constructing such DSLs can be quite a challenge, I gave an overview and 7 recommendations for DSL development in my previous post.

A second important aspect for Model-Driven SOA is the traceability of decisions throughout the model chain. It should be possible to trace a model element in the component implementation model back to the organization model and the other way around.

This also leads to the last important aspect for Model-Driven SOA I want to mention: change propagation. The organization model and service model are very useful for gaining a quick insight in the function of the system. The real power of these models will become available if a change in the organization model leads to an automatic impact analysis on the service model. The same should hold for changes in the service model and how they affect the organization model and component model.

If you look back at Figure 2, I think it is a good recommendation that models which are connected horizontally (organization model - service model - component model) do not automatically propagate changes, they should report inconsistencies and provide a change impact analysis. Vertically connected models (the models used in the engineering step) will need automatic change propagation to provide productivity improvements, otherwise you're half of the time busy with keeping the models consistent. In case of organization, service, and component models, you should be busy with that because in the consistency between these models lay most of the design decisions.

Conclusion

I hope this article gave you more insight in my ideas with respect to Model-Driven SOA. I have only described the framework including the needed models and transformations between these models. An important aspect to talk about in the future is the languages these models are specified in, i.e. the exact form of these models. Other aspects are needed tooling and technology (runtime engines). Moreover, every time we talk about such a model-driven approach, we shouldn't forget these 10 misperceptions and challenges of Model-Driven Development.

I'd like to hear your thoughts on this model-driven SOA framework! I'm also interested in your experiences with using a model-driven approach to build software supporting (part of) an organization. What are your important learned lessons? What challenges are you dealing with?

---------------------------
[1] J.L.G. Dietz and J.A.P. Hoogervorst. Enterprise Ontology and Enterprise Architecture - how to let them evolve into effective complementary notions. GEAO Journal of Enterprise Architecture, 1, 2007.

[2] J.L.G. Dietz. Architecture - Building strategy into design. Academic Service, The Hague, The Netherlands, 2008.

[3] Thomas Erl. Principles of Service Design. The Prentice Hall Service-Oriented Computing Series from Thomas Erl. Prentice Hall, Upper Saddle River, NJ, USA, 2007.

Photo by Ride My Pony

DSL development: 7 recommendations for Domain Specific Language design based on Domain-Driven Design

Wed, 06 May 2009 15:50:00 +0100

The term Domain-Specific Language (DSL) is heard a lot nowadays. A DSL is a language developed to address the need of a given domain. This domain can be a problem domain (e.g. insurance, healthcare, transportation) or a system aspect (e.g. data, presentation, business logic, workflow). The idea is to have a language with limited concepts which are all focused on a specific domain. This leads to higher level languages improving developer productivity and communication with domain experts. In a lot of cases it is even possible to let domain experts use the DSL and develop applications.

The question for this article is: how to develop a Domain-Specific Language?

I'll first explain the DSL lifecycle, consisting of the phases: decision, analysis, design, implementation, deployment, and maintenance. Afterwards I'll give 7 recommendations for DSL development based on my experiences with developing non-trivial DSLs.

The DSL lifecycle

According to Mernik et al. [1] the DSL life cycle consists of five development phases: decision, analysis, design, implementation and deployment. Eelco Visser [2] adds maintenance as the sixth phase in the lifecycle of DSLs. Note that in practice DSL Development isn't a sequential process, the phases should be applied iteratively.

Let's look at each phase in more detail.

1. Decision

The development of a DSL starts with the decision to develop a DSL, to reuse an existing one, or to use a GPL. If a domain is very fresh and little knowledge is available, it doesn't make sense to start developing a DSL. In order to determine the basic concepts of the field, first the regular software engineering process should be applied and a code base supported with libraries should be developed [2].

In other words: if you never have developed an application for a certain domain by hand and you have no existing code base, it isn't smart to start implementing a DSL and its associated code generators or execution engine.

The situation differs of course for non-executable DSLs. However, as you need experience with existing code for executable DSL, along the same lines you'll need a deep understanding of the domain you are modeling for non-executable DSLs.

2. Analysis

In the analysis phase the problem domain is identified and domain knowledge is gathered. The output of formal domain analysis is a domain model consisting of [1]:

a domain definition, defining the scope of the domain,
domain terminology (vocabulary, ontology),
descriptions of domain concepts, and
feature models describing the commonalities and variabilities of domain concepts and their interdependencies.

The information gathered in this phase can be used to develop the actual DSL. Variabilities indicate what elements should be specified in the DSL, while commonalities are used to define the execution engine or domain framework.

If you, for example, analyze a couple of existing code bases in a certain domain, you can split the elements of this code in two parts: the parts that differ and the parts that are the same for each code base. The static parts (the commonalities) can, depending on your implementation approach, be part of the execution engine interpreting the DSL or can be put in a domain framework which is used by the generated code. The parts that differ (the variabilities) should be specified in the DSL, these are the parts which a user of the DSL needs to ‘configure'.

Eelco Visser [2] recommends an inductive approach which, in opposite to designing the complete DSL before implementation, incrementally introduces abstractions that allow to capture a set of common programming patterns in software development for a particular domain. He also states that developing the DSL in iterations can mitigate the risk of failure. Instead of a big project that produces a functional DSL in the end, an iterative process produces useful DSLs for sub-domains early on.

In the second part of this article I will give 7 additional recommendations for the analysis and design phase of a DLS, based on my own experiences.

3. Design

Approaches to DSL design can be characterized along two orthogonal dimensions: the relationship between the DSL and existing languages, and the formal nature of the design description [1]. A DSL can be designed from scratch or it can be easier to base it on an existing language.

Mernik et al. [1] identify three different patterns of design based on existing languages:

piggyback: existing language is partially used,
specialization: existing language is restricted, and
extension: existing language is extended.

Besides the relation with existing languages the formal nature can range between:

informal: a DSL specified in natural language and/or with examples, and
formal: a DSL specified using one of the available semantic definition methods, e.g. regular expressions, grammars, etc.

It is important to decide what approach to take, however, it is maybe even more important to keep this lesson in mind [3]:

Lesson T2: You are almost never designing a programming language.
Most DSL designers come from language design backgrounds. There the admirable principles of orthogonality and economy of form are not necessarily well-applied to DSL design. Especially in catering to the pre-existing jargon and notations of the domain, one must be careful not to embellish or over-generalize the language.

Lesson T2 Corollary: Design only what is necessary. Learn to recognize your tendency to over-design.

4. Implementation

For executable DSLs the most suitable implementation approach should be chosen. Mernik et al. [1] identify seven different implementation patterns, all with different characteristics:

interpreter, DSL constructs are recognized and interpreted using a standard fetch-decode-execute cycle. With this pattern no transformation takes place, the model is directly executable.
compiler/application generator, DSL constructs are translated to base language constructs and library calls. People are mostly talking about code generation when pointing at this implementation pattern.
preprocessor, DSL constructs are translated to constructs in an existing language (the base language). Static analysis is limited to that done by the base language processor.
embedding, DSL constructs are embedded in an existing GPL (the host language) by defining new abstract data types and operators. A basic example are application libraries. This type of DSL is mostly called an internal DSL.
extensible compiler/interpreter, a GPL compiler/interpreter is extended with domain-specific optimization rules and/or domain-specific code generation. While interpreters are usually relatively easy to extend, extending compilers is hard unless they were designed with extension in mind.
commercial off-the-shelf, existing tools and/or notations are applied to a specific domain. You don't have to define your DSL, editor and DSL implementation approach yourself, you just make use of a Model Driven Software Factory. You can, for example, use the Mendix Model-Driven Enterprise Application Platform targeted at the domain of Service-Oriented Business Applications.
hybrid, a combination of the above approaches.

While the different approaches can make a big difference in the total effort to be invested in DSL development, the choice for a particular approach is very important.

5. Deployment

In the deployment phase the DSLs and the applications constructed with them are used. Developers and/or domain experts use the DSLs to specify models. These models are implemented with one of the implementation patterns presented in the previous section (e.g. the models are interpreted by an engine). Such an implementation results in working software which is used by end-users.

6. Maintenance

While domain experts themselves can understand, validate, and modify the software by adapting the models expressed in DSLs, modifications are easier to make and their impact is easier to understand. However, more substantial changes in the software may involve altering the DSL implementation. So, like any other element of software a DSL will evolve over time. Therefore having a DSL migration strategy is very important.

Besides migration strategies, I have two recommendations which alleviate the maintenance risks of DSLs:

use a good DSL tool, which at least generates editors and generators / interpreters from your language definition, and
implement models specified in different DSLs with different loosely coupled engines. In this way maintenance of a specific DSL or its compiler does not affect the whole system and the engines can make use of different technologies. In case of business applications, this calls for an architecture encouraging the development of Service-Oriented Business Applications, i.e. an application composed of multiple loosely-coupled services.

Seven Domain-Driven Design based recommendations for DSL Development

Now the lifecycle of a DSL is clear I want to share some of my experiences with the analysis and design phases of the DSL lifecycle. The other phases are left for a future article.

Before going into the details of DSL design let's try to understand the context of these experiences. First of all, they are focused on creating multiple connected DSLs, i.e. you can create models expressed with different DSLs referring to each other. For example, in a Form model you can refer to elements from your Data model. More specifically, we are talking about a set of DSLs covering all system aspects of a Service-Oriented Business Application.

Another important point in the DSLs we're talking about is that they are all aimed at non-programmer domain experts. For most cases this means domain experts can create models expressed in these DSLs, in a few cases this means they can at least read them. This of course always leads to finding a balance between flexibility and complexity.

Based on my experiences, influenced by the concepts of Domain-Driven Design [4], I have the following 7 recommendations for DSL development:

1. Capture domain knowledge in a metamodel

If you talk about models for DSLs you will stumble upon the term metamodel. For a lot of people this sounds scary enough to stop reading. However, it's just a model of the abstract structure of the language. In other words: a metamodel models the concepts of a language and their relationships. Just as you model the concepts ‘Order', ‘Product', and ‘Customer' if you are building software like an order entry portal.

A metamodel is essential for constructing a DSL. It captures the knowledge of the domain the DSL is aimed at. The model reflects how the team developing the DSL structures the domain knowledge and what they see as the most important elements. The binding of model and implementation ensures that the experiences with earlier versions of the DSL can be used as feedback in the modeling process.

2. Communicate using an ubiquitous language

The metamodel is also important for communication purposes. When designing a DSL a lot of communication is needed between the users of the language (domain experts) and the developers. The metamodel is the backbone of a language used by all team members. Because the model is bound to the implementation, developers can talk about the DSL in this language. They can communicate with domain experts without translation.

You should play with the model when talking about the DSL. If you can't talk in terms of the model about a scenario, the model should be adapted until you can. If the domain experts don't understand the model, there is something wrong with the model. Domain experts should object to terms or structures that are awkward or inadequate to convey domain understanding. Developers should watch for ambiguity or inconsistency that will trip up design.

3. Let the metamodel drive the implementation

Don't forget that a language definition is more than just a metamodel (abstract syntax). A language definition also contains a concrete syntax and semantics. When designing and implementing a DSL the concrete syntax is captured in the solution workbench, i.e. an environment in which you can specify models using the DSL with either a textual or a graphical concrete syntax. The semantics of the language are captured in the transformation rules or model interpreter (based on the used implementation pattern, see above).

It is important that the metamodel drives the implementation of the solution workbench and interpreter, i.e. the metamodel should driven the implementation of the DSL. If the implementation doesn't map to the metamodel, the metamodel is of little value. At the same time, complex mappings between metamodel and implementation are difficult to understand and in practice difficult to maintain as the design changes. A deadly gap between metamodel and implementation opens, so that insight gained in each of those activities does not feed into the other.

Therefore, design the metamodel in such a way that it reflects the implementation in a very literal way. However, demand at the same time that a single metamodel also serves the purpose of supporting the ubiquitous language. The implementation must become an expression of the metamodel, so a change to the code may be a change tot the metamodel and the other way around. To tie the DSL implementation and metamodel in such a way, usually requires DSL tools that let you generate big parts of the DSL implementation from the metamodel. Figure 1 exhibits such a scenario in which part of the solution workbench and interpreter are generated from the metamodel.

Figure 1 - Metamodel-driven DSL implementation

4. Isolate the domain

As said before, DSLs will evolve over time. In the previous recommendation we've seen that it is important to tie model and implementation, the model should drive the implementation. However, to do so you need to isolate the domain. If the domain code, representing the metamodel is diffused through the code it is very difficult to make changes to it. Changes in the GUI of your modeling environment or the infrastructure of your interpreter can actually change your domain code.

In principle the recommendations for ‘normal' software also hold for DSL implementations. Divide your code into layers and concentrate all the code related to the domain model in one layer which is isolated from GUI and infrastructure code. The domain objects should be free of the responsibility of displaying themselves, storing themselves, managing application tasks, etc. They should focus on expressing the domain model.

In the previous recommendation I stated that the model should drive the implementation, and I meant to do that as literally as possible. This is possible if you isolate the domain! Using the Generation Gap Pattern you can generate all the domain code while isolating it from your other code.

So, isolate the domain to ensure that the model can evolve to be rich enough to express the domain and to keep track of the changes in that domain.

5. Refactor continuously

Along the same lines you should refactor all the time. You should refactor while you're knowledge crunching. You should refactor while you're communicating using the metamodel. You should refactor while you're busy with implementing the DSL. You should refactor while you're generating code from you metamodel. To say it with Eric Evans [4], you should especially refactor if:

The design does not express the team's current understanding of the domain.
Important concepts are implicit in the design (and you see a way to make them explicit).
You see an opportunity to make some important part of the design suppler.

I think it doesn't need any explanation that such an approach needs close involvement of all team members including the domain experts.

6. Maintain metamodel integrity

To effectively abstract a complex domain with domain-specific models, you need more than one DSL. In complex projects multiple DSLs are usually necessary in order to cope with different concerns. In other words: multiple domain-specific models (DSMs), specified in different DSLs are needed to accurately abstract complex systems.

Total unification of the metamodel (remember: the metamodel describes the concepts of the DSL we are designing) for a large domain will not be feasible or cost-effective. The most important reason for this is that attempting to satisfy everyone with a single metamodel (and thus a single language) will lead to complex options that make the language difficult to use. This is the reason we are designing a DSL at all! Different domain experts will have a need for their own domain specific language to define their aspect of the system.

So, we need multiple domain specific languages, hence we also need multiple metamodels. However, the boundaries and relationships between different metamodels need to be marked consciously. Some recommendations on multi-DSL development:

Explicitly define the context for each metamodel, i.e. define the domain (e.g. system aspect) the DSL is designed for.
Continuously integrate the implementation of a metamodel and make the interfaces to other metamodels part of the automated tests.
Model the points of contact between the metamodels and use that model in your ubiquitous language. These points of contact define how models expressed in different DSLs can refer to each other. For example, a GUI element can refer to an element in the data model.
Think about your reference resolve strategy. If you use interpreters / engines to execute the models expressed in a DSL you can use late-binding, i.e. use soft references and resolve them at runtime. The advantage of this strategy is flexibility and adaptability. The approach usually used with code generation is early-binding, the references are explicitly reflected in the generated code. Performance can be a reason to follow this strategy.

7. Use a people-oriented approach

Executing a DSL implementation process, especially in a way as recommended in the previous points, is not easy. It requires an effective team of developers and domain experts. My last, and most important, recommendation is to use a people-first approach in DSL development. DSL development is highly creative an professional work. Developers need to make the technical decisions, they are the best people to decide how to conduct their technical work. Domain experts live the domain, hence they are best suited to decide on the applicability of the concepts of the language.

Although I strongly recommend the way of working reflected in the previous six points, the team has to decide on the process. Accepting a process requires commitment, and as such needs the active involvement of all the team.

Key take aways for DSL development

Capture domain knowledge in a metamodel
Communicate using an ubiquitous language
Let the metamodel drive the implementation
Isolate the domain
Refactor continuously
Maintain metamodel integrity
Use a people-oriented approach

----------------------

[1] Marjan Mernik, Jan Heering, and Anthony M. Sloane. When and how to develop domain-specific languages. ACM Comput. Surv., 37(4):316-344, 2005.

[2] Eelco Visser. WebDSL: A case study in domain-specic language engineering. In R. Lammel, J. Saraiva, and J. Visser, editors, Generative and Transformational Techniques in Software Engineering (GTTSE 2007), Lecture Notes in Computer Science. Springer, 2008.

[3] Wile, D. S. 2004. Lessons learned from real DSL experiments. Sci. Comput. Program. 51, 265-290.

[4] Eric Evans, Domain Driven Design: Tackling Complexity in the Heart of Software. Addison-Wesley, 2004.

Photos by Gail S and Hélio Costa