The Enterprise Architect

Designing the next programming language? Understand how people learn!

Thu, 14 Feb 2013 12:01:00 +0100

Somehow it is a recurring theme in computer science: create a “programming” system that is easier to use and learn than the existing programming approaches. I am not just talking about better tools, like IDEs, but also new languages. It seems as if each self-respecting programmer creates his/her own language or tool-set nowadays, right?

Okay, I have to admit that not all efforts are focused on making things easier, often the focus is on productivity. However, if we, for example, look at the "programming" languages created in the Model-Driven Development there, we see quite some focus on involving domain experts in development by creating higher level, domain-specific, or sometimes even visual languages. Although there are much more reasons to do Model-Driven Development, it is the ease of use and the lower entry barrier that captures the imagination.

There is a fundamental flaw in our thinking around these approaches, though...

Language fundamentals

Let’s look at the fundamentals of a language before we dive into the details. A language specification consists of three main elements:

Concrete syntax: the concrete syntax defines the physical appearance of language. For a textual language this means that it defines how to form sentences. For a graphical language this means that it defines the graphical appearance of the language concepts and how they may be combined into a model. Multiple concrete syntaxes can be defined for the same language. You could, for example, define both a textual and a graphical concrete syntax.
Abstract syntax: the abstract syntax defines the concepts of a language and their relationship to each other.
Semantics: the semantics describe the meaning of a sentence or model specified in some language. In the context of programming this means that the semantics of a language describe what the effect is of executing the statements of that language.

All these elements play an important role in how easy it is to learn a language.

How do we learn a new programming language?

In general we learn in a number of different ways. Up to 10 different learning styles have been defined, but I think they can boil down to 3 main styles:

Listen: let someone educate you.
See: read stuff, watch someone else do it.
Experience: just start and try it yourself, experiment, trial and error.

Most people combine multiple styles when they learn something new.

Let’s go one step further: when do people understand how something works (we are not talking about factual knowledge here)? If they hear, see, or experience cause and effect. That’s when we connect the dots. If you hit a play button and the music starts to play you understand the function of that button, and if you hit different kinds of buttons on different systems that all lead to “the music starts playing” you will probably understand that the triangle icon means “play”.

When we learn how a system works or more specifically when we learn a new programming language, we can have different learning styles, but in the end it is about relating cause and effect. Whether you hear someone explain it, see someone do it, or experience it yourself.

The difference between simple and complicated, and thus easy or difficult to learn, is due to the “distance” between cause and effect. If there are many steps between cause and effect it can be difficult to connect the two. If a “system” is a black-box with multiple inputs and outputs, with a complex relation among inputs and outputs it is difficult to determine cause and effect and hence it is difficult to understand the system.

If we want to create a new programming language that has “easy to learn and use” as a core design principle, we should aim our efforts on an easy-to-understand cause-effect relationship between language concepts and the actual semantics of the language.

Let’s see how approaching a language from this angle influences concrete syntax, abstract syntax, and semantics.

On concrete syntax: how does it convey meaning?

Most discussions (and/or flame wars) around languages are focused on concrete syntax (sometimes combined with discussion about abstract syntax because there is a close relation between abstract and concrete syntax in most languages). Syntactic sugar like white-space and symbols as well as how concise the language is, are hugely interesting discussion topics

Our main question around concrete syntax, however, should be: how does the language convey meaning? If we read the language, do we know what the words mean? Do we understand the meaning of the data (inputs, variables)? Can we easily follow the flow? These are the things that matter!

You want examples you say? Well, this may be the worst readable language of all, that’s the goal at least. The microflow language we use in our platform is a bit easier to read and understand and hence conveys the meaning of the program much better (and it even leads to art).

On abstract syntax: being declarative is not the holy grail

In the world of Model-Driven Development and Domain-Specific Languages the focus tends to be more on the abstract syntax. The core tenets of Model Driven Development, or Model Driven Engineering if you will, are abstraction and automation. Normally if you want to create a language that is easy-to-use by domain experts (often non-programmers) you focus on creating an abstract syntax that leaves out technical details and is as declarative as possible, right? Do not focus on technical details (abstract them away) and specify the “what”, not the “how” (declare the program).

However, abstracting away technical details does not automatically make a language easier to understand. It all depends on what the relation is between the language concepts and the actual behaviour of the application you see (remember: cause and effect). A low-level language creates too much of a distance between cause and effect as, for example, machine-level instructions are not easy to relate to application behaviour. On the opposite side we have language that are too high-level and thereby create a difficult relation between cause and effect too. If one line of code (or a single activity in your process diagram) leads to the execution of a range of actions that can result in a plethora of results, it can be hard to grasp the impact of changing something.

This leads us to the subject of declarative languages. Declarative languages can be very powerful as you just declare the result, not how to get it. Example languages are SQL (well-known, more declarative than procedural), Prolog (if you studied computer science you probably know it), or the languages used by most business rule engines. The nice thing about declarative language is that you abstract away all the “how”, you avoid implementation details. What these languages promise is that programs created with them are easier to understand and maintain, and sometimes that’s completely true.

However, what always bugged me is that users of our App Platform learn to use our procedural DSLs quicker than the more declarative ones. In hindsight that’s maybe not that difficult to explain. It’s probably best understood if you imagine a large system that consists of 1000 rules (or predicates) that are automatically woven into a working system. Do you think a “programmer” can easily connect cause and effect when changing a rule? The lesson: being declarative shouldn’t be a goal on its own, it can even make things more difficult if not applied well. Please note that I focus on the learnability of the language; productivity, conciseness, etc. are different considerations that may influence your choice for declarative languages.

In the end, the abstract syntax of a language should help to reason about the problem you are solving using the concepts of the language. It should be a proper abstraction of the resulting application, this abstraction can be as high-level as possible as long as this abstraction decreases the “distance” between cause and effect. The abstract syntax, the concepts of the language, should also help to decompose a problem in manageable pieces and glue them back together as a complete solution.

On semantics: live programming helps to understand cause and effect

The semantics are the most important aspect for people to learn and understand a language. Semantics connect cause (a certain language construction) and effect (the result of executing that language). As explained in the previous sections the choice for concrete and abstract syntax is important as it defines the “distance” between cause and effect and hence how easy to understand the semantics of the language are. In addition to the language itself, the tools are of the same importance in conveying the semantics of a language.

The environment you use can, for example, greatly enhance the readability of a language by providing mouse-overs, integrated documentation, highlighting, etc. But an environment should do more. It should help you to sculpt your program. It should provide refactoring options that help you to start concrete and refactor to more abstract code when needed. It should provide ways to step forward (and backward!) through the program while inspecting state and behaviour.

A great debugger can really help you to learn and understand the cause-effect relationship of your language and the actual execution. A nice example is the visual debugger we feature in our platform, and you maybe also know the Eclipse Java debugger (or even the Smalltalk debugger) that allows you to change code during a debugging session (within some limits).

Changing code and directly seeing the effects of it (dubbed “live programming” sometimes) is the ultimate way to closely connect cause and effect. If you combine that with stepping forward and backward through the execution (and thus modify state and code at the same time) you have a powerful environment that really helps a user to relate cause and effect and hence understand the language.

Warning: live programming really helps in understanding the language, but not on its own. Please do not forget the things I previously mentioned about concrete and abstract syntax. Cause and effect can be clearly connected, but if you cannot understand that connection when you read the language it will still be hard to understand the language.

Conclusion

If you want to design the next programming language that is easy to learn and use, you should first understand how people learn. The relation between cause and effect plays an important role in the learnability of a language. If you want to clarify the cause-effect relationship you should focus on both the language design and the tools.

If you like this subject you should definitely read this excellent essay (not written by me, you can safely click the link and will not regret it ;) ) that goes into great lengths to explain how we can let people understand programming.

A tale of a 7 year journey in developing software for the enterprise

Sat, 05 Jan 2013 16:07:00 +0100

Much is written about lean startups, agile software development, continuous integration or even continuous deployment. All with the goal to help us understand the dynamics of successful software companies and their development teams. I am inspired by these stories, they show me what the ideal situation is like, they challenge me to improve our current way of working.

Today I want to give something back. I want to share with you our tale of developing software for the enterprise over the last 7 years at Mendix. Not to show you the "ideal situation", but to share with you the challenges we encountered in scaling our teams and in getting customers that depend their business on our App Platform. We experienced exponential growth ever since we started and much of the phases we went through are not specific for our situation, so I hope you enjoy the story and perhaps learn something from it.

Start up: what to build?

I never forget that moment, more than 7 years ago when I first entered the "office". Just a few people in one room starting a business. It felt exciting and I couldn't wait to be part of it. One day later I was sitting at a desk with code in front of me (and two other people constantly using the phone for "sales").

Did we know what to build? Did we know what our product would look like, even in a month's time? No. We did have a clear vision, though. A vision that, we believed, would change the world. Our one and only goal in these early months was to create features as quickly as we could. To make our vision tangible and share it with potential customers. To say it in other words: we quickly iterated with the "market".

The first customers: we need this feature, it blocks this important deal

And then it became even more exciting: the first customers started using our software! However, there was often an accompanying message to the development team: we really need these new features otherwise we cannot deliver what we promised. The situation for our development team in principle stayed the same: cranking out new features as fast as possible. Only we did it for real customer cases this time, instead of for "just" our own vision.

I think every developer recognizes this situation where sales people "demand" new features to be able to close a deal. In this phase of your company you should welcome this (within certain boundaries of course) as your main goal is to search for the right market. The main challenge often is to make sure you do not keep doing it this way, to determine the right moment to switch to a more structured approach.

In the first year we mainly flip-flopped between the customer- and vision-driven method of operation. How long you iterate between these two modi depends much on the stability of your vision and how well it fits with the actual market you find.

Stabilizing: not all customers want to "live on the edge"

Luckily we found a market that was waiting for a product like ours. There was a lot of pain we could take away with our product. So, we stabilized our product focus and our user base grew quickly. This doesn't mean our messaging and positioning has been the same since then. We continuously change and improve those, but the core vision of our product is the same today.

A growing customer and user base also introduced new challenges. No longer was our focus just on cranking out new features as quickly as possible. There are these customers, you know, that value stability over new features. Whether they do depends on the "maturity" of your customer, but even if they do not value stability, they will quickly learn to. It goes like this: the first time you tell the customer "we fixed the issues you reported and we also added this cool new feature" they will be very happy. But, after learning that every tiny change in software also brings new risks they will start asking for releases without these "cool new features".

I know, your quality assurance should be so good that users of your product trust every change you release. However, no quality assurance approach can match the "quality increase" a piece of software goes through when it is hardened in production. So, when developing mission-critical software you will end up with different types of releases. In our case we just followed the best practice of having major, minor, and patch releases with the accompanying 3-part versioning scheme (x.y.z).

The additional challenge this brings is how to plan these releases. Ideally, as a development team using an agile software development method, you have a single backlog and you work on a single sprint. Just work with different branches in your version control system to make sure you can release features separate from bug fixes. However, we did experience two difficulties with this approach. First, it is becoming a full-time job to discuss the importance of incoming issues with the caller. They quickly learn that the resolve time of their issue depends on how well they explain the severity of their issue. Second, it is difficult to keep focus if you mix issues that address any part of your product with the tasks that cover the specific major feature you are working on.

We ended up with a process we still happily use today. We divide our work in chunks of 8 weeks. Each 8 week period contains the following elements:

5 weeks of uninterrupted "roadmap" work. We can really focus on difficult problems, be creative, be innovative, bring our product to the next level. These 5 weeks are divided in multiple sprints and each sprint results in a working piece of software. One or more of these 5 week periods result in a major release.
3 weeks of "maintenance" work. This period results in a minor release and is focused on processing defects. We include minor features too, but only if the use case is very clear and the risks are low. The changes we do in these minor releases are always merged back to the main line and will thus be included in any future release.
2 "FedEx" days. A monthly day during which developers build a new feature they like, research a new technology, etc.

Organizing our work in this way gives us the following advantages:

Our users know that we do a low-impact minor release every 8 weeks. Only if a defect is so severe a user cannot wait we have a discussion about a patch release. In general our users really like the clarity of the release scheme and issue deadline before each release.
It makes product management easier. It is very difficult to decide on the priority of small(er) feedback items versus strategic product changes. Now we always dedicate time for the smaller feedback items and we know the amount of time we have for "roadmap" work.
It ensures time and focus for innovation, for bold new ideas. It is so easy to just be busy with maintaining the current status quo. You will keep your current users happy, but risk becoming irrelevant in no time.

So, business is coming in, users are happy, development is structured, enough said right? Well, that could have been the case, but when going through these changes we didn't see the dark clouds gathering on the horizon...

The 2nd system effect: let's do it all over again, but better (we thought)

When you develop a product like ours, you make choices, lots of choices, everyday. Especially when under pressure to deliver, you need to make choices about leaving out features or implementing a feature less elegantly than you would have done in an ideal situation. Combine that with all the new insights you get when your product is used by more and more customers and slowly you get the feeling some fundamental elements need to change to keep up with the growth and the new usage you see (and didn't predict before you started). At least, that happened to us. We clearly identified a new vision for our product, one that extended the initial vision we had. We were not afraid to aim high, to have ambition. It truly was a so-called Big Hairy Audacious Goal.

So far, so good. We had this great vision and we used our roadmap time to start with the implementation. During the maintenance periods we maintained the current version of our product. The first problems arose when we were in the middle of refactoring each and every element of the core of our system when an important, bigger feature was needed by our users. What to do? We only had one option: build it in the current stable line and just keep working on the new major stuff in a separate line. Guess what happened? A couple of months later almost the complete team was working on the "stable" line and the new amazing upcoming major release parts were gathering dust somewhere in a corner of our version control system. Finally, more than half a year after we started, we decided to kill the development of the new major release. In hindsight that was one of the best decisions we ever made, and I can tell you that it took a lot of courage.

What did we learn from this experience? First, these kind of "big mistakes" quickly become some sort of a legend that is told to every new team member that joins our company. Second, we realized we had fallen victim to the so-called second-system effect. This effect refers to the tendency of small, elegant systems to have big, feature-creeped systems as their successors (luckily we never released the successor we had in mind). You try to compensate for all the stuff you didn't get to do in the first version of the system. We thought it was heroic to take the time to re-think everything and to dare to re-implement big parts of our product. We were partly right, it wasn't the re-thinking that was the problem, it did go wrong in the way we defined "working software", in the way we defined the slices of work for each sprint.

We still worked in sprints and did sprint demo meetings with stakeholders involved, but we focused too much on horizontal slices. Slicing a system horizontally means implementing a system one layer after another, e.g. first build the database layer, followed by the business logic layer, and finally the UI layer (or any other layers you have in your system). Using this approach you cannot show a real end-user feature until you finish almost all layers. Slicing a system vertically means developing just the parts of each layer that are needed for a specific feature, followed by the parts that are needed for the next feature, etc. Vertical slicing enables to deliver working functionality as soon as possible. It also helps to focus the development of the underlying technical layers, i.e. only build what you need.

By focusing on horizontal slices we moved into a modus of brainstorming and speculating about everything that was needed in a certain layer. This results in discussions and ideas that lead to an endless increase of the scope. In fact, years later we can still refer to this version we were trying to build. When we start to build a new feature we often realize: "this was also part of our big master plan years ago, right?". The amount of times this happens is truly astonishing ("did we really think we could include all this in a single major release?").

So, we changed some things, or better formulated, we emphasized some parts of our working process. First, the 3 week maintenance period is for minor stuff, if it doesn't fit in the 3 weeks, it should be part of our roadmap time. Second, during roadmap time we focus on vertical slices and work in sprints with a maximum of 2 weeks that always result in working software.

Sounds trivial right? Why is it then that so much people can relate to a story like this? I think because doing it right takes a lot of continuous discipline. As developers we tend to think about the horizontal slices when we plan the implementation of new features. Furthermore, a focus on vertical slices tends to create waste. You have to build stuff that you know you will throw away if you implement one of the next features, or you already know what other things you need in a certain layer, better create it directly, right?

We had this experience in the past, but it is an ongoing challenge to keep our focus. From time to time we need to "reset" ourselves and make sure we focus on having a big vision, but operate in small steps. What we truly experience: complexity is easy, simplicity is difficult. And this tendency to the "complex" isn't the only challenge we had in keeping our focus...

Ramping up: introducing "departments"

Focus is more than "not doing everything at once". It also means uninterrupted time to really focus on doing highly-skilled work. Our 8-week rhythm is helping with that, but as you can imagine we cannot close the doors for 5 weeks if we are in our roadmap period. However, we also know that a developer that operates in "the zone" is exponentially more productive than one that is interrupted multiple times per hour. If your company grows this becomes quite a challenge.

What we did over the years was introducing departments around product development. There is no definite guide that tells you when to introduce a "department" and how, but these are my main observations:

As long as your company has less than 20 employees you won't need formal departments. You probably do not even have team leads or "managers" except for the founders of the company.
Once you grow bigger than, let's say, 20 employees you will probably need to introduce team leads and departments as the founders cannot manage everything themselves anymore.
There will be a point in time when the team lead(s) of your product development team(s) are completely occupied with "operational stuff". That's a dangerous situation and you have to detect the early signals and make sure you start to "outsource" part of the work to other (probably new) departments.
From a development team perspective one of the first things you will need are people that connect new customers, and in our case to actually implement a solution for customers using our product. As a company you have to do that yourself to make sure you have the right "lighthouse customers" that help you attract new customers (customers and users of your product need examples to be convinced to start using your product). The dynamics of this type of work are completely different from product development, hence you do want to split that to keep your focus.
Another type of work with completely different dynamics is "support". You do want dedicated attention for that as early as possible. Embedding a good support process in your organization that ensures knowledge sharing is a key element to scaling your organization. No proper support means that you become victim of your own success: more customers means more developer "distraction". There are of course multiple ways to solve this, but in the end you want a team with dedicated attention for users, creating FAQs, handles incoming issues, etc. And, do not forget we are talking about enterprise software, hence we have some SLAs we need to comply with.
Tech sales is also something you want to formalize quite early. Selling software to enterprises means answering a lot of questions. And not just an explanation of how your product works, that's something that needs to be explained and written down by developers that build the features in my opinion. But, think about the positioning of your product compared to other types of products, compared to competitors, creating and giving demo's, convincing customer architects, etc. Again, there are smart ways to do this, but someone needs to do it.
What you should NOT just split out of the product development department is "operations" (i.e. everything that is needed to run your software - we're talking "as-a-Service" here). I strongly believe in cross-functional development teams that are responsible for running their own parts of the products (you built it, you run it). This creates a culture of responsibility, developers will think about the consequences of their changes thoroughly. It also ensures that a vertical slice for a feature incorporates the operational aspects, meaning that finishing such a vertical slice really results in working, releasable software. You can (and probably should) of course create a separate team that supports the other teams in the way they deploy new versions of their software, monitor it, etc. by automating it and providing the developers with the right tools.

In gradually moving to a situation with more "departments" we were able to return to more focused development and enough time for innovation. Every change creates a new challenge, though. Let me tell you a story... after a short intermezzo...

Knowledge transfer: how to keep other departments and users up-to-date?

Outsourcing part of the work to other departments (other than your product development teams) leads to more "distraction" instead of less in the short time. The main reason: knowledge transfer. Support and pre-sales for example need to know everything about the product, new releases, etc. We have seen two things that really help with this.

First, you need developers with writing skills. I do not mean "writing code" but rather documenting new features in a reference guide, explaining how to use new features in a how-to, and explaining the rationale behind new features in technical blog posts.

Second, we have a "tech-of-the-day". That's one of the developers who is available for answering questions from other departments. We rotate this role every day. This approach makes sure that other departments can always approach someone with in-depth product knowledge and at the same time it ensures focus for all other developers. It also ensures that each developer is exposed to user feedback and discussions about how to use or sell our product.

The ivory tower: how to keep the feedback loop connected?

Let's go to the story I promised you. So, we released this cool new feature. We, as a development team, were enjoying the moment. You have to celebrate your milestones! However, within 5 minutes someone from our "professional services" department stood at my desk with a less joyful face. He had just glanced at our new release and then ran to my desk to explain to me: "this feature you just released is completely useless". That's fun, right? What wasn't so funny was the fact he was right.

How is it possible that we were crafting a new feature for weeks and thought it was the best thing since sliced bread, while one of our users could see the flaws in 5 minutes? Shielding the product development team as much as possible from external "distractions" brings some serious issues with it. Isn't one of the core principles of agile software development to value "customer collaboration"? That's for a reason. Without the collaboration with your users and stakeholders you are building stuff you think is valuable, but the better and longer the shield works, the more you are moving away from reality. Product development slowly turns into an Ivory Tower. How to find the right balance between focus and collaboration?

To start with: these two things, focus and collaboration, are not mutually exclusive. However, we have experienced first hand that finding the right balance is a challenging, continuous process. Especially in a fast growing company. Let's look at some things we introduced over the years (some more recent than others) to fight the creation of an Ivory Tower:

The most logical moment to involve users in the development process is the sprint review meeting. This meeting is meant to demo the completed work to the stakeholders. We always try to involve a user in this meetings. This is often quite challenging as these users have a busy schedule. If it really isn't possible to have a user in these meetings regularly the Product Owner should put more effort in aligning with users (at more flexible times) and being the "voice of the user" during the meetings.
We start new projects with an "inception" phase that is used to identify the vision and story for the release, bring stakeholders to agreement, and create an initial idea of the user experience. We literally start with writing the press release for the release, present it to stakeholders and write down all the questions that arise (and their answers) in an initial FAQ. This really helps in guiding and scoping development and involves users as early as possible. What we essentially do is to start with the customer and work backwards until we get to the minimum set of technology requirements that satisfy what we try to achieve. Using such a continuous, explicit customer focus we aim to drive simplicity throughout the process. But again, we continuously need to remind ourselves: have a big vision, operate in small steps.
All of our products include a feedback mechanism that makes giving feedback really effortless. This increases the chance that a user shares his thoughts, frustrations, and awesome ideas with us. This of course also brings an additional challenge: processing all this feedback in such a way users feel their feedback is valued and taken seriously.
We sometimes invite users of our products to present at one of our Friday-afternoon-beer-sessions. In our case this means we have users from partner companies that present the enterprise apps they have build for their customers using our PaaS. This really helps to broaden the perspective of our development teams. Additionally these users are always asked to share with us what they like and dislike. In our experience users are always happy to share their work (and opinion) with us, and they are rewarded: almost every time they present something that really annoys them (or a hint for something they really would like) we can just include that in the next minor release.
Product management should not purely be based on user feedback. Product management should be based on multiple influences, amongst them: your own product vision, market trends, marketing, sales, research and development, user feedback, etc. That being said, product management is a tough job. Most of the time you should be able to value different inputs and say "no" to a lot of them. If you want to please everyone, don't become a product manager, you will create awful, feature-laden products. However, you shouldn't become arrogant too. Using just your own vision as input will not lead to the best product.

This list is by no means complete. We feel we still need a lot of improvement in this area and it has our continuous focus to improve this.

Entering the real enterprise world: when you thought functional requirements are the only things that interest customers

We talked a couple of times about developing "enterprise software". What do we mean by that? Actually, that changed over time. We did not start with a fortune 100 company as customer. What we first saw as a big company is small compared to the current customers we have. And in delivering software to the real big companies we also encountered additional requirements. If you ever built enterprise software this shouldn't be a surprise.

First, the product or service you deliver should be covered by a Service-Level Agreement (SLA). I am not a legal expert, so I won't go into the details. Just one piece of advice: make sure you carefully read and understand the SLAs of the services your product depends on. If you for example build software that runs in "the cloud" like we do, you need to understand the details of the SLAs of the Infrastructure-as-a-Service (IaaS) providers (hint: they can be meaningless). Your own SLA is limited by the SLAs of your providers, if it isn't you probably want to contact an insurance company.

For us as a product development team the nice thing is that these kind of agreements with customers lead to higher uptime requirements. Hence, more time on the roadmap for things like high-availability, horizontal scalability, failover mechanisms, etc. Which is interesting stuff, right?

Another requirement some of our customers (especially in the financial and healthcare industries) have is that we as a company are audited every year. These customers are under a lot of regulations and are only allowed to do business with companies that comply to a number of standards. Examples are the ISAE 3402 and SSAE 16 standards. So, what do auditors look for in your product development processes? In my experience these are the main elements:

A traceable software development lifecycle: have a look at the current version of your product that runs in production. You have to be able to show who approved the release and when, what the test report was, the source code for that release including the commit history, and the requirements and design decisions for the new features in that release. This maybe sounds bureaucratic, but with the proper tooling it is fairly easy without much overhead. We use a version control system connected to a Scrum tool (part of our own platform) containing the user stories (including related discussions), the results of our test runs (connected to builds) are saved, and we have a lightweight approval process for releases (just me giving a written approval as response to a request by one of the development teams that contains build number / revision and test report).
Disaster recovery: you have to have a disaster recovery plan (and not just for audit purposes). You have to test the plan at least once a year and document that. Again not just for audit purposes, an untested backup (or failover environment) equals no backup.
Maintenance processes: you have to have change management processes in place that make sure that changes on environments are properly scheduled, approved, and traceable. Furthermore, you have to establish processes that make sure that (security) patches of the third-party software you use are noticed and applied as soon as possible.
Documentation of procedures: most people I know aren't fans of audits because of the focus on documentation and I have to admit that was my point of view, too. Documentation is, of course, the only way an auditor can verify your processes, however, in my experience most are quite pragmatic and do not ask for multi-page process descriptions. You can keep it short and to the point. In a growing organization with a growing product you inevitably need more documentation. Not everybody will automatically know and understand everything (as opposed to the early days) and new employees need to learn a lot. Early-day employees often don't realize how complex the current organization is for new employees as these early-day employees never had to learn everything at once, they have just grown into it.
Staff training: for me this is probably the most "difficult" part of the audit. You need to show that your employees are capable and know their stuff (especially with regard to security). That's difficult to show if you have a team of A-players that are autodidactic and use blogs and social media to keep their knowledge up-to-date. We hire people with such skills, we do not look at certifications. So, we do not organize formal trainings. However, organizing an in-house training day (read: a hacking contest - learn a lot about security by hacking example systems in an increasing order of difficulty) with an attendee list and short description of the day can often do the job too.

The main challenge? Fighting the obvious: becoming a bureaucratic, inert organization. So, keep it lightweight and be pragmatic. Focus on the reasons behind these questions and controls, in the end it is all about quality and continuity of service, which are good things to focus on.

SLAs and audits are just two examples of so-called non-functional requirements for enterprise software. In all this, it is again about finding the right balance: non-functionals need more attention than in the early years, but do not stop the innovation from a functional perspective.

Scaling the team: how to keep the "startup" vibe?

If you are still with me, you probably sensed the growth we had over the last years. That growth also influenced the size of our product development team. We started with just a few people, the company was one team. As explained, over time "departments" were introduced and thus a product development "department" was created. Currently, a couple of years later, our product development department consists of 5 teams and within a couple of months team 6 and 7 will be created as teams that grow bigger will be split.

In product development we didn't grow very fast. If you grow too fast, teams are more busy with getting new people up-to-speed than with new versions of the product. Furthermore, it is important to not only get people up-to-speed with the technical work, they should also become part of the culture, they should breathe it. We call that: painting them Mendix-blue. Your culture is a competitive advantage that cannot be copied, foster it! Part of a focus on "culture" is the way we evaluate if someone fits in the team. We do not just look at skills, we let them talk with some team members and see if they add up to something more than the sum of the individual people. It is about passion, enthusiasm, and thereby pushing each other to excel.

So, culture is important, but how to stay focused, aligned, and flexible while also growing the product development team? Here are some of the things we do to prevent ourselves from becoming the next big inert organization:

We keep the teams small (8 people at maximum) and autonomous.
Teams are cross-functional. A team is responsible for a product, part of product (a service), or a certain feature. This responsibility includes everything from design and implementation to releasing and running it.
Each team has a Scrum master who is part of the team. In principle a Scrum master is a developer with an additional role. With the small teams we have that's working successfully.
Everybody is a "maker". Everybody creates stuff, whether it is code, tests, infrastructure automation, or designs. We do not have multiple layers of "management".
We have weekly Scrum-of-Scrums meetings to align the teams. Each team writes down the "what did we do" and "what are we going to do" stuff on beforehand. During the meeting (which is attended by a team member from each team) we discuss impediments and resolve issues that affect multiple teams. Everything we share and discuss is posted on our internal "social network" so that each team member always knows what's going on within the other teams.

Much more can be written about the subject of growing a product development team. For now, just a final observation: system architecture tends to follow team structure, even if communication among teams is well-organized. Keep that in mind when you think about your team structure.

Moving on

Well, this piece became a bit longer than initially planned, but I hope you enjoyed it. Please share your experiences, comments, or questions in the comments!

We will continue our journey and continuously improve our way of working. Because it's fun, but also because it's necessary. We need to stay innovative, we need to challenge ourselves, to be able to stay successful.

Let me finish with a meta-observation: the word "focus" is mentioned quite a lot of times in this story (30 times actually). That surely aligns with my experience: nothing beats a motivated, highly focused team of professionals with a clear goal!

By the way: we are hiring for multiple development positions including a software development manager / agile coach. So, if you enjoyed this article and want to join (and impact) our process of continuous improvement you should check out our open positions.

You can also discuss this article on Hacker News.

3 steps to free your business from the IT stranglehold (and vice versa)

Tue, 27 Nov 2012 09:28:00 +0100

It's not just IT that slows down the business. In a recent study 20% of companies reported they have NO innovation strategy and more than 50% of companies reported that their innovation strategy was mis-aligned with their business strategy and that their culture poorly supported it [1]. However, if we look at the IT side of an organization we often see the same kind of figures: 70% - 80% of IT budget is used just to run and maintain existing apps [2]. Maybe that's even explainable since starting a new IT initiative is often a bumpy road: 68% of all new software projects are NOT successful [3].

These numbers aren't new. Why do we all know them and, yet, they seem to stay the same? The interesting thing is that business nor IT is denying the current state-of-affairs. IT and business are equally frustrated about the current situation in most companies. When asked "are you happy with how IT is proactively engaging with Business Leaders to drive innovation?", 74% of the non-IT execs state they are unhappy and 70% of the IT execs [4].

It seems like it is time for change...

I think the next 3 steps will provide a way to really change these numbers...

Step 1: start to believe in an agile enterprise

It wouldn't be strange if you accepted the status quo: change is difficult, change is slow, especially in a big company. However, that's not how it ought to be.

I believe that successful Enterprises:
Are agile.
Can respond quickly to changes in the market.
Because all departments are fully integrated in the overall value stream.
They have a big vision, but operate in small steps.
And gather feedback continuously and early-on.
They always work back from the customer and let customer-value drive new initiatives.
They focus on eliminating everything that's not contributing to the value stream.

I believe that such Enterprises need Apps:
Flexible and focused apps.
That perfectly fit the business.
They are easy to find.
And easy to access, from any device.
They have an integrated user experience.
Their lifecycle is agile.
They change along with the business.
Users are engaged and their feedback is processed fast.
New apps are build in a collaborative process involving all stakeholders.
And they are easily delivered to the selected user base.
They align with enterprise needs around security, integration, and governance.
Instead of slowing the business down, they drive business innovation.

Utopia?
"Live your beliefs and you can turn the world around." (Henry David Thoreau)

Step 2: add a ‘new productivity platform' to your enterprise architecture

If we really want to change the way IT supports the business, we need to change the way we build new apps and manage the lifecycle of apps. Most IT teams are struggling with the amount of work they have, often resulting in a "no" to the business if they demand new business functions provided in software. And let's just not talk about the constant change of business processes and policies that need to be reflected in existing software.

According to Forrester analyst John Rymer, Java and .NET are often no longer the best choices for the fast delivery of new business applications. Instead, we need, what he calls, new productivity platforms, to speed-up initial application delivery and ongoing updates. He defines "new productivity platforms" as [5]:

Platforms that speed application delivery and ongoing evolution through: visual tools, hot deployment and continuous innovation, built-in administration and management, and active participation of business experts in application delivery.

In my view these platforms are a special breed within the Platform-as-a-Service (PaaS) category. I'd like to call this sub-category Application Delivery PaaS to emphasize the focus on the complete application delivery lifecycle and not just deployment (as unfortunately a lot of PaaS platforms do).

Figure 1 - The App Delivery Lifecycle (source)

Figure 1 shows the four important phases in the application delivery lifecycle. The phase at the top is about requirements capturing as a collaborative process among all stakeholders. This collaboration is continued in the next phase when the new ideas and requirements are converted into models (or existing model templates are selected to fulfil the wishes of the stakeholders). In this phase the focus is on highly productive, collaborative development using visual models. These models are 1-click deployed to a runtime environment where they are executed and become real apps (I dubbed this Model-Execution-as-a-Service in the past). The key in successful app delivery is to continue to the next phase and engage with users, listen to their feedback, and use that feedback to come up with new requirements, thereby continuing the next cycle.

The phases are not numbered on purpose. The only way to really improve app delivery is to create shorter feedback cycles to increase collaboration with all stakeholders within an app delivery project. Where you start doesn't matter, the app delivery lifecycle is a continuous process in which each cycle is as short as possible. And yes, this process should fit in an overall agile process to be able to function in the way described above. Or, in other words: the agile process needs to be extended outside development, we could call this 'enterprise agile'.

In my opinion you should start to use an Application Delivery PaaS. The ‘agile enterprise' as described in step 1 will become much more within reach.

Step 3: know what's agile and what's not

If you are an enthusiastic user of one of these "new productivity platforms" (an Application Delivery PaaS) you may think this is the way to go for all application development. Something along the lines of "if all you have is a hammer, everything looks like a nail" [6]. Don't go that way.

These platforms aren't good for everything. Their real value shines when used for applications that need to be agile, that change on a weekly or even daily basis. You probably don't want to build your bookkeeping software from scratch using such a platform. You should look at your processes and applications and distinguish different categories. Gartner distinguishes between commoditized processes supported by commoditized applications (e.g. bookkeeping software in most organizations) and differentiating processes supported by differentiating apps. Differentiating processes are the processes of your organization that really distinguish you from your competition. These processes will probably change frequently and time-to-market is important.

Ron Tolido (CTO at Capgemini) describes it way more colorful in his whitepaper about the five different application lifecycles that address different IT dynamics in organizations [7]. He uses a transport metaphor to identify these five different application types:

Trains: stable, robust, standardized, no customizations, lots of users.
Buses: also relatively stable, but more flexible. Some organizations may choose to own or hire their own buses.
Cars: much more agile, individualized means of transport. Owners will adapt and configure them.
Scooters: lightweight, extremely flexible and individual method of transport. They can bring you anywhere and you can even rent them for just a day or so.
The hub: connecting the above.

You need to address each of these application types in a different way. In short: buy standardized solutions for trains and buses. Build apps using an Application Delivery PaaS if you need cars and scooters.

Conclusion

To summarize:

start to believe in an agile enterprise,
add a ‘new productivity platform' (an Application Delivery PaaS) to your enterprise architecture,
and know what's agile and what's not.

If you do so, I'm sure the numbers mentioned in the introduction will really start to change!

-------------------------------

[1] Booz & Company - 2011 Global Innovation 1000 Study

[2] "In IBM's experience, the 70-80 percent figure is roughly correct; little funding is left for innovation" https://www.ibm.com/developerworks/mydeveloperworks/blogs/invisiblethread/entry/enabling_smarter_decisions?lang=en

[3] Chaos Report 2010 by the Standish Group

[4] McKinsey Quarterly Dec 2011

[5] John R. Rymer, The New Productivity Platforms: Your Solution To The AD&D Crunch. November 1, 2011.

[6] Abraham H. Maslow, The Psychology of Science, p. 15, 1966.

[7] Ron Tolido, From Train to Scooter - Five Application Lifecycles That Address Differing IT Dynamics Within Your Organization, 2011. http://www.capgemini.com/insights-and-resources/by-publication/from-train-to-scooter/

The Evolution of PaaS in the Enterprise

Wed, 17 Oct 2012 22:33:00 +0100

This morning I was part of a panel at the GigaOM Structure:Europe 2012 conference in Amsterdam., titled: The Evolution of Private PaaS solutions. The abstract:

Enterprises are starting to take interest in running PaaS solutions virtually, as app developers want to focus on building apps rather than dealing with infrastructure issues. Enterprises that use PaaS solutions almost always go down the private route. In this session we focus on private PaaS offerings and look at the considerations and what will happen if one day enterprises want to use PaaS solutions in the public cloud.

The idea was to discuss private vs. public PaaS and, as it was a theme throughout the conference, the differences in adoption rates in the US vs. Europe.

You can view the full video coverage of the panel at the GigaOM website.

Here is an overview of some statements made during the panel to highlight some of the discussions (in my own words, so could be biased, but you can watch the video to make up your own mind ;) ):

Public vs. Private PaaS

Enterprises want PaaS within their own environment, behind their firewall, i.e. Private PaaS.

Public PaaS is focused on the lowest common denominator, limited to certain framework, database, etc. Private PaaS adapts (can/should adapt) to everything used in the enterprise, any language, framework, database, etc.

In our customer base we roughly see a 50%-50% split between applications deployed on our Public PaaS vs applications deployed on our Private PaaS. It mainly depends on data, integrations, and policies.

Mobile is actually helping public PaaS adoption. It is a new subject, and within such a new space enterprise seem to be more open to new ideas.

PaaS and productivity

PaaS is a huge enabler for Cloud Computing. PaaS is "clicky-clicky": development and deployment that took months brought down to minutes.

PaaS is more driven by productivity gains than the need to move to "the cloud". The business wants a solution for a business problem. Fast. Keep evolving with the changing business. PaaS should cover the complete application lifecycle.

A PaaS covering the application lifecycle in this way requires a new way of thinking. Connected to agile, continuous integration, etc.

PaaS can help to transfer old way of working seamlessly to cloud. It is just a small step.

No, you need to change the way you work. Modularize applications, no application silos.
If you really want to speed-up. We should also stop programming, start doing MDD.

MDD is possible now... no programming involved perse (also read: Why aren't we all doing Model-Driven Development yet?).
Enterprise are adopting it.

PaaS should add a layer of abstraction. It should add a software layer. Key is that it is just the first step. New language constructs will abstract even further.

PaaS and the application lifecycle

PaaS can bring harmony between operations and development.

The opportunity is even bigger: it can bring all stakeholders in the application delivery process together. Even end-users providing feedback, and business owners collaborating on requirements. The complete application lifecycle should be covered by PaaS. Not only covered, also speed-up.

Even end-users doing app development and deployment?

Back to private vs public. I believe in hybrid. Even if enterprises want to go with private PaaS, they often want to have development and test environments in the public cloud. Once they start using production data in acceptance or actual production they move the app within the premise of the company.

Real-time updates and modularity

OSGi has been mentioned a couple of times, what is it? A dynamic modularity system for Java. The only industry standard that enables modularity, currently in the Java space but will become available for different languages.

But... enterprises are not asking for OSGi or these kinds of standards.

No, they do not ask for OSGi specifically. But if an enterprise starts to user your PaaS you will encounter something like this: build first small app fast, in an agile way with a small team.

If they start to build a huge app, i.e. with a lot of functional components, people will start to think about modularisation. If you want to release a new feature you do not want to redeploy the complete app. But only the changed component, and if needed the dependencies of this component.

Customer ask indeed about real-time updates in PaaS.

Version/configuration management becomes important: what version of my software is running. And what versions of all the components?

Layers

PaaS is about virtualising away lower layers. However, PaaS can also bring back to the table an understanding (and management of) what an application actually needs with respect to the underlying infrastructure, e.g. data needs to go over the same switch, caching is sensitive for locality, etc.

There is no clear distinction between layers. IaaS and PaaS players are moving into each others space. Amazon started out as IaaS, but nowadays delivers a lot of higher level services that you could categorise as begin part of the platform layer. Microsoft Azure and Google are moving the other way around: from PaaS player to also delivering IaaS.

Is there room for language specific PaaS?

It depends: if you only have .Net it makes sense to have a .Net specific PaaS. However, most organisations are heterogenous. So you need to support multiple languages.

You see that because we are in a transition, people still use their current methodologies. As PaaS becomes stronger, full stack, there will become PaaSes that focus on one language. That will need a different way to build the application. Because you need advantages like auto-scaling, dealing with locality, etc.

If you really want to change, you need to disrupt. Do not cling to your existing methodologies and languages. I truly believe there is a big opportunity for PaaSes with focused languages.

Forrester defined space within PaaS: new productivity platform. Step to higher level languages. Different roles that build applications.

Platform should run components defined in all kinds of languages.

It depends on how you define PaaS. It is not just a deployment platform. It should also cover the other aspects of the application lifecycle like development and maybe even requirements/feedback management.

If you really want to disrupt the field and make it a lot easier and faster to build applications (that's in the end what businesses are looking for), than you also need to change the way you develop applications not just the way you deploy it.

PaaS adoption in Europe vs. North America

Interesting thing last 2 years: not really a difference. Everywhere businesses are looking to improve their software delivery. There is no business innovation nowadays without software delivery involved. Businesses want improvement, productivity gains.

Then the question is will it be a private or public PaaS. We see the same thing in US and Europe: it depends on the kind of data and the kind of company. Banks will probably go for private. But a lot of other companies will go with public PaaS, especially for their customer facing portals.

It all depends on the usecase. Not on region.

PaaS often starts small.

That's an opportunity. We take a week to create a working app for new customers and they have something tangible to show within their own organization.

No difference between Europe and US. Almost all private PaaS.

It seems like we are saying that if you are an enterprise of course you will have private PaaS. That's not true. A lot of big companies are using public PaaS. Of course there are enterprises that go with private PaaS, but it is use case dependent. Not dependent on the size of organizations.

Why do we see so much private PaaS? Well, it could be that if you product supports private PaaS that it is a lot easier to sell that.
People feel comfortable with private PaaS, that's what they are used too. If they get used to the idea of cloud and see advantages they will move.

Enterprise Agile: extending the agile process outside development

Fri, 10 Aug 2012 17:10:00 +0100

I previously wrote about Enterprise Agile, not because I just want to put the "enterprise" moniker in front of everything, but because I think there are some fundamental challenges in moving to agile software development practices within enterprises, as opposed to startups.

This follow-up post is triggered by the following comment on my previous article about this subject:

Where we are struggling as an organization is the whole enterprise adopting agile. Sales and marketing expect each release to have a "big splash". Services hates the rapid iterations since they have a waterfall mindset where you accept a new release every 6 months and spent time chewing on it. I would love some pointers on resources that help move these parts of the value chain to agile. Almost everything I've read focuses on Engineering and Product management and I need to think broader.

I think this is recognizable for a lot of people. I would like to share my recent musing about this subject as I am struggling with the same kind of challenges within a fast growing company.

Why "Enterprise" Agile?

Being part of a fast growing company starting with a couple of guys in one room to being an international company of more than 125 people has taught me that there is a difference between an agile team in a start-up (or even a bigger company), working in some sort of vacuum, and an agile team operating within the broader context of a bigger company. I think you can imagine what the difference can be with a big enterprise.

What's so different?

Solutions need to be fit in an existing ecosystem: the definition of done needs to include compliance, data security, data ownership, and integration with existing systems.
Multiple teams work towards a common goal: there needs to be alignment with (so-called) enterprise disciplines such as portfolio management, enterprise architecture, and strategic reuse.
Agility is only possible when the whole organization adopts the mindset: in an agile enterprise the marketing and sales side of the organization is balanced with product development. In an agile enterprise the entire business is organized in a way that it can respond quickly to changes in the market. All departments are fully integrated with the overall value stream, there is end-to-end agility.

Isn't "Enterprise Agile" moving away from the core principles of Agile?

You mean, does it deteriorate into following the Manifesto for Half-Arsed Agile Software Development? That's a fair question. It is definitely easy to end-up with something that's quite different from the core ideas of agile. However, enterprise agile as I see it is not doing that by definition. It is an awareness within development teams for the bigger picture and alignment with enterprise requirements. It is also a mindset change within the whole company as every department within a product company will be affected when product development needs to become "agile". It is not bending the agile principles to be able to say "we're doing agile" and at the same time trying to keep all existing processes in place.

What does Enterprise Agile mean for development?

If you want to extend the agile process outside the development processes, do start by looking at the development processes themselves first. Are they indeed agile? Agile development / releases are only possible with the right infrastructure in place. Think automated testing, incremental changes, agile architecture, automated deployments, focused and autonomous teams, a different (project) management approach, etc. The next step would be to connect the development value stream with "external" elements like business goals, strategy, portfolio management, architecture, training, marketing, sales, services, etc. Let's look at some of the mentioned departments to see how creating an overall agile value stream would affect them.

What does Enterprise Agile mean for marketing?

The first thing to do is to make sure marketing understands the value of small, incremental releases. Frequent releases in a steady cadence make it easier for marketing to create their own cadence in messaging, but even more important, it brings value to users as soon as possible. But what if big yearly announcements or events are needed to attract the right press attendance? Well, incremental releases do not forbid that. You can still organize yearly events, issue big press releases, etc. In most cases the group of people being happy with the incremental releases (i.e. your users) are not the main group you target with your press releases.

The second thing to do is to make sure marketing isn't seen as adding some sauce to the cool stuff engineering did come up with. Ideally software development is aimed at working on a set of marketable features. Marketing should be involved in the process even before development starts. Start the process of adding new features by writing the press release first as well as the FAQ. This really helps in defining the minimal set of technology needed to implement the feature. It also helps to integrate development and marketing in one agile process, and to make sure sending a new product release "through marketing" isn't slowing down the process, but just a step in finalizing the things that already have been started.

What does Enterprise Agile mean for services?

There are of course quite some similarities with the previous section about marketing. We should again look at awareness for agile within services and involving services in the overall product development process. The comment quoted in the introduction of this article states that services hate rapid iterations. I think they often have good reasons to do so. If each new release leads to a lot of early problems and invokes a process to re-design the way-of-working and how support is done I wouldn't like frequent releases either. That's why I started above by saying that agile development is only possible with the right infrastructure in place. Agile development is not possible if a release can only be used in production after multiple patches spread over a long period. Services should have the trust and confidence to switch to new releases quickly, you have to earn that.

Additionally, services should be involved early-on in the process too. Make sure you listen to their feedback on the product, involve them in early iteration / sprint meetings, and give them the opportunity to start acting on new releases before you actually release. If you work backwards from the press release and user experience to actually starting development (see also previous section about marketing), make sure to involve services to cover aspects like methodology (i.e. way-of-working) and support related stuff. If you issue quality releases including features valuable for your users (and thus services) I am sure services will like frequent releases as much as you do.

How to start?

Approach it in an agile way! Do not start top-down by creating policies, etc. Start with one team, as soon as you have multiple teams "doing" agile you probably need something to coordinate among teams. At this point you probably want to start with Scrum of Scrums meetings. If your development process within your team is agile, start to widen your definition-of-done. Involve other departments one-by-one and make sure upper management is supporting the move to an overall agile value stream.

Share your experiences!

There is a lot more to say about this subject. What is the impact on other departments? What are the requirements from a management and company strategy perspective? I will maybe address these kind of questions in a future article. This article is based on my personal ramblings, not a scientifically tested approach. Please enlighten me by sharing your experiences / thoughts in the comments!

Photo by Matija Grguric

7 ways a Platform can fuel the App Economy

Wed, 27 Jun 2012 17:19:00 +0100

Earlier this year a Technet sponsored study showed that in February there were roughly 466,000 jobs in the "App Economy" in the United States. This so-called App Economy had zero jobs just 5 years ago, before the iPhone was introduced.

The term "App Economy" isn't formally defined but is often used to refer to the economy that has been created due to the development and delivery of software applications for smart-phones, tablets, and Facebook.

This App Economy was created by the introduction of new, innovative platforms. I think there are 7 ways a platform can fuel this App Economy. Not just how Facebook and the iPhone did it, but also ways to further expand this economy with new innovations in a platform or Platform-as-a-Service. Although the current App Economy is consumer-oriented, platforms for enterprise apps can and should fuel the App Economy in the same ways.

Let's jump into the 7 ways an App Platform can attract users and developers to use and build apps and therefore fuel the App Economy:

1. A platform should make the delivery of Apps easy

One of the key success factors of the iPhone (and iPad) is access to great content. Developers can easily publish their apps to users, and users can easily access and "install" these apps. The App Store as a distribution mechanism has proven to be hugely successful. The concept of an App Store is becoming a key element of all successful platforms. Amazon recently introduced the AWS Marketplace and Facebook introduced it's App Center (also aimed at apps for other platforms).

A platform should make it easy for user to access content by means of an App Store (or marketplace if you like) mechanism. If a platform becomes successful one of the main challenges is to provide users with a way to find the right apps for them.

In addition to making it easy for users to use apps, a platform should also make it easy for developers to deliver apps to users. Most platforms only make the delivery of the client-side (i.e. the user interface) easy via an App Store. I think platforms should go further and provide a way to deliver the full application, including its server-side components easily. This means a platform should also provide deployment options, you should be able to put code in and get an App (i.e. a service) out. In other words a platform should become a true Platform-as-a-Service.

2. A platform should attract and bind users with a great user experience

Another success factor of platforms, and again driven by Apple, is the user experience. And not only for consumer apps. The emphasis on beautifully designed consumer products and apps is also driving the, so-called, consumerization of enterprise apps. A great user experience has proven to be a way to quickly build a big user community, and not only a big one, but also a community willing to pay for apps.

The lesson learned? An app platform should foster a great user experience by providing default layouts, guidelines, and of course by setting the baseline in it's own UI. In 2012 this also means making sure apps delivered on the platform are accessible from any device, including smart-phones and tablets (this is of course only applicable if the platform itself isn't an operating system, like iOS on the iPhone and iPad).

3. A platform should contain end-user value itself to create a base community and let developers build on top of that

What comes first? The chicken or the egg? Or, in the context of this article, a user community willing to buy apps, or a developer community willing to build apps?

In case of the iPhone the "platform" contains a lot of value and out-of-the-box killer apps to make it great, even without third-party apps. Facebook was first an app and evolved into a platform for social apps and as an identity manager for virtually any modern web-app out there. Force.com is an app platform mainly for apps (or extensions) of Salesforce.com. These are all examples of platforms that took off around an existing user community.

A platform centered around a killer-app of course also limits the applicability of the platform as the APIs and functionalities are focused on that core capability. Platforms can also be successful without a killer-app, as AWS (Amazon Web Services) and Heroku prove. These platforms are more focused on the infrastructure and deployment parts. Although you could argue that Amazon itself is the killer-app of AWS, there is not really a relation between users of Amazon and users of other apps running on AWS.

The lesson? A killer-app can definitely solve the "chicken-or-egg" problem of platforms. However, don't forget that too much focus on the platform itself or its killer-app often also limits the applicability of the platform.

4. A platform should provide great APIs to foster re-use of multi-tenant platform services and make development easier

As said before, great content is key for a successful app platform. So, make it easy for developers to deliver great content! How? By providing the basic building blocks and templates to lay the foundation for a great app. Make sure developers can focus on the great ideas they have and do not loose any time on "infrastructural" stuff.

Facebook offers for example APIs for authentication, the social graph, and game APIs like achievements and scores. Building a social game means focusing on the game itself, you do not have to develop a social graph and communication channels. iOS offers APIs for UI, media, and network, but also notifications, messaging, a game kit, and in-app puchases, to name a few.

I think platforms can go even further by providing multi-tenant cloud services to developers and their Apps. A platform provider can provide these services cost-efficiently and do the difficult work of scaling these services. Examples of such services are deployment / provisioning, storage (like Apple's iCloud), identity management (Facebook being the example), messaging (and other social services), but also higher-level services specific to the type of platform. Think about CRM or HRM services for enterprise apps, or things like Apple's Game Center for game apps.

The more business- or consumer-oriented these services become, the more they move into the direction of a killer app for the platform (as discussed in the previous section) instead of a mere building block.

5. A platform should attract developers with great tools

It is not just the APIs that attract developers to a platform. The tools developers need to build content are as important. These tools also define to a large extend how big the community of developers can become. Just imagine that everybody with a great app idea only had to express that idea with visuals models to make their dream come true. The potential "developer" community could be way bigger than the group of professional programmers. Model-Driven Development helps to take a step into this direction.

Aside from the development language, platforms can support developers by providing the right documentation, tutorials, wizards, admin tools, debuggers, analyzers, etc. In the end, developers are users too, and user experience matters.

6. A platform should support and ease the entire application lifecycle to ensure quality and agility of Apps

If a platform really wants to fuel the App Economy it should make sure the content created on top of the platform is valuable. Valuable content, i.e. apps, are apps that align with the needs of users. While user wishes change over time, apps need to change too. Apps need to have a short time-to-market and have to be flexible to evolve along with the changing needs of their users (or: "the business" for enterprise apps). This means the full lifecycle of such apps, from the initial idea to first deployment, and from that first deployment to long-term evolution, needs to be as agile as possible.

In my opinion Application Lifecycle Management (ALM) is key to the success of a platform. Development is just a small part of the application lifecycle. Platforms should also support developers in collaborating with all stakeholders involved in the delivery of the app. Platforms should empower the app delivery team with the deployment and monitoring tools they need. Platforms should help the app delivery team to engage with the users of the app (i.e. feedback management), to be sure to develop the next version of the app into the direction of the user needs. Platforms should support (and ease!) the entire app delivery lifecycle to ensure quality and agility of apps.

7. To stay successful a platform should ensure that users and developers upgrade to the latest version

The lifecycle of the platform itself should be managed too. In the same way as an app created on-top-of the platform evolves, it is important to evolve the platform to stay aligned with the wishes of users and developers. However, how do we make sure we are not creating a fragmentation of platform versions? If we look at iOS and Android we see a huge difference in the adoption rate of new versions: iOS 5 captured approximately 75% of all iOS users in the same amount of time it took Gingerbread to get 4% of all Android users.

Fragmentation of platform versions is a bad thing. Users want the most recent features and security updates. Developers want to focus on a single (or maybe a few) platform versions to minimize development, testing, and marketing efforts.

In case of iOS and Android the users decide to upgrade or not. In case of web platforms often the developers can make this decision (i.e. what API version to use). I think we can learn from Apple how to let users upgrade as soon as possible: great marketing, shiny new features, and some gentle reminders. Web platforms should be single-instance if possible, with versioned, backwards-compatible APIs for developers. Developers will have incentives to upgrade to new API and development tools versions if these new versions make their life easier, contain much-needed development features, or contain features that users ask for.

In my opinion, these are 7 ways an App Platform can really fuel the App Economy. What are your thoughts? Please share them!

Photo by cool3c

18 lessons learned during the 6 year roller-coaster ride called Mendix

Sat, 14 Apr 2012 20:29:00 +0100

Last week I gave a talk about the lessons learned during the exponential growth of our company the last six years. I was invited to to talk at the Appsterdam meetup in Delft about "growing pains". In other words: what lessons did we learn when our company grew from 3 to 100 employees. I enjoyed the preparations as much as the actual talk, as it forced me to step back and think about the things we did and did not last years.

I covered lessons learned in four categories: company, culture, product development, and product evolution. Attempting to cover all these subject automatically means that you stay at a high level. However, I think I was able to share some valuable experiences from the trenches. It will be interesting to dive deeper in the separate categories in the future!

The slides:

App Delivery Platform-as-a-Service - How we revolutionized the app development market

View more presentations from Johan den Haan.

Do you recognize these lessons? What are your most important lessons learned? If you have questions, don't hesitate to ask them in the comments!

Will App Delivery PaaS make cloud more relevant to the business?

Fri, 16 Mar 2012 18:35:00 +0100

I finished my last blog post by introducing a Platform-as-a-Service subcategory called "Application Delivery Platform-as-a-Service" as a way to distinguish platforms that focus on improving the entire application delivery lifecycle (and not just application development or deployment). I would like to clarify my views on Application Delivery and PaaS a bit more. My first attempt has been published on InfoQ yesterday.

The short summary: business agility is key, so focus on improving the app delivery lifecycle for maximum business value and cloud will fit in naturally (as well as agile software development, Model-Driven Development, and social collaboration).

The Need to Focus on App Delivery Lifecycle in PaaS

Cloud is the next best thing since sliced bread, isn't it? It increases responsiveness, gives a business agility, and it enables new business models. Well, it seems not. Nearly three-quarters (74 per cent) of UK IT managers think that cloud computing has no relevance to their business. Is it fear? Are security and privacy concerns the main barriers for cloud adoption as stems from most related surveys? Or should we ask the CEO instead of the CIO?

Apparently there is a mismatch between expectations (or promises) around cloud and how a large group of people perceive it. My conclusion? We should just stop talking about cloud. It doesn't help your business, cloud is irrelevant! Let's explore why.

Read the full article

Every platform will evolve into a Platform-as-a-Service (and the resulting subcategory explosion)

Thu, 01 Mar 2012 09:44:00 +0100

Platform-as-a-Service (PaaS)... I guess you heard this term quite a bit last months. I wrote about PaaS earlier when arguing that Model-Driven Engineering is essential for the success of a PaaS and when I announced the new major release of our platform.

If you look at the industry perception of PaaS the main definition is something like: it is a public cloud platform where you put code in and get a service out. This is such a broad perception that it is no surprise that new subcategories are emerging every day. I will contribute to this movement by introducing a new one too...

PaaS is a fuzzy term, some subcategories

Let's first look at some example PaaS subcategories that have been introduced lately. Gartner, for example, sees a main dinstinction between Application Platform-as-a-Service (aPaaS) and Integration Platform-as-a-Service (iPaaS). I think you can imagine what the focus of each of them is. Gartner defines aPaaS as follows:

"Highly productive, easy to learn and use development environment that delivers business applications that are customizable, changeable, capable of implementing serious business functionality and, when deployed, offered with massive scalability, high-end enterprise-class (and eyond) performance and reliability, supporting massive amounts of data, all at SMB prices."

One could argue that to deliver enterprise-class applications implementing serious business functionality implies middleware and integration capabilities, but I can see the difference between delivering an application server as-a-Service and an ESB as-a-Service.

Other, more domain-specific, species I stumbled upon recently are pPaaS and bpmPaaS. A Portal Platform-as-a-Service and a Business Process Management Platform-as-a-Service.

I think we can agree on the fact that every type of platform can have a cloud twin and thus result in a PaaS subcategory...

Let's forget the public cloud as the main requirement for a PaaS

However, is being a public cloud platform the main differentiator between a "platform" and a "Platform-as-a-Service"? In my opinion PaaS is all about abstracting (or automating) away from underlying technology layers. You can just start using the services of the platform without bothering about the configuration and maintenance of the underlying infrastructure, operating system, middleware, and runtimes.

It all started with public cloud providers delivering such platforms as a service, but that doesn't mean it is limited to such providers. In my opinion it doesn't matter where the infrastructure is. PaaS can be very usefull too if it runs on-premise or maybe in a hybrid infrastructure approach. Just how I think the open source Infrastructure-as-a-Service software project OpenStack will more and more be used in the near future to build private and on-premise cloud infrastructures.

Why every platform will evolve into a PaaS

I think PaaS is another manifestation of what we always strive for in computer science: a clean separation of concerns. Let the platform builders bother about how their platform plays well with the infrastructure, OS, and middleware. Let the users of the platform focus on delivering solutions on top of the platform. There should be a clear contract between de platform and the software running on it, which specifies the language a user can use to define the solution. As IaaS is gaining ground in private datacenters near you, I strongly believe that existing platforms will evolve into a PaaS too.

Introducing a new subcategory

I just don't want to disappoint you by stopping here. I think there are some important things missing in the current industry views on PaaS. If we want to abstract away from the lower level details and move our focus to things like application development we need to go futher than just the development level. What a business needs are apps that perfectly fit in the business. These apps need to have a short time-to-market and have to be flexible to evolve along with the business. This means the full lifecycle of such applications, from the initial idea to first deployment, and from that first deployment to long-term evolution, needs to be as agile as possible.

I don't think we can reach this with traditional development approaches. Why not abstract away from low-level code in the same way as PaaS abstracts away from infrastructure by introducing visual modeling? And if our focus should be on apps that fit the business, we need more focus on collaboration with the business. Hence, we do not only need a development and deployment platform. We also need a social collaboration platform.

Ideally a Platform-as-a-Service should support the full application lifecycle. It's main goal shouldn't be to support development, it's main goal should be to deliver applications that fit the business. From capturing requirements to developing and deploying apps, as well as gathering end-user feedback.

Hence, my PaaS subcategory input: Application Delivery Platform-as-a-Service.

Photo by Robby Virus.

Blog overview 2011

Mon, 09 Jan 2012 16:56:00 +0100

In good tradition hereby the blog overview of last year. A bit late this time, but for good reasons (see the overview of the month december).

This year I expect to write a bit more. The subjects will stay the same, and as you could sense from the posts of last year my focus will be on all phases of the application lifecycle. So, Model Driven Development will still get attention, but also cloud and Platform-as-a-Service, as well as requirements capturing, social, and agile. As my first (and only) post with video material was the second popular post last year, I will try to post more video content this year.

Top 10 search terms

These are the top 10 search terms people have use when visiting my blog via a search engine:

model driven development
model driven engineering
enterprise architect
architecture vs engineering
domain specific language
soa framework
model driven software development
service component architecture
architecture definition
service identification

Blog summary for 2011

January

Interview on Pattern Based Engineering: Lee Ackerman, asked me a couple of questions about the use of patterns at Mendix and the relation with Model-Driven Development.
Why there is no future for Model Driven Development: this post was based on a talk I did in Nantes (France) and includes the slides and a full video of my talk. The main message is that we need to focus on all phases in the application lifecycle, not just development.

February

The quest for Business Agility: Emergent Architectures: a long article which clearly describes my views on designing, modeling, and building an agile enterprise with help of emergent architectures.

March

Code Generation 2011: just an announcement of the fishbowl session I organized at the Code Generation conference.

April

Why aren't we all doing Model Driven Development yet? An analysis of the main concerns which prevent people to use MDD. Initiated an interesting discussion.

May

Language Workbench Competition 2011: A lot of new initiatives in the field of language workbenches are arising to facilitate the creation of Domain Specific Languages (DSLs) and their accompanied code generators and IDEs. They all have their own strengths and weaknesses, so a competition to learn about them sound like a great idea back in 2010... and it was! This article describes all tools presented at LWC2011 as well as the opinions of the attendees. A nice overview of DSL tool space!

June

Making Model-Driven Software Development live up to its promises: the "sound bites" of the fishbowl discussion I hosted at Code Generation 2011. It was a good discussion on the future of MDD in four sections: marketing, alignment with programming, focus on problem domain, and education.

July

The Platform-as-a-Service for the Agile Enterprise: I shared a bit more about my work this time by describing the new major release of our products. This article perfectly fits in the subjects I normally cover like Model-Driven Development, app development, agile, and cloud.

August

Managing your application landscape: engineered or emergent: this article was triggered by the Application Landscape Report of Capgemini and defines two different approaches to manage an application landscape.

September, October

Well... moving to a new house doesn't leave you with a lot of time...

November

Enterprise Agile: 3 characteristics: there is a difference between an agile team in a start-up (or enterprise!), working in some sort of vacuum, and an agile team operating within the broader context of an enterprise. This article looks at some characteristics to make the difference more clear.

December

My first... :)

Blog overviews of previous years:

The Enterprise Architect

Designing the next programming language? Understand how people learn!

Language fundamentals

How do we learn a new programming language?

On concrete syntax: how does it convey meaning?

On abstract syntax: being declarative is not the holy grail

On semantics: live programming helps to understand cause and effect

Conclusion

A tale of a 7 year journey in developing software for the enterprise

Start up: what to build?

The first customers: we need this feature, it blocks this important deal

Stabilizing: not all customers want to "live on the edge"

The 2nd system effect: let's do it all over again, but better (we thought)

Ramping up: introducing "departments"

Knowledge transfer: how to keep other departments and users up-to-date?

The ivory tower: how to keep the feedback loop connected?

Entering the real enterprise world: when you thought functional requirements are the only things that interest customers

Scaling the team: how to keep the "startup" vibe?

Moving on

3 steps to free your business from the IT stranglehold (and vice versa)

Step 1: start to believe in an agile enterprise

Step 2: add a ‘new productivity platform' to your enterprise architecture

Step 3: know what's agile and what's not

Conclusion

The Evolution of PaaS in the Enterprise

Public vs. Private PaaS

PaaS and productivity

PaaS and the application lifecycle

Real-time updates and modularity

Layers

Is there room for language specific PaaS?

PaaS adoption in Europe vs. North America

Enterprise Agile: extending the agile process outside development

Why "Enterprise" Agile?

What's so different?

Isn't "Enterprise Agile" moving away from the core principles of Agile?

What does Enterprise Agile mean for development?

What does Enterprise Agile mean for marketing?

What does Enterprise Agile mean for services?

How to start?

Share your experiences!

7 ways a Platform can fuel the App Economy

1. A platform should make the delivery of Apps easy

2. A platform should attract and bind users with a great user experience

3. A platform should contain end-user value itself to create a base community and let developers build on top of that

4. A platform should provide great APIs to foster re-use of multi-tenant platform services and make development easier

5. A platform should attract developers with great tools

6. A platform should support and ease the entire application lifecycle to ensure quality and agility of Apps

7. To stay successful a platform should ensure that users and developers upgrade to the latest version

18 lessons learned during the 6 year roller-coaster ride called Mendix

Will App Delivery PaaS make cloud more relevant to the business?

The Need to Focus on App Delivery Lifecycle in PaaS

Every platform will evolve into a Platform-as-a-Service (and the resulting subcategory explosion)

PaaS is a fuzzy term, some subcategories

Let's forget the public cloud as the main requirement for a PaaS

Why every platform will evolve into a PaaS

Introducing a new subcategory

Blog overview 2011

Top 10 posts

Top 10 search terms

Blog summary for 2011