Rinat Abdullin

Lokad Shared Libraries 1.5 and Autofac for Silverlight 3.0

Rinat Abdullin — Fri, 17 Jul 2009 13:16:58 +0000

Just a few announcements.

Lokad Shared Libraries 1.5 Release

There have not been any major changes since the last official release. Some highlights include:

Various tweaks and fixes coming from projects actively using Lokad Shared in production
MockContainer implementation to simplify testing in component-driven development scenarios such as MVC (described here and here)
WaitFor construct for executing fragile code with timeout options.

Download options include:

Merged libraries as they are used in projects of Lokad.com.
Separate assemblies for people that want to use only some bits.

Autofac 1.4 for Silverlight 3.0

Autofac IoC Container for .NET is now checked for compatibility with Silverlight 3.0 in a continuous integration manner. Version 1.4 compiled for Silverlight 3.0 runtime is available within the download page.

People interested in checking out how the same code-base could be used to target various .NET runtime versions are welcome to check out the latest source code for the container.

Deployment and Updates of Desktop Applications: WiX, ClickOnce and Other Scenarios

Rinat Abdullin — Wed, 08 Jul 2009 11:28:22 +0000

As we all know, web applications became quite popular these days. One of the reasons is that they generally do not need any installations in order to be used. Upgrading them is simple, too. When you got new version to roll-out – just update web server and everything below. Users will get new UI automatically. At most you might want to ask them about refreshing their cache.

This simplified deployment routine adds the benefit of reduced development friction, allowing developers to have shorter development cycles in an agile manner. This eventually allows having better feedback loops leading to better products (ceteris paribus).

Desktop applications do not have such flexibility and simplicity. Almost always a development team has to deal with the packaging, deployment and updates in a manual or semi-automated way. This increases development friction.

Let’s take a look at some deployment scenarios in small and medium development projects.

Scenario 1. Project deliverables are produced by running the build, then manually putting its output into a zip file and sending it to the customer. That’s the simplest option one could get. The approach fits scenarios where some non-critical tool with a short life-cycle (i.e.: under a month) is being developed.

This scenario has a hidden cost of high development friction and also high probability of a human error. Although this sounds like something non-important (what could be more complex than zipping a few files, right?), this consistently slows down development and eventually leads to some reputation-hurting mess-up.

Scenario 2. Project deliverables are produced by starting some special build on a continuous integration server. After running a common integration routine, it grabs build outputs, applies certain configuration settings, packages into a nice zip and copies to a well-known distribution location. Usually that’s a product/version folder on a web server (it optionally could be protected by some authentication and exposed to the Internet).

Optionally the build might update common download.html file pointing to the download locations. This way the cost of releasing the product (on developers’ side) is about two mouse clicks and sending notification to customers. Customers still have to download, unzip and update manually.

This scenario works well in situations where the project deliverables are consumed by tech people. For example, Lokad Shared libraries are distributed this way. You can check out the build script for that, if interested.

Scenario 3. This scenario is a slight variation of the previous one and targets non-geeky customers. To make customer experience friendlier we might produce installable package instead of the raw zip archives (deployment process stays similar). So customers will simply have to download the installer and walk through some wizard in order to get the thing running. Additionally, the actual desktop application might have simple update framework capable of checking for the latest deployed version and helping user to download and install it as needed.

Let’s see how this could be implemented in .NET world. There actually are at least two major options.

Option 3.1. First option would work well for simple deployment scenarios. It is called ClickOnce. Basically, this is a Microsoft technology providing simple deploy and update experience. It is tightly integrated into .NET and Visual Studio and hides some complex decisions from the developer.

Advantage of this technology is its simplicity and ability to get started in minutes. Disadvantage is the same one – simplicity and inability to handle more complex scenarios. This makes this technology suitable and efficient only within a specific set of scenarios.

Here’s how the deployment process might work:

Integration server, in addition to the usual routine, runs ClickOnce publishing tasks exposed by .NET SDK.
These tasks produce deployment files that could be exposed to the outside world via the web server.
End-user only knows about some html URL that allows him to either run executable bootstrapper or launch the actual ClickOnce application. Bootstrapper is a Win32 application capable of automatic download and deployment of all the required prerequisites (i.e.: .NET Client Profile).
User clicks Next a few times, looks at some progress bars (giving him a sense of accomplishment) and then is presented with actual running application.

Another nice feature of ClickOnce is about the way it handles updates. You can configure application deployment in such a manner that at every launch will check deployment server for the new updates (automatically downloading and installing them if needed). Or this could happen once in a week or so.

This simplifies the actual deployment process. If you have customer with an issue and a deployment server, then issue fixing is just a matter of:

Writing unit test.
Fixing the issue.
Committing it to version control repsitory.
Telling Integration server to perform the deployment after the build.
Asking customer to restart the application.

Auto-magic will take care of the rest.

Option 3.2. Sometimes ClickOnce is too limiting for the project deployment purposes. Some limitations might show up when you need to:

share components, folders and files between multiple products;
configure application to start up with Windows or perform some other complex tasks;
deploy installations with multiple optional features or do anything that requires user’s input;
handling upgrades gracefully;
deploy applications to a large number of machines at once.

In order to handle these tasks we need to give up on the simplicity of ClickOnce and switch to something more powerful. My favorite option at the moment is Windows Installer XML 3.0 by Microsoft (or WiX, for short).

WiX allows creating MSI files by writing some XML-related code, and provides some tools and even Visual Studio integration to make developer’s life easier. Yet, the technology still has a decent learning barrier. Other disadvantages are:

Programming in XML (even writing complex configurations) is inefficient and error prone by default.
You can’t unit test your installation (ICE evaluators can offer some help but they fail to run on the integration server sometimes).

Yet at the moment that’s my favorite technology for handling complex scenarios that ClickOnce can’t handle.

If you are interested in some samples, I invite you to have a look at some projects by Lokad. Seeing WiX in action is as simple as:

Make sure that WiX 3.0 is installed on your machine
Checkout Call Center Calculator or Safety Stock Calculator (via version control).
Execute go /t:CreateInstall /p:Version=1.2.3.4 in the root project folder

Sources are made public under the new BSD License.

Here are some additional links on WiX that I found useful while learning the technology:

Windows Installer Components Introduction
WiX Wiki.
WiX tricks and best practices on StackOverflow.
Deploying additional components with GenerateBootstrapper task.
How do I start an application after install?
How do you create an MSI to deploy a VSTO 3.0 Add-in?
System Folder Properties at MSDN
From MSI to WiX article series.

Deployment and auto-update process on top of WiX might work like this:

Integration server, when asked to publish release, does the following:
- It creates a new MSI installer by building the project and then running WiX-related MSBuild tasks. This MSI uses major upgrades to deliver new functionality and update files on user’s computer.
- Updates the download page by redirecting links to the latest package
- Updates custom deployment manifest, which could be as simple as ProductName.Release.txt file containing latest version number
- Copies latest released MSI to ProductName.Release.msi in a well-known location
When user checks for the updates (or software does it for him) following takes place:
- A simple web request to the version file somewhere over the network will tell if we are up-to-date
- If we are not – just inform the user about that and offer him to perform the upgrade
- When user accepts, close the application and launch ClickOnce bootstrapper bundled into the MSI. Bootstrapper does not actually care about the ClickOnce, it merely checks for the prerequisites, then downloads and installs the MSI it knows about. In our situation this MSI is ProductName.Release.msi.
- Launched MSI is implemented as a major upgrade, so it will simply offer user to click Next a few times (firing off UAC if operating system supports it), perform the upgrade and then re-launch the application

The process of deployment and auto-updates might feel slightly complicated, yet it involves very little coding (except for the MSI creation). The most interesting part is that it has to be done only once. After that fixing issue for the customers is just a matter of:

Writing unit test
Fixing the issue
Committing changes to version control repository
Telling Integration server to perform the deployment after the build.
Asking customer to click the "Check for updates" button in his application.

This brings us closer to the Agile development methodologies in the development of desktop applications with complex deployment scenarios.

Related links:

Testing MVC Elements and Interactions with Mock Container

Rinat Abdullin — Wed, 24 Jun 2009 21:38:38 +0000

Lokad Shared Libraries project includes implementation of the MockContainer that significantly simplifies testing of Model-View-Controller elements in separation. It does this by automatically generating and injecting mock objects, should the tested class ask for them in its constructor.

These mock objects are persisted in the local container instance as singleton, so that additional expectations could be recorded against them. Expectations are verified when the container is disposed.

There is a little overhead of creating and disposing container and instances every test, since we are using Autofac IoC Container internally here, which has a really good performance.

Interface and internals of this MockContainer have been polished heavily since its introduction a few months ago. This (with the help from a base fixture class providing a nice syntax) has allowed to write tests like this one:

[Test]
public void Display()
{
  View.SetTitle("Some title");

  // we expect the view to ask IViewspace to display itself
  WhenViewIsShown
     // you always set Form.CancelButton  for user's pleasure
    .CancelButtonShouldBeSet()
     // assert that title as as expected
    .TitleShouldBe("Some title");

    // create 0 to 20 random models and display them in view
    View.BindModel(RandClientModels.NextEvents(0, 20));
}

This is an actual unit test against actual Windows.Forms.UserControl component. It does what it says. Ability to write unit tests like this brings following benefits for free:

In this unit test all calls are executed against the interface. So should we need add Silverlight or WPF view implementations to our library, we would already have tests for the first step of Test Driven Development.
Ability to write simple unit tests really pays off in scenarios, where a single component is reused in multiple projects. Finding a bug in one of these projects, expressing it in a unit test and fixing, immediately makes all projects downstream more stable.
Using randomly-generated Models allows to cover a wider range of the possible failure scenarios. Should some case go wrong, we'll be able to reproduce it by checking out rand seed used in the integration run (Nondeterministic reproducible testing will be the subject of another article).

Here's another type of unit test. It could be invoked explicitly just to see how our view looks alone (without launching an entire application and clicking our way to this View):

[Test, Explicit]
public void Show_view_for_debugging()
{
  // when view asks to show itself, actually display it
  WhenViewIsShown.ShowAsDialog();
  // load random values and display
  View.BindModel(RandClientModels.NextEvent());
  // display view, while using ValidEvent rule 
  // to validate all input
  View.GetModel(EditEventController.ValidEvent);
}

ExplicitAttribute in NUnit tells us that unit test is excluded from the usual unit testing run and has to be launched manually.

This kind of debugging unit test for the reusable View components speeds up all the testing and UI tweaking process. Do you need to check out how View validates data, scales or displays that recent UI tweak? This is just one click away:

Sometimes we may want to go deeper and check out how various components integrate together. This is not a normal unit test (in the sense of Unit Testing), so it is marked with ExplicitAttribute as well.

[Test, Explicit]
public void Test_for_debugging_with_controller()
{
  // view is registered by test fixture
  WhenViewIsShown.ShowAsDialog();

  // simplified syntax for registering additional components
  // that we don't want to be replaced with mock objects.
  Container.Build(builder =>
  {
    builder.Register<ConnectionEditController>();
    builder.Register<WorkScopeView>().As<IWorkScopeView>();
  });

  // resolving controller and launching it with a random connection
  var controller = Container.Resolve<ConnectionEditController>();
  controller.Startup();
  controller.Edit(RandApiModels.NextServiceConnection());
}

Although code like this does not participate in the Continuous Integration process, having it around speeds up the development and simplifies debugging, should something go wrong.

This specific test creates UI elements like this:

Unit tests samples come from open source project Lokad SDK that allows developers to introduce forecasting capabilities of Lokad.com into their applications.

Cloud Bursting Scenarios For Small Companies

Rinat Abdullin — Sat, 20 Jun 2009 09:50:01 +0000

Almost every software developer dreams about starting up his own MicroISV company that would keep bringing him money even during the sleep. Let’s see how cloud computing can help us out here by making it easier to succeed.

This article continues the Cloud Computing series.

These days it became so easy to start a one-man company in the IT industry, like it had never been before. A lot of people are doing that, while the MicroISV and Micropreneur are becoming rather hot topics on self-development blogs.

One of the reasons for this is that you no longer need to have a physical hardware or other resources in order to be able to provide software and services. Almost everything could be virtualized and hosting could be simply rented. It's quite flexible and convenient, isn't it?

As turns out, Cloud Computing can add even more flexibility and market maturity to small companies at a fraction of the cost. That’s because one of the major driving forces behind the Cloud Computing hype is the actual ability to buy hardware capacities and resources on demand without paying for them upfront.

Let’s have a look at first business scenario of pushing CPU-intensive computing into the cloud and the technological availability of implementing it today.

Note: all figures, names and numbers are purely imaginative. They are used just to show you the logic behind the cloud bursting and scale of savings it can bring.

Let’s imagine a small and young one-man company providing some software as a service. For example IdealDotNET that specializes in providing custom code quality reports and optimization recommendations for .NET development projects, based on some proprietary algorithms. These algorithms are running against assemblies being uploaded by the customers. First 10 assemblies are analyzed for free.

We’ll put aside financial feasibility of this endeavor and concentrate on the technological implementation side. It could be as simple as this:

As we all know, running complex introspection rules against any assembly is a rather CPU-intensive process, especially if these rules include some extensive checks. So this system will be able to handle only a limited amount of users per hour. Let’s assume that this setup can process 30 .NET assemblies per hour on a deployment that costs 100 EUR per month (entire system is located on one semi-dedicated server).

As long as the number of uploads does not exceed 30 per hour, this setup will provide solution that just works for its money. Here’s how the usage statistics might look like:

Everything will go just fine till IdealDotNET starts some marketing campaigns or simply gets mentioned by Scott Hanselman in his blog (he once twitted about the power to take down a small site by linking to it).

We’ll get into a nasty situation, when this happens. Number of upload attempts per hour will exceed our capacities:

If the system had been designed to handle such a situation, then it would not crash down everything. Instead, all excessive visitors would be simply turned away by saying something like "We are sorry, but upload limit has been reached, please come later".

This approach means lost profit, because some of these visitors had a chance of liking the service and becoming customers. More than that, some of the existing customers may hit the limit, get discouraged by it and turn away to the competitors.

What could we do?

One option is to have more servers from the start. This way IdealDotNet will have more capacity to handle usage spikes before they happen. Let’s say that the company got two more servers from the very start, boosting upload limit to 100 per hour. Then the Slashdot effect would look much better for us:

That’s an easy solution, yet it comes with two problems:

Additional servers come at a cost (for example, additional 200 EUR per month) and they will still sit idle most of the time. If IdealDotNET had existed for mere 2 months, then total cost (simplified) of handling such a spike is 400 EUR.
Buying exactly the right processing capacities the start is impossible. Even with the best estimate you’ll either have excessive capacities or will miss some spikes.

That’s where the technological opportunities of Cloud computing come to the rescue. With some tweaking infrastructure of IdealDotNET could be modified to use only one server most of the time, while gracefully scaling up to handle the spikes.

We’ll talk about the implementation details later. Let’s focus on the economical benefits first. With this configuration our processing limits start looking more interesting - they elastically adapt to handle the demand:

If IdealDotNET had used Amazon EC2 Medium CPU Virtual Machines to handle these spikes (their price is 0.20USD per hour each), then the total cost of handling this spike is around 40 EUR. That looks better than 400EUR, does not it? Plus we get the flexibility of elastically scaling to 1000 uploads per hour (or more) as needed.

This scenario is called Cloud Bursting or Bursting into the cloud.

Note: theoretically IdealDotNET could get a better deal from Mosso (their cloud machines start at 0.015 USD per hour). But it does not look like they have all the technological pieces in place for implementing cloud bursting scenario efficiently.

How hard is it to implement a system that can elastically handle as much stress as it is needed? Answer is: it is not extremely hard, if you change your way of thinking a little bit towards elastic capacities.

We start by adding a Manager component to our system (it could be a separate process or simply a loop in a background thread). It will continuously watch for the number of uploads pending to be processed. Should it become obvious that our processing capacities are not enough to keep up with the demand; it will issue requests to the Cloud Computing API to deploy new virtual machines (Amazon EC2 already has such REST-based API available, while Windows Azure claims to release it later).

Each virtual machine usually comes preconfigured with some worker role. This pre-configuration can be done by:

deploying an application or scripts to the cloud (if this cloud service is a Platform as a Service provider like Google App Engine or Windows Azure);
uploading preconfigured VM image (if the cloud is an Infrastructure as a Service provider like Amazon EC2).

When this worker boots up, it only needs the addresses of the task and result queues in order to start processing jobs. These parameters are usually passed as arguments along with the deployment calls.

In order to let the cloud workers access these queues in a secure way we will need to expose them via some encrypted service API, implemented in a flavor of communication framework of your choice.

When the Manager detects that we no longer need the some processing power from the cloud, it would gracefully shut down these workers, saving energy for the society and money for IdealDotNET.

Note: obviously, there are few more technological challenges that have to be dealt with. These might include: reliable cloud management, remote exception handling and monitoring. However these problems do not have any black holes in their domain space (at least from my perspective) and thus are almost guaranteed to resolve to a finite and defined work time which could be used in project management schedules. This work time could be further reduced by applying established frameworks and development principles.

Given all these advantages of the cloud computing why wouldn’t IdealDotNET move everything to the cloud? There are several reasons for that:

Cloud computing solutions are generally more expensive when it comes to long-term consumption (compared to existing offerings by hosters).
This would be a more complex solution to implement (from the delivery and maintenance standpoints).
This comes with a certain lockdown cost. Once you have your entire infrastructure in a single cloud, it is not that easy to move it to a different cloud (especially since we don’t have well-established cloud computing market, yet). It is safer to keep primary resources closer to you, while using the cloud only to handle the spikes.

In fact, in order to reduce the dependency on a single cloud provider, IdealDotNET company could use several cloud computing providers, picking whichever is better for the situation.

Does not this example of cloud bursting create an itch to use such a great resource to solve some business problems? There are more examples to follow in the Cloud Computing series.

You can subscribe to the updates, if you are interested.

Related links:

Software Development Body of Knowledge - ABC 02

Rinat Abdullin — Tue, 16 Jun 2009 14:14:53 +0000

In this Software Development ABC article we'll walk over the development terms that were quite a paradigm shift for me in the first years of my software-related career.

There will be a short intro, followed by the compact definitions of 3 terms with the links leading to full articles (and even more links).

The first paradigm shift has happened while I was coding forecasting algorithms for my diploma in economics (back in Delphi and Pascal these days). It was about Version Control Systems. They have allowed to stop worrying about manual backups or doing something wrong. Ability to instantly roll back to the previous working copy (or any other previous moment in history) gave an invaluable sense of confidence and flexibility. I also was finally able to get rid of incremental archives scattered across the hard drives (all these "copy2.zip" and "copy2version4.zip").

The second paradigm shift was about Unit Testing which is an extremely simple concept on its own - split the code in units and write assertions to rapidly verify correct behavior of these units. Unit testing has allowed to concentrate on the development by significantly reducing the risk of breaking something with a change (which is often the case with mathematical code). Green became my favorite color afterwards.

The third paradigm shift took place some time later, when I started working remotely in my first development project. It was .NET 1.1 with a lot of fragile and complex legacy code. Changes there rarely went without breaking something else.

Yet, the sales people didn't care about things like development friction and constantly demanded new features. They often went directly to the developers in order to get them. Planning and schedules were often sacrificed for the sake of moving forward with an important customer (which actually made sense in that project).

The only way to deliver reliable functionality in such a brittle and fast paced environment was about using some methods from the Agile Software Development. And, as it has turned out later, they did work and made development process more efficient and stable.

So let's get into these three terms in a more detail.

You can either continue reading through expanded definitions (which have links leading to even more material) or subscribe to receive updates on this topic.

Version Control System - VCS

Version Control System (VCS) is a software that allows to manage changes of documents, programs, images and other information that is stored in form of computer files. Changes are usually identified by an incrementing number or letter code also known as revision number or revision.

The simplest usage of versioning is - you can easily go back to the previous working version of your files, should you mess something up with the latest changes.

Changes could range from fixing a typo in a text file up to a huge refactoring in a software project, spanning hundreds of files. Each change usually has name of the person introduced it, time of the change and an optional description message.

Proceed to the full article on Version Control Systems

Unit Testing

Unit Testing in software development is a way to quickly verify that smallest blocks of software (units) behave as expected even as the software changes and evolves.

Here's how it works.

Any program could be logically separated into distinct units (in object-oriented programming the smallest unit usually being a class). Developers, while coding these program units, also create tests for them (code blocks containing some assertions and expectations about units). These tests could be used to rapidly verify behavior of the code being tested.

When some other developer introduces new units or changes something in existing units, he can run all the tests available for the program and verify that everything is still operating as expected. Usually running unit tests is a fast operation (less than 30 seconds), so developers are encouraged to do that often.

Proceed to the full article on Unit Testing

Agile Software Development

Agile Software Development is an iterative and incremental approach in software development, where requirements and solutions evolve through a tight collaboration of self-organizing cross-functional teams.

Agile works quite well in situations when some unique software project has to be delivered really fast in order to seize the market opportunity. This works because Agile software methodologies allow development teams to adapt quickly to the rapidly changing business requirements. This happens because of:

short development iterations allowing to start getting feedback early in the project's life-cycle and incorporate it into the product fast by delivering working software repeatedly and in a timely manner;
continuous integration helping to reduce development friction and improve quality of a rapidly developing project in self-organizing environment;
feature-driven development focusing development on delivering features (functionality) bringing real business value to the client (and the project);
lean software development laying out common principles for efficient software development process;
eXtreme Programming pushing lightweight software development methodologies to the extreme.

Proceed to the full article on Agile Software Development

Interested in the series? You can stay tuned by subscribing to the RSS feed of this Journal.

Software Development Body of Knowledge - ABC 01

Rinat Abdullin — Thu, 11 Jun 2009 19:40:14 +0000

There is a great discipline out there called Project Management. It is about planning, organizing, and managing resources to bring about the successful completion of specific project. Projects can range widely from writing a diploma to delivering an innovative automobile factory ready for mass-production.

Core principles of this discipline are captured in a book called Project Management Body of Knowledge which basically lays out a foundation for a successful communication between the project managers and other people involved in the project (stakeholders).

These principles could be applied to software development, too. In fact, they should be applied, since Project Management is the tool that allows to deliver really complex development projects on time, within the budget and quality constraints.

This becomes even more important these days, as the complexity of enterprise applications keeps on growing with the introduction of new concepts like Software as a Service, Cloud Computing.

However, before the Project Management approach could be applied consciously to the development domain, some steps ought to be taken. The first one starts with a dictionary of common terms from the domain of software development projects.

So let me introduce Software Development Body of Knowledge - ABC series to you. Each article will present definitions of 3 terms essential to the software development these days. These definitions will attempt to provide common high-level overview of the term's meaning without going too deep into the details.

As always, all feedback is extremely welcome. Please, feel free either to comment on this post, specific terms or to drop me a line directly.

Test-Driven Development - TDD

Test-Driven Development (TDD) is a software design method that requires developers to create automated unit tests that define code requirements, before writing code itself. These unit tests contain multiple assertions (returning true or false) that could be used to rapidly verify correct behavior of the code being tested. Unit Tests allow developers to evolve and refactor code in a more secure manner.

Writing a test before the actual code helps to achieve higher quality, because:

it forces developer to refine code contracts before working on the implementation;
it forces developer to make and keep code easily testable. This requires loose coupling and a separation of cross-cutting concerns;
presence of unit tests gives developers more psychological freedom to experiment with the code and refactor it; managers get lower risks related to introducing high-level changes to the project.

Proceed to the full article on Test-Driven Development

Continuous Integration

Continuous Integration (CI) in software development is a set of practices and principles that speed up delivery of software projects and increase quality of their deliverables.

Simply put, Continuous Integration takes a lot of mundane and boring things away from the process of software development, while letting developers to concentrate on more enjoyable and creative tasks.

As the name implies, Continuous Integration is about integrating changes to the development project continuously and automatically. These changes usually come in form of modifications and additions to the source code. When we keep them as small as possible and automatically run all checks available (starting from the compilation process and up to the full test routines), we can detect most breaking issues whenever they are introduced. Since the changes are small, it is much easier to isolate and fix the root cause.

Proceed to the full article on Continuous Integration

Inversion of Control - IoC

Inversion of Control (IoC) is an approach in software development that favors removing sealed dependencies between classes in order to make code more simple and flexible. We push control of dependency creation outside of the class, while making this dependency explicit.

Usage of Inversion of Control generally allows to create applications that are more flexible, unit-testable, simple and maintainable in the long run.

Proceed to the full article on Inversion of Control

Interested in the series? You can stay tuned by subscribing to the RSS feed of this Journal.

Developing Component-Driven Applications powered by IoC Containers

Rinat Abdullin — Wed, 03 Jun 2009 19:53:30 +0000

This article continues talk on the topic of Component-Driven Development with IoC Containers. We'll expand on the topic of the actual development process of new applications and migrating existing legacy applications.

Developing New Applications

In the ideal-world situation with established component libraries for the domain and an IoC ready to provide infrastructure, delivering an application becomes a pleasurable experience. Yet, this is rarely the case when a new project starts.

In reality we neither have the actual components nor the project-specific knowledge required for rolling out the stable component library right in the first phase. Experience and shared libraries from similar projects may provide a good start, but they still will not be able to get you all the way to the delivery.

This is somewhat similar to developing a syntax for some new language - it is impossible to foresee and model all key words and syntax elements for the specific domain without a plethora of actual use cases and some iterations.

So the only working way is about evolution, where design comes first and application evolves from top to the down (as we learn about efficient ways to reflect domain knowledge into the development realm of reusable components).

Let's walk over the simplified development process.

First, as it usually happens in any project being initiated properly, we define what business functionality has to be delivered by the application (and then refine the definition a few times with all the major project stakeholders).

Second, we implement this functionality by developing against some high-level service contracts, which are defined in the very process. A few short prototyping sprints might be needed here before we get a satisfactory design to proceed with. Deliverables of this step are interface specifications for the components within the service layer (prototypes are discarded).

Side note. Best results on this stage are obtained, when a single architect is capable of reimplementing the entire application in his mind, while evolving it to the best design within multiple iterations. Quality of the deliverable depends on the detalization level of the design, that a person can hold in his mind, while working it through and evolving. In this situation numerous real-world iterations (drawing away valuable and scarce development resources) could be saved at the cost of some head ache.

Third, we create mock implementations for these services, hiding them inside the IoC Container.

At this step we already have got a real application (although completely based on mocked services) that could be used for the demo and presentation purposes. This allows us to start receiving feedback from the involved people. At this point we can switch from the pure prototyping and project management approach to more flexible iterative development. Continuous integration and automated deployment services will have to be established within the development environment in order to achieve this, while reducing development friction.

Then, we implement actual service layer below high-level service contracts, gradually replacing mock classes with the actual implementations. These actual implementations may (and probably will) require help from some other service contracts from lower levels (which would be implemented right away, mocked or implemented in a simple form).

It is really common to discover some unforeseen interactions, threats or opportunities in the process, that will require for the top-level service contracts to evolve along with their implementations. This brings some sort of change shock, that originates at some point of the application logic and affects related components, requiring their redesign (or even change in service contracts).

However, since components handle fine-grained responsibilities, while being loosely coupled by the IoC , such a shock usually does not go far in the infrastructure, usually being absorbed at the service contract boundaries. That's what makes the difference between "Oh, we've got to rewrite 50% of the code in order to implement this feature" and "Simple, we've just got to isolate this concern and swap the implementation".

We work this way all the way to the bottom of the application, gradually stabilizing higher levels and evolving lower levels. Cross cutting-concerns are injected into the architecture along the process.

In the end we might end up with an application that works and is flexible enough for the planned changes. Yet, this may not be the end of the evolution. Good components and even infrastructure bits tend to be reused in other similar applications (just because it is so easy to do that). This brings new constraints and requirements, creating new change shocks that require further evolution of the domain components.

Good separation of concerns between components and logical component layers, their responsible reuse (sometimes it is better not to reuse in order to avoid unnecessary complexity) and a lot of planning can help to absorb these shocks, while letting the application to grow further, efficiently adapting to further changes in business requirements.

Migrating Legacy Applications

Unfortunately, greenfield scenario is not the most common in the world. More often we encounter situations, when existing legacy application can't evolve any more due to being strangled with manual wiring and plumbing code (let alone the complexity of the hard-coded interactions). Yet, these legacy applications happen to have a lot of business value and must be adapted to new business requirements.

One of the options of loosing up the tension is to gradually introduce IoC and component infrastructure. Since this is not a complete redesign, application has to stay more-or-less production ready most of the time (although short destabilization spikes are allowed, as long as they are known and handled). So we can't mess with the top logical layers that represent business functionality (we would not be able due to the complexity anyway) and we'll have to evolve it from bottom to the top.

At this point we are considering that continuous integration is established to enforce consistently high quality of the new changes being introduced, while providing us with some protection against quality regressions that are likely to happen in the legacy codebase.

We might start evolution in the lowest logical section of the application - in repository and DAL layers. We do that by abstracting the functionality behind the service contracts that are implemented by the components.

At this point of time we do not have any IoC infrastructure in our application yet. So for the evolution process it is allowed to introduce injection points via the static service locators. However, as the evolution process goes, these static service locators would move higher and higher in the application logic till they either hit infrastructure limitations (i.e. resolving components in the web services) or get finally settled down in the places they belong to (application hosts).

Temporary static service locator might look like this in .NET (implementation for the Autofac IoC Container):

public static class GlobalSetup
{
  public static readonly IResolver Resolver;

  // this is fired when container is accessed first time
  static GlobalSetup()
  {
    var builder = new ContainerBuilder();
    builder.RegisterModule(new ConfigurationSettingsReader());
    var container = builder.Build();

    // create IoC-decoupled resolver
    Resolver = new Resolver(
      t => container.Resolve(t),
      (t, s) => container.Resolve(s));
  }
}

Where Resolver merely implements following IResolver interface via the lambda parameters:

public interface IResolver
{
  // Methods
  TService Get<TService>();
  TService Get<TService>(string name);
}

You could look up its implementation in the Lokad Shared Libraries, but its really trivial.

When legacy code uses components directly, we could reference their service contracts like:

var repository = GlobalSetup.Resolver
  .Get<ICustomerRepository>();

When we don't control the resolution point (i.e.: this happens with .NET-instantiated classes like web services or object data sources), we could use simple proxy classes. These classes bring control of the component instantiation back into our hands and could probably be created with your favorite refactoring tool in less than in a minute.

Sample IoC proxy class for ICustomerRepository might look like:

public sealed class CustomerProxy : ICustomerRepository
{
  private readonly ICustomerRepository _repository;

  public CustomerProxy()
  {
    _repository = GlobalSetup.Resolver
      .Get<ICustomerRepository>();
  }


  public Guid CreateCustomer(Customer customer)
  {
    return _repository.CreateCustomer(customer);
  }

  public Customer GetCustomer(Guid customerID)
  {
    return _repository.GetCustomer(customerID);
  }
}

In order to quickly create such a component proxy with Visual Studio 2008 and R# we merely have to code the class declaration with the constructor:

public sealed class CustomerProxy : ICustomerRepository
{
  private readonly ICustomerRepository _repository;

  public CustomerProxy()
  {
    _repository = GlobalSetup.Resolver
      .Get<ICustomerRepository>();
  }

  // place cursor here
}

Then place cursor below constructor and call for R# autogeneration popup menu (Alt+Ins for me). There we select "Generate | Delegating Members", hit [Space] to select everything and then [Enter] to generate. That's it.

The technique also works with the situations, where proxy has to implement multiple interfaces at once (i.e.: web service aggregating a few service contracts).

There are a few more tips and tricks on developing components in for IoC infrastructure and enforcing common design guidelines upon them. So there might eventually be a third article in the series some time later.

Meanwhile you can:

Subscribe to the journal RSS feed to stay tuned for the updates.
Share you thoughts about this article either in comments or privately.
Do you have a real world experience with component-driven development in complex projects? I'd love to hear about that!

Component-Driven Development and IoC Containers

Rinat Abdullin — Sun, 31 May 2009 19:42:07 +0000

Nicholas Blumhardt has recently made an appearance on .NET Rocks (that's the podcast I'm listening on a regular basis). The talk was extremely interesting - starting from Autofac and going all the way to the component driven development philosophy.

Let's try to reiterate and summarize some ideas about the Component Driven Development, IoC containers and evolving enterprise applications in general.

What is a component-driven development?

We could try to describe component-driven development as an approach in which code is fragmented into reusable and testable components that are combined together to form application foundation for delivering business functionality. The combination and management of components is generally delegated to Inversion of Control Container.

A component itself is a class that implements some service contract and explicitly defines the dependencies that it needs in order to fulfill this contract. Actual implementation is hidden from everybody else outside the component.

An example of component would be:

[Immutable]
public sealed class CustomerRepository : ICustomerRepository
{
  private readonly IDalSettings _dal;

  public CustomerRepository(IDalSettings dal)
  {
    _dal = dal;
  }

  public Guid CreateCustomer(Customer customer)
  {
    return _dal
      .CreateCommand(Queries.CreateCustomer)
      .With(() => customer.Name)
      .With(() => customer.CompanyUrl)
      .ExecuteAndReturn<Guid>();
  }

  Customer ICustomerRepository.GetCustomer(Guid customerID)
  {
    return _dal
      .CreateCommand(Queries.GetCustomer)
      .With(() => customerID)
      .ExecuteAndReturn(Queries.CustomerSelector);
  }
}

As you can see, component explicitly declares its dependency on the IDalSettings and implements following service contract:

public interface ICustomerRepository
{
  Guid CreateCustomer(Customer customer);
  Customer GetCustomer(Guid customerID);
}

Components tend to be rather concrete and stick to the single responsibility principle, where each class is dealing with a single and a rather specific task.

I personally perceive components as low-level building blocks that are just one level higher than the language elements and elements from the Base Class Library of .NET. Or you could also look at the components as elements from the Domain Specific Class Library, that is unique to your specific domain.

Side note. In reality, the boundary between these BCL and DSCLs is blurred a bit. This allows, for example, to move certain components from domain libraries (provided, these building blocks are abstracted and reusable enough) into libraries shared between multiple projects. Most of the time these components are about cross-cutting concerns of an application. An example of such a library would be Lokad.Shared library.

These components are used to create a foundation for delivering an actual enterprise application. This foundation is established by wiring and configuring components together through an Inversion of Control (IoC) technology. Inversion of Control Containers generally take care of various things, relieving us from the need to do them by hand. Examples of these things are:

injecting cross-cutting concerns into the application;
managing life cycles of components and ensuring their deterministic disposal;
handling dependency resolutions, wiring and configuration.

This forms the application infrastructure (almost an Infrastructure of Components) that could be used to deliver a variety of actual applications in the same domain.

No matter, how we organize our components, there are a few common rules:

Every component is registered in this infrastructure by a service contract, while the implementation specifics stay hidden from the higher logical levels and even neighboring components.
Components declare their dependencies explicitly in the constructor (it is advised to avoid property injection, as it tends to complicate the interactions).
Generally in .NET (although not always) components are sealed classes that do not inherit from any other base class, while implementing an interface or two. We are favoring composition here.
Most of the time, components are immutable and are merely an object-oriented representation of a function, which receives (and memorizes) its parameters via the constructor.
Additionally, it usually is a good practice to implement interfaces explicitly for IoC in .NET. This will reduce the temptation to use components directly, instead of referring to their service contracts (aside from the fact that explicitly implemented classes do not trigger "Missing XML documentation" warnings).

Logical encapsulation and reliance on the service contracts significantly simplifies code reuse and unit testing, thus promoting good development practices. It also increases flexibility, since you could rather easily substitute one component for another, as long as they both abide to the same contract.

Component Layers

Sometimes building blocks are logically organized into the layers in order to make application feel more simple and decoupled. This makes it more manageable and flexible in the long run.

Components in one layer tend to share some common features for the services they are providing. Also they tend to:

be located in the same project and even namespace;
have similar dependency requirements;
use similar resources and libraries;
be grouped within the same container if nested container hierarchy is used.

Let's have a look at a sample picture below:

For example, DAL Layer is about basic components that provide raw access to the database, abstracting away configuration settings. They are consumed by other components implementing certain service contracts about persisting, querying and manipulating objects (that's the Repository Layer). Repositories could be consumed by the decorators that adhere to the same contracts but inject cross-cutting concerns along the way (most common are: performance monitoring and retrying on dead-lock and timeout exceptions).

Repository interfaces could be exposed to the higher-level component layer within the infrastructure - Service Layer. This layer, in its turn, would use these contracts to expose functionality for:

API adapters, exposing services provided by an application via WCF, SOAP, REST or whatever communication technology is being used (validation, paging, authentication and authorization concerns are injected along the way).
Web applications, providing human-manageable UI to work with the same services Automation Engines, leveraging repository and service interfaces to provide some business functionality (notifications, workflows, external integration etc).

If we look at the component infrastructure from the perspective of the top-level consumer, then we will not see something solid. There is just a set of service contracts that are readily available to be consumed and used.

We don't have to worry about relations and dependencies between the components, because IoC Container handles it all for us, while take care of complex object hierarhies and activation chains. More than that, logically we are not even able to care about such things, because the implementation and interface details are hidden from as as well.

That's our highest level development language, if you wish. In it, every service contract represents some syntax element that could be combined with the others in order to deliver business functionality.

That's the first article from the short series of two on Component-Driven Development. In the next one I'll talk about some common strategies for developing greenfield applications CDD-style and migrating existing legacy applications towards the brigher Component-driven and IoC-powered future (which happens to have less development friction and consume less resources).

Meanwhile you can:

Check out the article on Structuring .NET applications with Autofac IoC (which shares a lot in common).
Take your time to listen to Nicholas Blumhardt on Autofac Show.
Subscribe to RSS feed to stay tuned.
Drop a line in the comments, sharing your feedback which is really important for me.

Maintenance and Monitoring Infrastructure for the Enterprise Software Systems

Rinat Abdullin — Mon, 18 May 2009 20:22:27 +0000

Enterprise application software systems are about the software that organizations buy or develop to solve their business specific problems.

Usually these systems start rather small. However, over the time they tend to grow more and more complex, increasing the number of subsystems involved and developing numerous interactions and synergy effects between them.

For example, a company SmartForce might get started with a simple intranet web site and a database behind it. That's the most common scenario of getting started with an in-house enterprise system, that I've seen. That's the easiest one, too.

As the company evolves, so does the software solving its problems. So over the time complex business rules and long-running workflows might be added to our system of SmartForce. These workflows might involve integrating with the external information systems or scheduling document processing for the nights. End-users might get interested in receiving custom notifications or scheduled tasks along the way. One of the most common implementations for this scenario (I'm generalizing here) is the introduction of automation server(s) to our web site deployment with a database backend.

Then, the complexity of the enterprise software system might increase even further, as project stake-holders decide to introduce rich business functionality, integration with office tools or offline capabilities to the external client applications. This might lead us to Smart Client architecture implementations (whether they are done as desktop or browser-hosted applications). This kind of architectural change, generally requires some sort of the public API to be deployed. External integration scenarios also require such an API.

The overall setup looks slightly more complex already. As if it were not enough, we might eventually get to the point of having multiple API versions to maintain and support (that's what usually happens to any API that is exposed to public).

So the overall enterprise software architecture at this point might look like the one displayed on the picture below.

Obviously, things might get a lot more complicated, but we are not showing them on the picture in order to keep it small and clear for the next part of the article. Yet, one can clearly imagine that the system might expand in size and complexity due to the following requirements:

introducing multiple file storage servers integrated into the system;
handling the traffic spikes by setting up a few API endpoints and web servers;
integrating automated issue reporting and deployment updates into the system;
distributing CPU-intensive tasks between multiple automation servers (or even delegating these tasks to the cloud);
provide additional setups of the entire infrastructure for people from development, QA and marketing to play with.

Such an enterprise software system definitely requires some efficient maintenance and health monitoring in order to function properly despite all the problems.

By saying efficient I mean maintenance that does not require having a dedicated team in order detect problems before they happen (that's when fixing them is most cheap), simplifies and speeds up the resolution of issues encountered.

Information is critical for this kind of maintenance (you have to know all about your patient in order to keep him healthy). We need a deep insight into the system, information on its behavior and its deviations under different conditions (as we all know, production reality may be quite different from the development despite all attempts to reproduce real-life stress and behavior profiles in testing scenarios).

For example, we might need to know such basic things as:

uptime and failures of subsystems;
CPU and memory consumption on the machines hosting subsystems;
utilization levels of the services and resources that our infrastructure provides;
repository (database) statistics and performance
network latency and response times of the external services;
various exceptions captured and handled (or unhandled, which should be a rather exceptional scenario) within the code, failed sanity checks and warnings from the internal subsystems;
low-level performance indicators in the code-base (i.e.: execution time, counts or loads of various methods, number of database dead-locks encountered and resolved by the reliability layer, custom counters).

It is not enough just to capture these indicators. If we want to be efficient we could also:

persist indicators in the operations database in order to see how system performance evolves over the time;
present the data in form of real-time reports, making it more easy to spot problems or check the state;
create and receive custom notifications when certain indicators or their aggregations go outside their threshold. For example, system going offline should result in an immediate notification to all involved parties, while slight performance degradation is just worth mentioning in the daily report.

Obviously this entire setup has to be implemented in an efficient manner that allows to:

change any indicator threshold in minutes;
modify or add report in minutes;
automatically version and deploy all changes.

Note, that when providing time estimates, I'm taking into the consideration just the human labor required for the change. Automated deployment processes on their own might take anywhere from 5 minutes and up to 24 hours. This depends on the change propagation policies and processes employed by the specific development infrastructure.

Let's take our previous picture of the enterprise system composition and add maintenance-related subsystems and interactions to the mix. One of the possible results might look like this:

Basically, at this point we are talking about the infrastructure for performing system-wide maintenance and monitoring related tasks.

This sample design works like this.

Every primary subsystem exposes a standard monitoring API that could be used by the monitoring server to capture information. Internally every subsystem is instrumented with simple performance counters, capturing low-level execution (and failure) information and making it available via the monitoring interface API.

Additionally, you may have your external client applications report exceptions directly to your issue management and monitoring systems.

Maintenance monitor then aggregates all the information, applies any notification and warning rules available (some of these may result in notifications being sent out), runs custom reports and publishes them to the dashboard.

That's not that hard to implement with existing open source tools. And having an ability to check how many seconds on the average does that complex repository call run against production db in real-time scenario - makes you wonder how could you live without such functionality before. It is like the ReSharper addiction.

By the way, it may seem logical to extend the picture above by adding the maintenance server that would:

aggregate information from the monitoring subjects across the infrastructure;
match that information with the development and deployment reports in order to provide better OLAP analysis of the infrastructure performance across the:
- resource consumption growth: How do our subsystems handle spikes? Do we provide graceful degradation as resource consumption growth (or is it an exponential curve indicating some critical bottleneck that should be handled soon)?
- development process: How do new versions affect the overall performance? Do they improve or degrade it?
Host and run SLA rules to watch for the resource consumption and provision/surrender resources depending on their consumption levels.

But that's another story, that gets closer to the topic of cloud bursting (aka using cloud computing provider to handle spikes in the resource consumption).

What do you think about this post?:

Do you have monitoring infrastructures in your enterprise application system?
If yes, is your implementation different from the provided simple architecture? What would you like to add to it, if you were able to?
If not, do you think, that it is worth having such an infrastructure?

Visual Studio 2008 Locks or Freezes in ASPX

Rinat Abdullin — Tue, 12 May 2009 09:22:17 +0000

Sometimes Microsoft Visual Studio 2008 (SP1) might lock down either in Solution Explorer or when you try editing ASPX file. All mouse clicks will be followed by Windows beeping sound (the one when you try to click outside the dialog window). Attempts to close VS 2008 will display a warning about waiting for a modal window.

There is one other small detail that identifies this specific problem: when Visual Studio hangs it happens to spawn a Setup process that never exits.

This setup.exe is located in the following directory:

C:\Program Files (x86)\Common Files\microsoft shared\
  OFFICE12\Office Setup Controller\Setup.exe

Running the setup.exe with administrative privilegies was not of much help, but reinstalling Microsoft Office 2007 did fix the issue.

Also if you get unexpected errors in Visual Studio 2008 while closing it or switching the window layout (this happens, for example, when you start debugging), then you've probably encountered "VS 2008 Layout bug".

Clicking "Menu - Window - Reset Window Layout" could help.