class Mp3Encoder : IMp3Encoder
{
    public void Encode(IEnumerable<string> wavFiles)
    {
        foreach (var wavFile in wavFiles)
        {
            var outputFile = /* create output file */;

            while (/* blocks remaining... */)
            {
                var buffer = /* read block */;
                var encoded = /* encode wav block as MP3 */;
                /* write block to output file */;
            }

            /* write ID3 trailing header */;
        }
    }
}

Except in trivially simple cases, there should always be a class boundary when shifting context from coordinating a collection versus performing actions on a single object.

The class above is violating this rule — it knows how to perform collection-level responsibilities as well as single-object responsibilities. It needs to be broken into two classes; one for encoding a single file and one for coordinating the group.

This rule is a form of the Single Responsibility Principle. For example:

Collection-scoped class responsibilities

• Coordinating ‘before all’ and ‘after all’ actions
• Looping through items
• Maintaining shared state (counting, accumulating etc)

Single object-scoped class responsibilities

• Coordinating ‘before each’ and ‘after each’ actions
• Performing actions on item

If you ignore this collection-vs-single-object contextual boundary, your classes will become messes of nested procedural code — especially when different behaviour is required for each item in the collection. Your classes will be that much harder to unit test, and you won’t easily be able to re-use them in single-object scenarios.

For example, in order to keep protobuf-net-data in sync with the latest protobuf-net, I need to publish a new package rebuilt against the latest protobuf-net data every time they release a new version. For this I subscribed to the following URL in Google Reader:

http://packages.nuget.org/v1/FeedService.svc/Packages()?$filter=Id eq ‘protobuf-net’

Matt Wrock has a few more advanced examples of this using ifttt.com to orchestrate sending emails and tracking package downloads etc.

However, for data that is tabular in nature, there aren’t so many options. In my past two jobs I’ve had a need to serialize data:

• That is tabular – not necessarily CLR DTOs.
• Where the schema is unknown before it is deserialized – each data set can have totally different columns.
• In a way that is streamable, so entire entire data sets do not have to be buffered in memory at once.
• That can be as large as hundreds of thousands of rows/columns.
• In a reasonably performant manner.
• In a way that could potentially be read by different platforms.
• Into as small a number of bytes as possible.

Protocol Buffers DataReader Extensions for .NET was born out of these needs. It’s powered by Marc Gravell’s excellent Google Protocol Buffers library, protobuf-net, and it packs data faster and smaller than the equivalent DataTable.Save/Write XML:

Usage is very easy. Serializing a data reader to a stream:

DataTable dt = ...;

using (Stream stream = File.OpenWrite("C:\foo.dat"))
using (IDataReader reader = dt.CreateDataReader())
{
    DataSerializer.Serialize(stream, reader);
}

Loading a data table from a stream:

DataTable dt = new DataTable();

using (Stream stream = File.OpenRead("C:\foo.dat"))
using (IDataReader reader = DataSerializer.Deserialize(stream))
{
    dt.Load(reader);
}

It works with IDataReaders, DataTables and DataSets (even nested DataTables). You can download the protobuf-net-data from NuGet, or grab the source from the GitHub project page.

For me, it’s a pretty long journey, up to an hour each way to get from Parsons Green to Canary Wharf — a large portion of which is spent deep underground, with no mobile or 3G coverage on my iPhone 4. During this time you can really tell which apps’ data strategies have been properly thought out and designed, and which ones have been hacked together in a hurry.

I’m not going to mention any offending games or apps by name, but these are the sort of things developers should be shot for:

Apps that don’t cache downloaded data for later viewing

Except in special cases (e.g. streaming audio/video, realtime or sensitive content), all downloaded data must be cached locally, and able to be viewed again without an internet connection. Apps that don’t store downloaded content are just crippled web browsers.

Offline App Store reviews

It is unacceptable to nag me to rate your app in the App Store when there is no internet connection. Honestly — don’t ask users to do the impossible.

Repeated connection attempts and repeated error diaglogs

It is unacceptable to repeatedly show error dialogs when there is no internet connection. By all means, try to continuously download updates, but do it silently and in the background. I do not need to be told every 30 seconds that it (still) couldn’t connect.

Locking the UI when checking for updates on startup

When launched, It is unacceptable to block the user in the UI from reading previously-cached data while downloading updated content. By all means, check for updates on start up, but while that is happening, the user must be able to read the previous cached copy, unless they explicitly asked for a refresh from-scratch.

Throwing away cached data then failing to update

It is unacceptable to leave the user with a blank screen because you threw away previously-cached data THEN failed to update. Local data caches should not be cleared until new data has been 100% retrieved successfully and loaded.

Lost form data, please retype

t is unacceptable to ‘lose’ form values and force the user to retype a message when a submit error occurs. Form values must be saved somewhere — either automatically requeued for resubmission, or as a draft that can be edited or cancelled by the user.

Ad priorities

It is unacceptable for an ad-supported app to crash, refuse to start or behave weirdly because it couldn’t connect to the ad server. Are you worried about people using your app, or seeing ads?

Occasionally connected apps

It is unacceptable for any app to refuse to start without an internet connection. Users should always be able to change settings, view and edit previously-cached data and enqueue new requests where possible (to be submitted when a connection is available).

Final thoughts

In general, occasionally-connected computing is a good model for iPhone/Android apps that push/pull data from the Internet. It just needs to be implemented correctly. Like web designers assuming no javascript and using progressive-enhancement to add improved behaviour on top of basic functionality, mobile developers should assume no internet connection is available most of the time — jump at the opportunity to sync when possible, but don’t interrupt the user or assume it won’t fail.

So please, if If you’re a mobile developer, think about us poor Londoners before you deploy your next version!

To help make this easier, here are a couple of MSBuild tasks I wrote that allow you to drop an existing database:

<SqlDropDatabase ConnectionString="Server=.\SQLEXPRESS;Database=AdventureWorks;Integrated Security=SSPI;"
                 Database="AdventureWorks" />

… and to create a new (empty) one.

<SqlCreateDatabase ConnectionString="Server=.\SQLEXPRESS;Database=AdventureWorks;Integrated Security=SSPI;"
                   Database="AdventureWorks" />

It’s also sometimes helpful to be able to parse individual individual keys out of a connection string (e.g. the the database name). This can be very tedious with RegexMatch/RegexReplace, so I wrote a separate MSBuild task to do it:

<SqlParseConnectionString ConnectionString="Server=.\SQLEXPRESS;Database=AdventureWorks;Integrated Security=SSPI;">
    <Output PropertyName="myDb" TaskParameter="InitialCatalog" />
    <Output PropertyName="myServer" TaskParameter="DataSource" />
    <Output PropertyName="myTimeout" TaskParameter="ConnectTimeout" />
</SqlParseConnectionString>
<Message Text="Parsed the $(myDb) database on server $(myServer) with timeout = $(myTimeout)." />

You can grab the code here: http://github.com/rdingwall/sqlmsbuildtasks

• Highest granularity – unit tests, quick to run, drive low-level class design
• Fragmented – integration tests, testing higher-level components and external dependencies (e.g. real SQL database, fake message bus)
• Whole system at once – slow end-to-end tests, testing for story/feature acceptance at the UI or client API level

However, it has become increasingly clear that fragmented integration tests (somewhere in the middle between unit and full-blown end-to-end integration tests) don’t really provide the same value as their brothers.

They suffer all the disadvantages of both unit and end-to-end tests:

• Like unit tests, they require setting up mocks/test doubles for collaborating modules
• Like end-to-end tests, they have a high cost of maintaining external components like databases and integrated third party systems (setting them up, populating them with the right test data, reverting changes for next time etc)
• Like unit tests, they are brittle and not friendly to refactoring
• Like end-to-end tests, they are slow to run

… and, over the lifetime of the code base, the only real benefit they provide is slightly faster feedback of problems than end-to-end tests. Is it really worth keeping them?

The method itself still works. There’s nothing wrong with it. As long as you’re only doing one session per WCF operation (or one session per ASP.NET MVC request for that matter), it works great. However, problems arise when you mix other session styles.

For example, our application has a series of long-running background threads for processing jobs on a queue (up to an hour each) — as well as synchronous WCF operations — all requiring NHibernate. In this case, the one-size-fits-all approach using AOP and the container simply couldn’t give us the control we needed over each session’s lifetime, so we removed the Facility from our Windsor container, and instead now do:

• Register the ISessionFactory in the container, singleton scoped.
• Inject ISessionFactory and call ISessionFactory.OpenSession(), ISession.BeginTransaction() and ITransaction.Commit() at outer boundaries of your application (e.g. WCF service implementations), and background jobs executing outside the context of a WCF operation.
• Pass ISession down through your application as a parameter into any methods that access the database. Callees must not hold any reference to it, or try to manage its lifestyle (e.g. closing it, opening their own transactions etc).

(Note: all NHibernate calls are wrapped in a transaction and committed. Even if they are only SELECT statements, it’s still a good practice.)

It has more-or-less the same end effect as AutoTx facility and CallContext-scoped SessionManager, but with the added control of being able to explicitly manage session lifetimes in other situations. It’s a few more lines of code, but it’s also a good example of stop trying to hide your ORM, and just embrace the ISession directly. And a reminder that technical solutions are never infallible, no matter how elegant they might seem.

A method is throwing a NullReferenceException. The stack trace indicates it is originating on a line that has only a closing curly brace on it (i.e. }). How could this be explained?

Note that:

1. The code is not optimized.
2. The PDB file is the correct version.
3. The method has no using statements.
4. There is no code generation or reflection happening.

Stumped? Okay, here’s a simple case to reproduce it.

void Test()
{
    try
    {
        (null as string).ToLower();
    }
    catch
    {
        throw;
    }
}

If you said this code reports a NullReferenceException on line 5, you’d be wrong. The CLR reports it coming from line 11. Weird, huh? It seems to be something about rethrowing unwrapped NullReferenceExceptions, either as throw or throw e. If I throw new Exception("...", e) it works as expected.

Anyone care to hazard an explanation for this behaviour?

A reference test case is available here, with observed output here. (Compiling under Any CPU, .NET 4 in Debug configuration in VS2010 SP1.)

One of the biggest challenges here is how to handle your app’s back-end database. Application files can be deleted and restored in seconds, websites rolled and and replaced a thousand times, but the database is a living, breathing beast, closely guarded by a separate team of DBAs who usually won’t even let you see what state its in, let alone make changes. If you screw up a database script, it could cost hours of downtime restoring from a backup. So how do you automate changes then? Here’s how we did it on a recent project.

Pre-requisite #1: portable script shipping

I’ll never forget the deployment process on my first major commercial project. During development, everyone shared a ‘working copy’ database, directly adding columns and tables via GUI tools. On release day, a designated person would open up SQL Compare, connect to the dev database, connect to the live database, and generate a script copying all the changes from dev to production.

Needless to say, this was a fairly sub-optimal solution. Here’s some of the problems we developers incurred on a regular basis:

• The diff scripts only work between specific database versions. If UAT was on a faster development cycle than PROD, we couldn’t use the same scripts between them.
• Only the final diff script was put in source control, so if the development database crashed or was accidentally wiped in between release cycles, we lost all our changes.
• SQL Compare can reverse-engineer schema changes, but it can’t reverse-engineer data changes/migration between tables. It’s not even a complete solution.

The last one was the real nail in the coffin for us. If you already have to script data changes by hand… why not go the whole hog and do ALTER/CREATE statements too? That way you can just check in each change as a separate sql script into source control — numbered, so you know which order they go in. No more generating scripts, no more losing our work in the dev database. No more SQL Compare.

Pre-requisite #2: database version tracking table

Given a live database and a stack of SQL scripts, how do you know which ones need to be run? You could start reading going through them, statement by statement, until you find a change that doesn’t appear to be applied and start there — or you could simply have each script write a row to the end of a changelog table like this:

ALTER TABLE ...;
DROP TABLE ...;
-- other DDL changes

INSERT INTO CHANGELOG (16, '0016_customer_info.sql', GETDATE(), USER);

That way, it becomes trivial to find out what scripts have run/need to be run against a database. And putting the special INSERT at the end of each change script ensures it doesn’t add a row unless all previous statements executed successfully.

Pre-requisite #3: database version awareness

When is increasing coupling in your application a good thing? When it’s between the database schema version, and the application running over the top. Your physical data model and application’s persistence model (ORM mappings etc) have to move in lockstep, so it’s a good idea to refuse to start if the database version isn’t correct.

You can do this quite easily by checking the highest-run script number in the CHANGELOG table, or introducing a separate VERSION table. Whatever the strategy used, a loud error on startup is much easier to diagnose than messy runtime errors when a column is missing (fail-fast).

Pre-requisite #4: script privileges

To make our lives easier, we introduced a new rule: all change scripts must be able to be run under the same database user/privileges as the application itself. This means we can use the application’s own connection string to run scripts, and eliminates the need for DBAs or sharing the database administrator password.

Running scripts with the application account means we have to grant our services rights to create and modify tables, views, stored procedures etc (i.e. Oracle’s RESOURCE or MS SQL’s db_ddladmin role). Before you freak out about applications running with elevated privileges, please remember one, this is not a shared database — it’s for exclusive use of our application, there is only one user — and two, this application does not run on desktops, so there is no risk of users uncovering credentials and logging in themselves.

This is the same self-upgrading database model WordPress uses, and we justified that if it’s good enough for the 12 million WordPress sites out there on the greater Internet, then it’s good enough for one little Windows Service running in a locked-down server network.

Anyway. If a script does requires system privileges to run, it probably means it has something to do with storage or security — e.g. storage files or table partitioning — that isn’t actually required for the application to run and thus doesn’t need to be part of the official database version history. Those scripts can thus be run independently of the automatic-upgrade process, as required by DBAs.

Pre-requisite #5: clean up existing installations

We have a number of copies of our applications running in the wild, in various states of disrepair. Scripts that were never run, or ran with ‘continue on error’ enabled, and ad hoc customizations by DBAs, and missing entries from the CHANGELOG table all needed to be smoothed over to make a solid platform before any automated process can take over. In this case, we did it by hand (ironically using SQL Compare), but depending on the number of installations out there, and effort required, you may wish to incorporate these clean up steps in your automated scripts.

End goal: self-upgrading application

From this point, it’s relatively easy to implement an automated upgrade process that checks the CHANGELOG table to find out which scripts haven’t been run yet, and run them. That way it works against any previous database version. Our upgrader tool also performs a number of prepatory and clean-up steps, such as creating the CHANGELOG table if required, and cleaning up any vestigal remains of the previous database deployment process.

This process can be designed, and unit tested, and audited to a greater degree than any human process could ever achieve. You can run it hundreds of times over to tweak it, and test it against all sorts of different database configurations with minimal effort. It can and should become part of your continuous integration and automated build and tests.

Failure and backups

Unfortunately, even with all the extra confidence you get from scripting, there is always a risk of something going wrong on release day. One part of the manual process we’re missing is backups, so we’re currently looking into to add an automated backup step before kicking off (probably a lightweight one, using something like a Snapshot in MS SQL or Flashback in Oracle). That way, we can eventually aim to provide automatic rollback capabilities in the event a script fails.

Final words

I’m not saying that the ours is the only way of achieveing automated database deployment, or that this is neccessarily the best way. We’re still working out the kinks, but I’d say it’s looking pretty good so far.

• Models
• Views
• ViewModels
• Commands
• Events

If Commands and Events aren’t strongly represented in your application, I’d say there is a strong chance it isn’t factored very well.

One problem I see quite often is the Fat ViewModel: a long, bloated class that violates SRP by presenting complex application Commands to the view, and implementing them too.

Symptoms

You can tell you have a Fat ViewModel if any of the following are true:

• Instead of having a separate class for each Command, your ViewModel uses DelegateCommands (aka RelayCommands) to invoke methods on itself.
• Your ViewModel has three properties, two commands, and is 300 lines long.
• Your ViewModel has a large number of services injected into it.
• Your ViewModel tests include assertions for both presentation and application behaviour.
• Testing simple UI behaviour — e.g. a button should be disabled after it has been clicked — requires excessive mocking of dependencies.
• Your ViewModel tests frequently call xyzCommand.Execute().
• Commands cannot easily be re-used by other ViewModels. Your application is not modular.
• ViewModels update their state directly, instead of listening for global application Events.

Instead, if you extract Commands into separate classes, that publish their results via Events, you will benefit by having:

• Skinny and light ViewModels.
• Commands that can be developed and tested in isolation.
• Commands that encapsulate and mask complex logic.
• Separation of presentation logic (ViewModels) from application logic (Commands).
• A catalog of clearly defined and reusable Commands that be composed into new ViewModels.
  
  

Whither DelegateCommand?

From the cases I have seen, I have no doubt that DelegateCommands are the primary cause of Fat ViewModels — they encourage developers to implement Commands using local ViewModel methods, which results in poorly-factored code. For this reason, I consider DelegateCommands to be in the same bad code smell category as ServiceLocator: as an anti-pattern, with few legitimate uses. (One is when the Command is entirely self-contained within the ViewModel (i.e. does not collaborate with any other object), the other when genuinely delegating to another object. But in that case, the other object should probably be a Command anyway.)

See also: Fat Controller, God Object.