Steve's Blog

Radio Silence

noreply@blogger.com (Steve) — Thu, 11 Jun 2009 13:55:00 +0000

For anyone subscribed to this blog (Hi Mum!), you've probably noticed a distinct lack of posts. Lately, I've mainly been posting on my company blog, so if you're interested in my ramblings then I suggest you head over to http://blogs.imeta.co.uk/sstrong and subscribe there. You can also follow me on http://twitter.com/srstrong.

I may still post occasional things here, but it'll be more personal stuff rather than work or tech related.

On Agile

noreply@blogger.com (Steve) — Wed, 03 Dec 2008 22:31:00 +0000

Following a couple of other posts relating to agile (here & here), I thought I'd give my thoughts on the subject.

There are numerous books and posts on the subject of Agile - searching for "Agile Development" on Google gives well over 1 million hits. As someone who is hugely interested in how to improve software development, I've read a good deal of material on the subject (although, being fair, it would only be a fraction of a percent of those Google hits!). What I've found very interesting is the number of different opinions as to what Agile means - it's pretty hard to find two books on the subject that agree on a definition of the term.

One thing that is for sure - Agile (whatever definition you may have) is no silver bullet. Just being "Agile" does not guarantee success. It does not guarantee the early delivery of rock-solid software to delirious customers. It does not guarantee that your staff will become highly motivated software ninjas overnight. Given that, what's all the fuss about? The fuss is that previous software methodologies have got a woeful delivery record. I'm not going to quote some specific number here, since there are a ton of research articles available which give differing statistics depending on exactly what is defined as failure. However, the general range is anything from 30% - 85%. Even if you take the best number there, it's clearly an unacceptable state for a multi-billion dollar industry.

Given the lack of consistency for what Agile means, and given the proliferation of processes that claim to be Agile (XP, SCRUM, Crystal etc.), what is the best way for you to move forward? Alas, I don't have a silver bullet here either. The best I can do is describe the process that I have gone through to get this stuff clear in my head.

The first step is to clearly understand what the key requirements are for your business. Ignore software, ignore the process, just focus on what matters to the business. For me, I came up with a single overriding principal:

To make a decent, long term, sustainable profit

I'm guessing that's not particularly contentious - any business for which that isn't a key principal is unlikely to last long. However on it's own that's not really enough. So I then pondered what aspects of the business would best support the primary principal. This gave the following, in no particular order:

To create solutions that give the customer the solution that he needs, when he needs it, and at a cost he is happy with
- Happy customers mean repeat business and referrals, which are the best and most profitable form of sales. Plus I'd much rather be working with a customer who's happy rather than one who's not.

To have control over the costs and timescales for each customer project
- Not having control means that the customer doesn't know how much he is paying or when he's going to receive the goods. Given that the software is probably only a part of a bigger project, the timescales in particular are critical - without confidence on delivery dates, how is the customer meant to schedule other aspects of the wider project such as training, marketing, manufacturing etc? In addition, if we don't know when projects are due to complete it makes it very hard to commit to start dates new customers.

To have a motivated, skilled team that share the same values
- Motivated staff take care and pride in their work, which gives rich rewards in terms of the quality of the product. They also tend to enjoy their work which reduces expensive staff turnover, and just makes the workplace a more enjoyable place to spend what amounts to a significant percentage of your life.

From this, I could then extract the aspect of Agile from the mass of books that I've read to provide, in effect, the essence. These are the areas that I consider important, and why:

Accept and embrace change.
- Anyone who thinks that they can prevent change from occurring during a project of any size is, quite frankly, living on another planet. Accept that change is essential to enable the software to meet the needs of the customer, and adopt a process that makes change as painless as possible. The change may be requirements, it may be technical, it may be staffing, but whatever it is, it's going to happen.
Develop in short iterations.
- There's a load of important things about the iterative approach, so I'll expand on this below.
Don't attempt up-front detailed design
- Again, there's lots of empirical evidence that this doesn't provide any benefit. By all means, do up front high-level architectural design - indeed, for the key structural aspects of the project (scalability, security, disaster recovery etc.), this is pretty much essential; getting those wrong or trying to reverse them into an existing code base can be very expensive and not something the "right click / refactor" is going to help with. But the low level stuff is best done with the compiler.
Make sure that testing is a first class citizen
- Testing should being as soon as coding begins (indeed, if you want to do TDD, then it starts before the coding). As far as possible, make the tests automated so that you can frequently run the full test suite. It's inevitable (regardless of whether you attempt up-front detailed design or not) that, at times, you'll need to refactor parts of the code base to support new functionality. At times like this, a large test suite gives a great safety net. In addition, the tests (if well written) also act as a form of executable (and hence up to date) documentation. Finally, and most importantly, testing as early as possible tends to promote a testable code base and gets quality in there from day one.
Empower your team to use their brains.
- If you've any sense, you spend a lot of time recruiting the very best staff. Recognize that, and let them shine. If they are committed to the business, then trust them to make sane choices and don't try to micro-manage tem. If they're not committed to the business, politely but firmly point them in the direction of the door.

Expanding on iterations, I think the following aspects are essential:

At the start of an iteration, plan in detail what you are going to achieve. For that plan to have any teeth, it is essential that changes are not allowed during an iteration. For this reason, iterations should also not be too long - my experience suggests that between 2 and 4 weeks works well.
During the iteration planning, ensure that the tasks being worked on are the most important to the project at this moment in time. Don't do the trivial stuff whilst there are important things to be done (important can either be those items that give most business value, or those items that present the most technical risk)
At the end of each iteration, deliver demonstrable, working software. This keeps the team focused and gives a clear view of progress to date. In addition if, god forbid, you fail to complete all the development tasks you will at least have a system that the customer could take. And, since you worked in priority order, it should include most of the stuff that the customer considers important. Telling the customer that you're not finished is never an easy conversation, but "We're not done, we estimate that we're about 80% of the way there, but here's a system in which all of the following function is complete and ready to go" is a much better chat than "We're not done, we estimate that we're about 80% of the way there. Sorry, but there's nothing you can take yet because until we've done the other 20% nothing will work"
At the end of each iteration, tasks are either done or not done. It's notoriously hard to determine how much work is left on a task when it's not yet complete (how often have you heard the phrase "it's 80% done", only to then find it takes another 100% of the elapsed time so far to finish?). In addition, done needs to be done. Code written, all functionality complete, all tests written and passing. Anything less is not done, and should hence be deferred to a future iteration.
At the end of each iteration, evaluate what has gone well and what has gone badly. Do more of the good things, and make changes to prevent the bad things from happening again. I have seen a number of teams running iterations who recognize that they are not getting things done as quickly as they need, and whose response is "ok, we recognize that things aren't going well. We'll try harder in the next iteration". Trying harder at something that isn't working is unlikely to yield the results that you want.
Don't queue up bugs. The tasks that you've worked on to date are, by definition, the most important ones. Bugs mean that they are not finished. Fix the bugs. If you don't, then at the end of the project you'll have a pile of important stuff that's not done. The customer is not going to like that.

The key thing that I've observed with well-run iterations is that they tend to surface problems early in the project. Pain is going to happen (what - you really think that nothing will go wrong?), and Early Pain is considerably more desirable than Late Pain. Early Pain means that there's time to take corrective action. Late Pain is what kills projects.

Those are the things that I see as the essence of agile - it's not rocket science, it's just working smarter. It's understanding that change happens, and making sure that you can handle it. It's understanding that things go wrong, and making sure that they can be spotted and fixed as early as possible. It's understanding that you have a hugely talented team, and using them. It's no silver bullet though.

What else can I add? Well, there are a few things that spring to mind:

Requirements - up-front or iteratively?
Contracts - fixed price or T&M?
What methodology? SCRUM? XP? Lean?
We're not Agile, but want to be - how do we change?
My customer doesn't want Agile - what are my options?

This has been a pretty long post, so although I've got things to say on those, I'll leave them for another day.

As a final remark, all of the above is just my opinion. I've intentionally not put in references to books etc. - for each reference I find that says one thing, I've no doubt you can find a reference that says the opposite. Such is the nature of our imprecise world. I hope, however, that this does give some food for thought, and perhaps helps you through your own thought processes around how (or indeed if) to adopt Agile.

I would certainly appreciate any comments that folk might have...

noreply@blogger.com (Steve) — Mon, 10 Nov 2008 10:07:00 +0000

This should have been sent last Friday, but other stuff got the better of me. Anyhow, here's last weeks "interesting stuff":

The reason why we have iterations: Early Pain

An example of a simple TODO language implemented in Mg. For those use to parsers, BNFs etc, this will be an easy read. For those that haven't come across such things, it's a nice introduction.

How SanDisk's ExtremeFFS works. Bets on how long it is before Joe gets one :)

A nice summary of PDC. Thanks, John - saves me writing it!

Access to the ETW APIs from .Net using NTrace. Not used it yet, but suspect this is worth keeping an eye on.

A new TFS Power Tools release. Surprised that Obama got quite so much press coverage considering that this came out the same week.

Code Contracts in C# (well, any .Net language in fact). It's due out with .Net 4.0, but it also looks like it works with VS2008. Getting into the habit of declaring the intent of your code in this way can only be a good thing - it helps out with documentation, static analysis, runtime checks and gives Pex something to get its teeth into.

A good reality check on some of the hype going on about Oslo right now.

Hadi is twittering. Hell has also frozen over.

For those of you with a media center and an iPhone / iPod touch, this is cool. I've been running it for a week or so (before this review came out), and although at $6 it's one of the most expensive iPhone apps I've bought, it's a load cheaper than the wireless mouse / keyboard that I would have bought otherwise.

Should we put comments in our code? Neal Ford has shares his views here - not sure I entirely agree with him, but I do respect his opinion and it's hard to refute his reasoning. Some of the comments are pretty decent, so it's worth reading the whole page.

Weekly digest of interesting stuff

noreply@blogger.com (Steve) — Fri, 17 Oct 2008 11:54:00 +0000

News that the D language that is part of the up-coming Oslo release has been renamed to M. Wonder what that stands for?
Silverlight 2 goes RTM. Various useful links:
- The announcement from Somesegar
- Information about up-coming Silverlight controls. The model for how these are getting released is worth reading - full source on Codeplex under the MS public licence
- A whole pile of useful stuff on what's new, what's changed etc.
- The release of Expression Blend SP1
Changing extensions in IIS 7 - the post is about removing the .svc extension, but the method used works for pretty much any type of URL renaming that you want to perform
A Channel9 video discussing the provider model in LINQ. I've not watched it yet, but Erik Meyer is a smart guy and normally worth watching. Beware of his shirts though, they are normally pretty shocking!
Apple released updates to their MacBook & MacBook Pro lines. These are nice looking machines, and with bootcamp they will run Vista just fine. They aren't the cheapest laptops around, but they do compare reasonably well on price with other manufacturers for similar specs / designs.

Visual Studio Tests and "The location of the file or directory xxx is not trusted"

noreply@blogger.com (Steve) — Fri, 17 Oct 2008 10:49:00 +0000

If you've ever downloaded stuff from the internet (from XP SP2 onwards), you'll know that the OS marks the file as un-trusted and that to use it you need to right-click, go to properties and click "Unblock". If you download a zip and extract it's contents without having unblocked the zip, then you probably also know that every extracted file will also be marked as un-trusted.

Referencing a file from an un-trusted source within Visual Studio (for example, adding a reference to the latest library that you've just downloaded) will work just fine, but when you try to run your unit tests they will barf with a "location of the file or directory is not trusted" error.

The top tip is to remember to unblock the zip file before doing the extract; that way, all the files that you extract will also be unblocked and the world will be a happy place (assuming that none of them have viruses, trojans etc!)

If you forget, and if you've now got files scattered all over your project such that deleting them and re-extracting is a pain, then here's another option - ZoneStripper is a handy command-line tool that can run recursively over a directory and get rid of the blocks. Works like a treat.

Running 32 bit .Net applications on a 64 bit machine

noreply@blogger.com (Steve) — Tue, 14 Oct 2008 13:52:00 +0000

Just been having a problem trying to get a .Net application running on my 64bit Vista machine. The app was compiled using the "Any CPU" flag, which means that it can target both 32 or 64 bit machines. With this flag set, the .Net bootstrap process loads up the appropriate CLR based upon your machine architecture - if you're running a 32 bit OS, then the 32 bit CLR is loaded and the JIT compiler generates 32 bit code. If you're running a 64 bit OS, then you can guess what happens - 64 bit CLR and 64 bit code.

For most apps, this is probably exactly the behaviour that you want (although do remember that you need to test on both environments). If there is a reason why you need to target a specific architecture, then you can change the build settings to force either 32 or 64 bit. Why would you want to force things? A good reason would be if you are loading up 32 bit native DLLs for some reason (perhaps your database vendor only ships 32 bit client DLLs, for example). In this case, quite clearly you need to make sure that your app is also running 32 bit - if not, then when launched on a 64 bit OS, it'll throw a BadImageFormatException at the point that it tries to load the native DLL.

That's all fine, but what if you've been given an app that needs to run in 32 bit mode, but was compiled with the "Any CPU" flag? If you've got the code then you could recompile, but what if that's not an option? Turns out that there's a handy tool called corflags.exe which comes with the SDK. Using this, you can flip the 32 bit flag in the application without requiring access to the source. For example:

corflags /32BIT+ /Force TheApplication.exe

the /Force flag is needed if the application is strong name signed - if you omit that flag, then it will fail when run against such assemblies. Obviously, once the bit has been flipped the strong name is no longer valid. If you've got access to the private key then you can re-sign. If you don't have the private key and you need to keep the strong name then I'm afraid you're out of luck.

Weekly digest of interesting stuff

noreply@blogger.com (Steve) — Fri, 10 Oct 2008 09:51:00 +0000

I read quite a few blogs and web sites over the course of each week, and thought it might be useful to do a weekly blog on the things that I've found interesting. If nothing else, it gives me a way of keeping track of things - but hopefully some of you will find it useful as well...

The Austrian's now have the first computer network protected by quantum encryption. This stuff has been in the labs for a while, but this is the first (reported) implementation "in the wild".
A largely speculative post, but it's one of the first to discuss the new name from Microsoft: Windows Strata.
Eager loading in the Entity Framework, and how it differs from LINQ to SQL.
New MSBuild extensions library released.
"Library Oriented Programming" - is this what we're all doing now?
A list of links that details the new stuff, and the broken stuff, in .Net 3.5 SP1. I know it's been out a while, but it's nice to have one place to refer to :)
A list of the talks that Don Box wants to see at PDC. And if Don's interested, then you probably should be too.
Some VSTS 2010 features. It's a while until we'll see this, so it's nice to see what's coming our way.
For anyone into encryption,I think you'll find the Silverlight Enigma Machine kind of cute.
I've not watched it myself yet, but any video of Anders talking about language design is likely to be worth watching. If you're into that sort of thing, of course!
A really good article on what makes a good software architect. If you think you are, or think you'd like to be, or work in a team with one, or have the misfortune to be married to one (or, worse, to be married to someone who thinks they are!), then it's worth a read.

WPF Localization, part II

noreply@blogger.com (Steve) — Fri, 03 Oct 2008 10:47:00 +0000

Following on from the previous blog, I'll briefly look here at how to localise strings that you build within your application as opposed to resources that are within the XAML. There are a variety of options here, but this approach seems pretty simple and plays nicely with the locabaml / csv approach described earlier.

Step 1:

Add a new item to your project, of type Resource Dictionary. By default, VS will name it Dictionary1.xaml.

Step 2:

Edit the App.xaml file to reference the new resource dictionary. The markup you'll need is:

<Application.Resources>
   <ResourceDictionary x:Uid="ResourceDictionary_1">
      <ResourceDictionary.MergedDictionaries>
         <ResourceDictionary x:Uid="ResourceDictionary_2" Source="Dictionary1.xaml" />
      </ResourceDictionary.MergedDictionaries>
   </ResourceDictionary>
</Application.Resources>

Step 3:

Let's now add some code and content. Assuming that you are using the "app" we built in the previous blog, double click on the button in the designer to add a Click event and the corresponding method in the code behind file. Within the method, add the following code:

MessageBox.Show((string)Application.Current.FindResource("buttonMessage"));

"buttonMessage" is the name of the resource that we want to display - edit the Dictionary1.xaml file to include it:

<ResourceDictionary
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:system="clr-namespace:System;assembly=mscorlib">

<system:String x:Key="buttonMessage">Hello World!</system:String>
</ResourceDictionary>

If you are running in the default locale ("en-GB" in our previous example), the app should run and you should get a message box with the text "Hello World" when you click the button.

Step 4:

Back to our friends from before. Do the following:

run "msbuild /t:updateuid" from your project directory
build the project
run "locbaml /parse" from your bin\debug directory
translate the corresponding csv file
run "locbaml /generate" to build the new resource dll

(I've not put in the full command lines here, since they are exactly as in the previous blog).

Once you've done this, if you switch culture to your new culture, you should find that the message box now displays the translated text.

As before, this isn't too invasive on the project - for each string that you handle in code, add a resource to the resource dictionary and use the FindResource() method to retrieve the string. That's pretty much it.

WPF Localization

noreply@blogger.com (Steve) — Fri, 03 Oct 2008 09:52:00 +0000

There are a good number of articles out there on how to localize WPF resources, including the main one on MSDN. However, they are all quite long-winded and having now gone through all the necessary steps I thought I'd just document exactly what you have to do. Hopefully this will be of some help to others.

The support for localization within WPF is actually pretty good, but unfortunately it is not quite as integrated with the IDE as one might like, and does involve some work on the command line. For all you "point'n'click" warriors, I think the best I can do is to send you here.

Step 1:

Download and compile locbaml. You can find it here. Once you've downloaded and "installed" it, head to the directory you chose and go into the "csharp" folder. In there, you'll see a csproj file. Either open that in Visual Studio and build, or run msbuild from the command prompt. Either way, you'll end up with a nice new locbaml.exe just a couple of seconds later.

Step 2:

Create yourself a new WPF project within Visual Studio, and the close the IDE straight away. Open up the new .csproj file in your favorite text editor, and add the following property to the first PropertyGroup section:

<UICulture>en-GB</UICulture>

where "en-GB" is the culture that you are developing in. For a full list of culture codes, head here.

Open up the project again, and open the AssemblyInfo.cs file. Add the following line to the bottom:

[assembly: NeutralResourcesLanguage("en-GB", UltimateResourceFallbackLocation.Satellite)]

If you build & run now, your app should work just fine. If you take a peek in the bin\debug directory, you'll notice that VS has created an en-BG directory and popped a new dll in there. This dll is a satellite assembly containing just the resources for that culture.

Step 3:

Before localizing, let's add some content to our application. Open up Window1.xaml, and add the following as a child of the Grid element:

<Button>Hello</Button>

You can look at all of Fran's blogs for details on how to make it prettier :)

Step 4:

Here's where we drop down to the command line. Spin up a VS Command Prompt, and cd to the directory that contains your project. Run the following:

msbuild /t:updateuid

This will walk all the XAML in your project and add Uid attributes to pretty much everything. It does mess the XAML up a smidge, but that's just how it goes.

Step 5:

Copy the locbaml.exe that you created in step 1 to the bin/debug directory. Then run:

LocBaml.exe /parse en-GB\WpfLocalization.resources.dll

If all goes well, you should now have a .CSV file containing all the resource information which locbaml has extracted from the default resources dll. Mine looks something like this:

WpfLocalization1.g.en-GB.resources:window1.baml,Window_1:WpfLocalization.Window1.$Content,None,True,True,,#Grid_1;
WpfLocalization1.g.en-GB.resources:window1.baml,Window_1:System.Windows.Window.Title,Title,True,True,,Window1
WpfLocalization1.g.en-GB.resources:window1.baml,Window_1:System.Windows.FrameworkElement.Height,None,False,True,,300
WpfLocalization1.g.en-GB.resources:window1.baml,Window_1:System.Windows.FrameworkElement.Width,None,False,True,,300
WpfLocalization1.g.en-GB.resources:window1.baml,Button_1:System.Windows.Controls.Button.$Content,Button,True,True,,Hello

The columns in this file are:

Baml Name - the name of the compiled XAML stream that is held in the resource dll
Resource Key - maps back to the Uid attributes that we created in Step 4. Actually, it's even better than that, since it doesn't just have the content of the element, it also allows for the modification of some of the other properties. So in the example above, on the element with the Uid of Window_1, I can change the $Content, Title, Height and Width properties.
Localization Category - A value from the LocalizationCategory enumeration
Readable: Is the value visible for translation
Modifiable: Is the value modifiable during the translation
Comments: Any comments - more on this later
Value: The value of the property. This is the chap that needs translating

Step 6:

Translate the file. Can't help you there :)

Step 7:

So you've now got a new version of the CSV file in the new culture (let's say it's es-ES). From this, we need to build a new satellite assembly. Drop the file into the bin\debug directory alongside the original, and then run this:

locbaml /generate en-GB\WpfLocalization.resources.dll /trans:WpfLocalization.resources.CSV /out:es-ES /cul:es-ES

It should be pretty obvious what's happening here - from the information present in the original resource dll, and with the updated values in the CSV, generate a new resource dll in the es-ES directory, for the culture es-ES. Note that you need to create the output directory first.

Step 8:

Test it. Either change your locale in the control panel, or add some code to the constructor of your App class (in App.xaml.cs):

public App()
{
   CultureInfo ci = new CultureInfo("es-ES");
   Thread.CurrentThread.CurrentCulture = ci;
   Thread.CurrentThread.CurrentUICulture = ci;
}

Note that you do need to do this pretty early on in the application lifecycle; once resource start being read, then you are fixed in whatever locale you were at the time. You can't use this approach to switch languages dynamically once the app is loaded - if that's a requirement, then I'm afraid you need to head back to google.

Conclusion

That's it. It's a bit of an effort first time round, but once you get used to things it's pretty simple. To summarise:

You can pretty much develop your WPF app as normal. (just add the UICulture to the .csproj and the NeutralResourceLanguage attribute to AssemblyInfo.cs)
At suitable points, run "msbuild /t:updateuids" to refresh your Uids, and run "locbaml /parse" to get your latest CSV
Once translated, run "locbaml /generate" to build a new resource DLL.

What I like is that it doesn't really impact day-to-day development, and it doesn't require a recompile of the main application to add additional languages.

I'll do another blog shortly that extends this to include the localization of strings that are used within code (e.g., exception messages etc.)

Apple drop iPhone NDA

noreply@blogger.com (Steve) — Wed, 01 Oct 2008 20:18:00 +0000

It appears that Cupertino does listen to what's going on in the wild...

http://www.appleinsider.com/articles/08/10/01/apple_drops_iphone_nda_for_released_software.html

It's only for for released software, but considerably better than the previous position.

There's a new Process Monitor

noreply@blogger.com (Steve) — Wed, 01 Oct 2008 07:38:00 +0000

It's even had a major version number change - you can get Process Monitor V2.0 here.

CompiledQuery and Enumerating Query Results

noreply@blogger.com (Steve) — Mon, 29 Sep 2008 21:50:00 +0000

More fun with compiled queries, this time when processing the results. Firstly, the simple non-compiled query:

using (DataClasses1DataContext context = new DataClasses1DataContext())

   var results = context.Employees.Where(e => e.EmployeeID == 1);

   Console.WriteLine("Number of employees: {0}", results.Count());

   Console.WriteLine("First ID: {0}", results.First().EmployeeID);

Simple stuff, and it works as you'd expect. We get both the number of employees and the Id of the first one. Note that under the covers, SQL gets executed twice. Perhaps not quite what you'd expect ;)

Here's the compiled version:

var compiledQuery = CompiledQuery.Compile((DataClasses1DataContext context) => context.Employees.Where(e => e.EmployeeID == 1));

using (DataClasses1DataContext context = new DataClasses1DataContext())

   var results = compiledQuery(context);

   Console.WriteLine("Number of employees: {0}", results.Count());

   Console.WriteLine("First ID: {0}", results.First().EmployeeID);

This one doesn't do what you'd expect. On the second Console.WriteLine(), instead of executing a suitable query, you get an InvalidOperationException saying that "The query results cannot be enumerated more than once". It's pretty clear what it means, and the fix is simple - make sure you only enumerate the results once, using something like the ToList() method:

var results = compiledQuery(context).ToList();

With this, you only hit the DB once, and you can then look at the list of results as much as you like.

The difference between the non-compiled and the compiled query is down to how the two different queries are processed. In the non-compiled version, there's an expression tree floating around, which is lazily evaluated when the results are enumerated. Because LINQ to SQL has the expression tree available, it can generate different SQL on each hit, so results.Count() generates a "select count(*) ..." statement and results.First() generates a "select top 1 ..." statement.

For compiled queries, the SQL is determined at the point you call CompiledQuery.Compile(). When the results are enumerated the first time, this pre-prepared SQL is executed and the results processed. Note that since the SQL is already built, the thing that you are doing with the results doesn't influence the SQL. So the call to "results.Count()" will execute a select of the entire dataset, which will then get enumerated and counted in the client.

Since repeatedly issuing the same SQL is unlikely to be what you want your app to be doing, the designers of LINQ to SQL quite wisely throw an exception if you try to do so. Instead, you need to stick in the explicit ToList() to make it clear that you understand the behaviour.

At first glance it seems a shame that you can't just swap normal queries & compiled queries but hopefully you can see that the semantics are quite different between the two, at which point having different client code which respects these differences is more acceptable.

Last point - before you start getting excited about the number of SQL queries that the non-compiled version may be executing, and start scattering ToList() calls everywhere, make sure that you understand the consequences. For the code above, ToList() is going to be a good thing, but if you consider a scenario where the Where() clause doesn't identity a single entry but instead hits perhaps many thousands of rows, then it may not be so good. The first time we look at the results, we just want the count which gets mapped to count() in SQL, and on the second time the First() method gets mapped to a TOP 1 clause in the SQL. Although it would mean two hits on the database, it would likely be far better to do that than to bring back thousands of records for processing in the client.

As with most abstraction layers, LINQ offers a lot of benefits but it cannot be used without considered thought as to what is happening underneath. I thoroughly recommend that when testing your LINQ queries you have the SQL profiler running so that you can see what's going on. Also, don't forget the DataContext.Log property which lets you dump the SQL out to a TextWriter. Using this, it would be quite possible to check within your unit tests that the DB interaction is running the way you expect, and also to spot when changes cause unexpected interactions.

Passing Predicates into Compiled Queries

noreply@blogger.com (Steve) — Mon, 29 Sep 2008 21:17:00 +0000

I've recently been looking at generating LINQ predicates on the fly in a mapping layer between a set of business domain entities and a set of related, but different, database entities. One of the problems that I've encountered is to do with the way in which predicates are handled when using CompiledQueries.

To start off, lets consider the easy non-compiled version. Here's a method:

// Get an employee by a predicate

static void GetEmployee(Expression<Func<Employee, bool>> predicate)

   // Just perform the select, and output the results

   using (DataClasses1DataContext context = new DataClasses1DataContext())

      var results = context.Employees.Where(predicate);

      Console.WriteLine("Number of employees: {0}", results.Count());

The usage of this is nice and simple, and the sort of thing in LINQ examples all over the web:

GetEmployee(e => e.EmployeeID == 1);

Calling this does exactly what you'd expect. My next step was to look at how this approach could be used with compiled queries. I started with a simple method:

// Get an employee by a predicate, using a compiled expression

static void GetEmployeeCompiled(Expression<Func<Employee, bool>> predicate)

   // Compile the query

   var compiledQuery =

      CompiledQuery.Compile((DataClasses1DataContext context) => context.Employees.Where(predicate));

   // and using the compiled query, output the results.  This crashes :(

   using (DataClasses1DataContext context = new DataClasses1DataContext())

      var results = compiledQuery(context);

      Console.WriteLine("Number of employees: {0}", results.Count());

Obviously, this is pointless since it just recompiles the query every time. But let's ignore that small fact - it should, after all, still work. Alas, it doesn't.

At the point where the results are enumerated, it explodes with a "NotSupportedException". Specifically, it fails due to an "Unsupported overload used for query operator 'Where'.". Looking at the expression tree that is being compiled, in conjunction with some help from Reflector to look at what LINQ is doing under the cover, it can be seen that the issue is down to how the predicate is included in the final query expression.

Remember, the compiler is not generating executable code here, it is just building a lambda expression. When it sees the parameter to the Where() method, it has little choice but to be "lift" this variable its own class, and it is a property on this lifted class that is passed as a parameter to the Where() method. Although the non-compiled version handles this just fine, it causes the CompiledQuery object to barf. This is just the same as any other query that uses a local variable or parameter.

I've experimented with a number of ways of constructing the query that I'm trying to compile, but all ultimately end up with the same problem. The solution I've found is a little nasty, but it does work. If I've missed a cleaner way, then I'd love to hear about it!

Anyhow, the solution. It is based on the fact that it is the act of "passing" the predicate into the Where() method that is the problem. So the solution is to not pass in the predicate, but instead pass in some dummy predicate. Then do some expression tree walking to swap out the dummy predicate for the real one. The code looks like this:

// Get an employee by a predicate, using a compiled expression

static void GetEmployeeCompiled2(Expression<Func<Employee, bool>> predicate)

   // Setup the required query, using a dummy predicate (c => true)

   Expression<Func<DataClasses1DataContext, IEnumerable<Employee>>> compiledExpression =

      context => context.Employees.Where(c => true);

   // Dig out the dummy predicate from the expression tree created above

   Expression template = ((UnaryExpression)((MethodCallExpression)(compiledExpression.Body)).Arguments[1]).Operand;

   // Swap out the template for the predicate

   compiledExpression = (Expression<Func<DataClasses1DataContext, IEnumerable<Employee>>>)

                                 ExpressionRewriter.Replace(compiledExpression, template, predicate);

   // Compile the query

   var compiledQuery = CompiledQuery.Compile(compiledExpression);

   // and using the compiled query, output the results.  This works :)

   using (DataClasses1DataContext context = new DataClasses1DataContext())

      var results = compiledQuery(context);

      Console.WriteLine("Number of employees: {0}", results.Count());

So the required query is itself stored as an expression, with a dummy predicate (c => true) used to get the correct "shape" of tree. This predicate is then located and the expression tree is rewritten, swapping out the dummy predicate for the real one. This new query expression then compiles and executes just fine.

For completeness, the ExpressionRewriter class is defined as:

class ExpressionRewriter : ExpressionVisitor

   static public Expression Replace(Expression tree, Expression toReplace, Expression replaceWith)

      ExpressionRewriter rewriter = new ExpressionRewriter(toReplace, replaceWith);

      return rewriter.Visit(tree);

   private readonly Expression _toReplace;

   private readonly Expression _replaceWith;

   private ExpressionRewriter(Expression toReplace, Expression replaceWith)

      _toReplace = toReplace;

      _replaceWith = replaceWith;

   protected override Expression Visit(Expression exp)

      if (exp == _toReplace)

         return _replaceWith;

      return base.Visit(exp);

where the ExpressionVisitor base class can be found on MSDN

Unsafe code without the Unsafe keyword

noreply@blogger.com (Steve) — Tue, 23 Sep 2008 12:09:00 +0000

I've been playing around with some code lately that uses dynamic method generation fairly extensively. In the course of doing so, I've written the odd dodgy bit of IL out. Interestingly, a couple of time I got some very strange results when assigning fields from one object to another - specifically, if I got the types mismatched I just got garbage in the destination rather than some form of Cast exception (which I'd expect the runtime to generate during execution) or Verification exception (which I'd expect when I finally surface my generated method through a call to DynamicMethod.CreateDelegate()).

Finally had some time today to take a closer look, and the results are very interesting and not at all clear from the documentation. Specifically, if you create a dynamic method using the following constructor:

public DynamicMethod(

    string name,

    Type returnType,

    Type[] parameterTypes,

    Module m

and pass in "Assembly.GetExecutingAssembly().ManifestModule" for the module, then it appears that all type safety within the generated code is turned off. i.e., you can pretty much assign anything to anything. The following code, for example, enables you to dump the memory address of any reference type:

/// <summary>

/// Return a method that gives the memory address of any object

/// </summary>

static Func<object, int> Get_GetAddress_Method()

   DynamicMethod d = new DynamicMethod("", typeof (int), new Type[] {typeof (Object)},

                                       Assembly.GetExecutingAssembly().ManifestModule);

   ILGenerator ilGen = d.GetILGenerator();

   ilGen.Emit(OpCodes.Ldarg_0); // Load arg_0 onto the stack (of type object)

   ilGen.Emit(OpCodes.Ret);     // And return - note that the return type is an int...

   return (Func<object, int>)d.CreateDelegate(typeof(Func<object, int>));

You can use this in the following way:

Func<object, int> getAddress = Get_GetAddress_Method();

const string greeting = "Hello";

// Get the address of the "Hello" string

int x = getAddress(greeting);

x now contains the memory address of the string "Hello". So what? Well, you can also write a method like this:

/// <summary>

/// Return a method that "maps" any type to a particular memory location

/// </summary>

static Func<int, T> Get_ObjectAtAddress_Method<T>()

   DynamicMethod d = new DynamicMethod("", typeof (T), new Type[] {typeof (int)},

                                       Assembly.GetExecutingAssembly().ManifestModule);

   ILGenerator ilGen = d.GetILGenerator();

   ilGen.Emit(OpCodes.Ldarg_0);  // Load arg_0 onto the stack (of type int)

   ilGen.Emit(OpCodes.Ret);      // And return - note that the return type is T

   return (Func<int, T>)d.CreateDelegate(typeof(Func<int, T>));

This chap lets you take any memory address, and "pretend" that an object of type T resides there. So you can do something like this:

Func<int, byte[]> getData = Get_ObjectAtAddress_Method<byte[]>();

// Get a byte array on the same location

byte[] data = getData(x);

where x is a memory location that you've acquired previously. It doesn't matter if the type that really resides at address x is a byte[] or not. This basically lets you get access to the whole address space within your AppDomain (and possibly the whole Win32 process) and write whatever you like into it.

This seems plain wrong to me - I haven't specified the "unsafe" keyword anywhere, nor is this code built with the "Allow unsafe code" box checked. Without jumping through those hoops, I should not be able to write code like this. I'll concede that this only works in a full trust environment, but it still smells like a very serious hole in the type safety of .Net. Interestingly, if you use the DynamicMethod constructor that doesn't take a Module parameter, then everything works as you'd expect - you are politely served a VerficationException when you try to compile the method. According to the docs, the constructor overload that takes a module is only supposed to allow access to internals of the specified module, not to skip type safety. I wonder if the implementation of DynamicMethod in that scenario is flawed.

Below is a big lump of code - it compiles and shows the issue quite clearly. I'd be interested in your views on whether this is a bug or "by design". If the latter, what exactly was the scenario that they were designing for?

using System;

using System.Reflection;

using System.Reflection.Emit;

using System.Text;

namespace ConsoleApplication1

   class Program

      static void Main()

         // Get some methods generated...

         Func<object, int> getAddress = Get_GetAddress_Method();

         Func<int, byte[]> getData = Get_ObjectAtAddress_Method<byte[]>();

         const string greeting = "Hello";

         // Print the greeting

         Console.WriteLine(greeting);

         // Get the address of the "Hello" string

         int x = getAddress(greeting);

         // Get a byte array on the same location

         byte[] data = getData(x);

         // Change some data...

         SetString("Bye!!", data);

         // And display the greeting again (remember, strings are immutable...)

         Console.WriteLine(greeting);

         // And just to show it against other bits of the framework...

         Console.WriteLine(Assembly.GetExecutingAssembly().FullName);

         SetString("Hacked!", getData(getAddress(Assembly.GetExecutingAssembly().FullName)));

         Console.WriteLine(Assembly.GetExecutingAssembly().FullName);

      /// <summary>

      /// Return a method that gives the memory address of any object

      /// </summary>

      static Func<object, int> Get_GetAddress_Method()

         DynamicMethod d = new DynamicMethod("", typeof (int), new Type[] {typeof (Object)},

                                             Assembly.GetExecutingAssembly().ManifestModule);

         ILGenerator ilGen = d.GetILGenerator();

         ilGen.Emit(OpCodes.Ldarg_0); // Load arg_0 onto the stack (of type object)

         ilGen.Emit(OpCodes.Ret);     // And return - note that the return type is an int...

         return (Func<object, int>)d.CreateDelegate(typeof(Func<object, int>));

      /// <summary>

      /// Return a method that "maps" any type to a particular memory location

      /// </summary>

      static Func<int, T> Get_ObjectAtAddress_Method<T>()

         DynamicMethod d = new DynamicMethod("", typeof (T), new Type[] {typeof (int)},

                                             Assembly.GetExecutingAssembly().ManifestModule);

         ILGenerator ilGen = d.GetILGenerator();

         ilGen.Emit(OpCodes.Ldarg_0);  // Load arg_0 onto the stack (of type int)

         ilGen.Emit(OpCodes.Ret);      // And return - note that the return type is T

         return (Func<int, T>)d.CreateDelegate(typeof(Func<int, T>));

      /// <summary>

      /// Little helper method to copy a string into a byte[]

      /// </summary>

      static void SetString(string requiredString, byte[] dest)

         UnicodeEncoding encoder = new UnicodeEncoding();

         byte[] requiredBytes = encoder.GetBytes(requiredString);

         // Need to do the copy by hand, since Array.Copy bleats

         // about the dimensions of the destination.  No surprise really,

         // since the destination isn't really an array...

         for (int i = 0; i < requiredBytes.Length; i++)

            dest[i] = requiredBytes[i];

Debugging Services

noreply@blogger.com (Steve) — Mon, 15 Sep 2008 15:59:00 +0000

We all know the problem - to debug a program (particularly if it's the startup procedures that you need to look at), you simply load the solution in Visual Studio and hit F5. Except if it's a service. With a service, you can't just run it but instead it needs to be launched via the Service Control Manager, which makes debugging its startup a real pain.

The solutions that I've used in the past have either been to have a command line option to enable the process to launch as a regular process as opposed to being control by the SCM, or to have it sleep for a number of seconds when it is launched. Either of these gives me a route to get a debugger attached before anything "interesting" happens. But both of these also mean I've got code present that isn't going to be in the live, and I much prefer to be debugging "the real thing" rather than some (albeit close) approximation.

But there's a third way, which I'd not heard of before - there is a registry setting for "Image File Execution Options" which, amongst other things, allows you to specify that when an app is launched (via CreateProcess(), which covers most scenarios), instead of just firing up the exe it instead runs the required debugger. Nice :)

For more details, check out this blog on the subject. There's also this entry which describes a few more of the settings that are available.

Naming Tests

noreply@blogger.com (Steve) — Mon, 08 Sep 2008 18:56:00 +0000

Following on from Hadi's post, I found this recent blog on the same topic. It says pretty much the same thing, and acts as a good re-enforcement to the general point, which is to give your tests good names so that the next chap who looks at the code understands what the code being tested is supposed to do. It's such a valuable addition to the project documentation.

Exactly which form you choose isn't really that important, providing that it is both descriptive and consistent. What do you use for your test names?

More on the Chrome EULA

noreply@blogger.com (Steve) — Thu, 04 Sep 2008 09:24:00 +0000

Google obviously read my blog^* and caved in without a fight. Clause 11 in the EULA has now been changed to:

11. Content license from you

11.1 You retain copyright and any other rights you already hold in Content which you submit, post or display on or through, the Services.

Much better.

^* Ok, perhaps it wasn't just my blog that did it :)

Chrome EULA

noreply@blogger.com (Steve) — Wed, 03 Sep 2008 13:13:00 +0000

There's an interesting clause in the EULA for Chrome:

11.1 You retain copyright and any other rights you already hold in Content which you submit, post or display on or through, the Services. By submitting, posting or displaying the content you give Google a perpetual, irrevocable, worldwide, royalty-free, and non-exclusive license to reproduce, adapt, modify, translate, publish, publicly perform, publicly display and distribute any Content which you submit, post or display on or through, the Services. This license is for the sole purpose of enabling Google to display, distribute and promote the Services and may be revoked for certain Services as defined in the Additional Terms of those Services.

where Services is defined as:

1.1 Your use of Google’s products, software, services and web sites (referred to collectively as the “Services” in this document and excluding any services provided to you by Google under a separate written agreement) is subject to the terms of a legal agreement between you and Google. “Google” means Google Inc., whose principal place of business is at 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States. This document explains how the agreement is made up, and sets out some of the terms of that agreement

Chrome is, I believe, a Google product, and so falls into the definition of Services. Hence, according to the EULA, Google can do pretty much anything with any information that you "submit, post or display". I suspect that this is a mistake on their part, and that they've just cut'n'pasted a little to much from the EULA's for some of their other services. It's certainly at odds with the privacy policy for Chrome.

However, if you're worried about such things, then I'd suggest that you don't use Chrome for anything sensitive. Note that this licence only applies to the executable installation; if you download the source and build it yourself then you are covered by a regular Open Source licence. It'll be interesting to see how long it takes them to re-word this...

U.S. Employer Identification Number (EIN)

noreply@blogger.com (Steve) — Wed, 03 Sep 2008 11:32:00 +0000

Any non-US company that wishes to sell iPhone applications in the US via the AppStore needs to obtain an Employer Identification Number to complete the contract details with Apple. The Apple site is not particularly helpful with regard to how this is achieved, and simply points you at the IRS Form SS-4 PDF. You can fill this out and send it off, but my understanding is that if you do so it will be several weeks before you receive your EIN.

Alternatively, try dialling this number: +1-215-516-6999 - I just did, and after a few minutes with a very helpful chap I received my EIN. Fast and easy, just how I like it :)

Google Chrome

noreply@blogger.com (Steve) — Wed, 03 Sep 2008 08:10:00 +0000

No doubt this is old news already, but for those that haven't seen it there's a cartoon strip that describes the new google browser here (although it's an odd format, it's actually quite a good read). The browser itself is available for download here. No Mac or Linux support yet, but supposedly that's on the way.

My first impressions are pretty good - it's seems to be fairly snappy and correctly renders most of the pages that I use. I like the UI - it's nicely un-cluttered, and having the tabs right at the top seems to work well.

Pseudo-Predicates in Specifications

noreply@blogger.com (Steve) — Tue, 02 Sep 2008 11:17:00 +0000

Following on from my previous blog entry in which I asserted that I don't normally bother just referencing other people's blog, here's another reference :)

It's quite a long article, but well written and definitely makes it's point. To anyone writing specifications, and to anyone reading them (which I think covers just about everyone who's reading this!), it's well worth a read:

Tasty Beverages

It's talking about predicates in the context of security, but I think the lesson is actually broader than that - I don't think it's all that unusual to see pseudo-predicates in pretty much any form of specification, and the danger with them is that the human brain is pretty adept at filling in what it thinks is missing (hence, they can be hard to spot). Alas, what the brain makes up isn't always what the author of the spec was thinking, leading to the wrong thing being developed.

Enable SSH on ESXi

noreply@blogger.com (Steve) — Fri, 29 Aug 2008 15:24:00 +0000

Don't normally bother writing blogs that just reference another blog, but this one gem me a while to find, so if only for my own reference I'm copying it here:

Go to the ESXi console and press alt+F1

Type: unsupported

Enter the root password

At the prompt type “vi /etc/inetd.conf”

Look for the line that starts with “#ssh” (you can search with pressing “/”)

Remove the “#” (press the “x” if the cursor is on the character)

Save “/etc/inetd.conf” by typing “:wq!”

Restart the management service “/sbin/services.sh restart”

Original entry was here, nice one :)

Targeting .Net 2.0 from VS2008

noreply@blogger.com (Steve) — Thu, 28 Aug 2008 10:02:00 +0000

Following on from the previous post about old assemblies being updated with the latest .Net 3.5 service pack, I thought it also worth mentioning the other gotcha that can happen when targeting old versions of .Net from VS2008.

If you have a project that is set to target .Net 2.0, VS will dutifully only allow you to add 2.0 assemblies to the project, ensuring that you don't accidentally make use of 3.0 or 3.5 framework features that perhaps aren't installed on the machines of your target audience. However, VS will still use the latest compiler to compile the code, hence you can make use of 3.0 language features such as var, lambdas etc. These work, even on a box with just the 2.0 framework installed, because they are all handled via compiler magic - there was no change to the resulting IL to support the new 3.0 language features.

Now this can either be a good or a bad thing. Good, because you can start making use of the new language tools without requiring your customer to upgrade anything. Why bad? Three reasons I can think of:

For a big project, switching compilers is not a simple decision. Your app has all been fully tested, both internally and through man-years of usage in the field, but by switching the compiler, you have (potentially) just changed every line of code. The only sane thing to do would be a full regression test of the whole system - who's to say that the new 3.0 compiler doesn't have some bugs in it, or (equally likely, I think), it fixes some bugs that were present in the 2.0 compiler, but which your code unknowingly relied on to function correctly.
If you start using 3.0 features on your dev machines, don't forget that you'll also need to upgrade your CI build servers. If they are only building your project, that might be an easy choice. If they are shared by several teams, that may be more difficult.
If you ship code to the customer rather than binaries, you need to ensure that your customer is also happy to upgrade from VS2005 to VS2008, otherwise they won't be able to build your code anymore.

What can you do? Well, if you're in the "it's a good thing" camp, then just get coding. If the bad things are important to you, then the bad news is that there's not much you can do on your development machines to help (other than go back to VS2005!). What you can do is to ensure that on your CI build machines you specify the TargetFramework option to MSBuild. This forces MSBuild to use the appropriate versions of the compiler and other tools. See this blog for more details. That way, if a dev uses some 3.0 language feature, it will cause the CI build to fail and hence get spotted nice and quickly. You don't have a CI server? Shame on you.

noreply@blogger.com (Steve) — Wed, 27 Aug 2008 14:09:00 +0000

There have been a number of posts about the changes introduced in .Net 3.5 SP1, but this is one I've not seen before. I'm not sure that the title, "breaking changes" is entirely correct, but it probably got your attention :)

In .Net 3.0, there was a new class, System.Collections.ObjectModel.ObservableCollection<T> (I'm sure you can guess what it does). It had two constructors, ObservableCollection<T>() and ObservableCollection<T>(List<T>). Clearly the guy who wrote it was having a bad day - List<T> as a constructor parameter? What's all that about?

This stayed the same in .Net 3.5, but changed with .Net 3.5 SP1. Now, ObservableCollection<T> has an additional constructor, ObservableCollection<T>(IEnumerable<T>) - i.e., the one that they should have had in the first place instead of List<T>. All is good in the world.

Now there's a nice feature in VS2008 whereby you can indicate in the project properties which version of .Net you are building for. This means that VS makes sane choices as to which assemblies you can reference, meaning that the code you write should run just fine on a machine that's only got old versions of the framework, even if you've got all the latest spanky stuff installed. (There's another issue to do with the compiler, but I'll cover that in another blog).

Here's the problem. If you have .Net 3.5 SP1 installed, and you create a project that targets 3.0, VS will happily only add 3.0 references. Including the reference to the assembly (WindowsBase, if you care) that contains ObservableCollection<T>. Which SP1 upgraded. So you can write code that takes advantage of the new constructor, such as:

ObservableCollection<int> x = new ObservableCollection<int>(new int[] { 1, 2 });

which will compile and run just fine.

When you finally complete all your coding and have all your tests passing, you pass it to your customer confident in the knowledge that you've done a good job. He installs and runs the code, and the very first impression he gets of your work is a MethodNotFound exception, since you're calling a constructor that he doesn't have on his machine.

So what do you do? There's not much you can do on your development system, other than be careful, which isn't much help. What you can, and should, do is to make sure that your build server and test environments exactly match the minimum that you expect your code to be running on. That way, a problem like the one above will get spotted during the development process where it can be easily fixed. Note that it's probably a good idea to turn off automatic updates on build & test machines, otherwise you'll probably end up getting stuff pushed down that you didn't want.

[Update]

Just found out that FxCop has a rule to detect and warn on the usage of such methods - see Brad Abrams blog posting on the subject here. That helps no end :)

Default Gateway of 0.0.0.0 in Vista

noreply@blogger.com (Steve) — Mon, 14 Jul 2008 12:53:00 +0000

Strange one just occurred - woke up my machine from sleep, and plugged in the various cables in no particular order. Everything looked good, except no network access. Could ping the local network (including the gateway), but nothing outside of that. Everyone else on the same subnet was seeing external IPs just fine.

A quick investigation with IPConfig showed that I had two default gateways, once with the correct address but another with an address of 0.0.0.0. And it was the one that was first in the list.

No idea where it came from - Vista does use that address if you are using dial-up networking, but I wasn't (although my mobile was plugged in through USB to charge it, so perhaps it is related in someway). Anyhow - that was clearly the problem, so it was just a case of getting rid of the rogue gateway address.

Tried "ipconfig /renew". Tried "ipconfig /release" then "ipconfig /renew". Tried unplugging network cables, mobile phones etc. No luck. Reboot was looking likely. Last attempt was to use netsh. Success :)

For those of you who may get stuck in a similar situation, here's the netsh rune that you need to cast:

netsh interface ipv4 delete address "Local Area Connection"
addr=a.b.c.d gateway=0.0.0.0

where a.b.c.d is your local IP address. netsh can do just about anything to your local network config - well worth some exploration if you're *really* bored.