Being an advocate of Continuous Delivery, I wanted to setup a pipeline for my software and I needed a consistent development and production environment and I didn’t like the idea of the repetitive configuration process every time I wanted to shut down, start up or create a new node. It isn’t too much effort to manually configure a single node but as I was being charged for my compute usage whilst the systems were running, why would I want to keep them running when I didn’t actually need them? The other Continuous Delivery principle at work here is that or Automating everything; after all, a manual process can never be described as reliably repeatable.

To this end, there are two main competing open-source automation systems in this space: Opscode Chef and Puppet. Both are primarily written in and reliant upon Ruby (and also annoyingly difficult to effectively google!). The primary different between them is that:

• Chef describes systems in an imperative fashion.
• Puppet describes systems in a declarative fashion.

I have previously experimented with Chef so decided to setup and configure my systems with puppet. It is worth reading the puppet introduction as well as the tutorials available on puppet labs.

I opted to use puppet masterless as I didn’t want to maintain a puppetmater instance. My second iteration of my Rackspace cloud infrastructure will include a puppetmaster instance, but I always feel you should learn to walk before you run!

First, I installed puppet onto my Ubuntu node. In this instance I will opt to install puppet and then create a handy snapshot, which I will be able to clone later to add a pre-puppet client’d node. If I can create a snapshot then why not just create a snapshot of my configured system and use that? The short answer is that snapshots are harder to maintain if you’re tweaking and re-configuring them.

I then created a directory with two sub directories:

-- jamiei-puppet
--- manifests
---- site.pp
--- modules

site.pp is where our node description lies. This is our puppet manifest, a basic unit of execution which can reference resources, other manifests and generally manage aspects or our system. A basic manifest, creating a single file, might look like this:

    file {'helloworld':
      path    => '/tmp/helloworld',
      ensure  => present,
      mode    => 0640,
      content => "Hello, world",
    }

We can test this first by moving to our jamiei-puppet directory and running:

puppet apply --noop manifests/example.pp

The –noop flag tells puppet to not actually make the changes, so my output looks something like this:

notice: /Stage[main]//File[helloworld]/ensure: is absent, should be present (noop)

Nicely informing me that it couldn’t find our file and so it would have created it.

We might only want some of our nodes to be decorated with our magical greeting message but not to duplicate our code across lots of different roles. Puppet manifests can contain what are essentially inherited roles:

node base_jamiei_box {
}

node default inherits base_jamiei_box {
}

node 'puppet-test.jamiei.com' inherits default {
    file {'helloworld':
      path    => '/tmp/helloworld',
      ensure  => present,
      mode    => 0640,
      content => "Hello, world",
    }
}

As you might be able to see from the above, there is an inheritance chain which allows me to layer my roles for a particular box accordingly, culminating with our helloworld file being applied only to this node only.

Puppet allows blocks of resource manipulation to be bundled into modules. For example, I quite like my machines to be provisioned with ntp so that my servers are all synchronised. To this end, I could include an ntp module in my default role. To do this, I created a subdirectory in the modules folder called ntp, with another subfolder called manifests and create a file called site.pp within it (site.pp is a convention). This file contains my class name and the declarations for ensuring that the ntp package is installed, before ensuring that the service is running:

class ntp {
  package { "ntp": ensure => installed }
  service { "ntpd": ensure => running }
}

If the package isn’t installed, it will be and if the service isn’t running then it will be started. I can then include this into a node’s configuration by simply include ntp ‘ing. In this way, I can build up my nodes configuration. I can inspect puppet’s view of my node via the puppet resource command. For example, the manifest below creates me a user on the box and then adds my authorised key to the box so that I can login.

node base_jamiei_box {
    include ntp
    include base-dev-packages
}

node default inherits base_jamiei_box {

  user { 'jamie':
    shell => '/bin/bash',
    uid => '1000',
    ensure => 'present',
    gid => '1000',
    groups => ['sudo'],
    home => '/home/jamie',
    comment => 'Jamie,-,-,-,-'
  }
  ssh_authorized_key { "jamie":
    ensure => present,
    key => "INSERT PUBLIC KEY HERE",
    name => "no@spam.com",
    user => 'jamie',
    type => rsa,
  }

}

node 'puppet-test.jamiei.com' inherits default {
    file {'helloworld':
      path    => '/tmp/helloworld',
      ensure  => present,
      mode    => 0640,
      content => "Hello, world",
    }
}

It’s never particularly sensible to re-invent the wheel needlessly and it should be no great surprise that there is a strong community which have created modules for almost every conceivable task. You can (and probably should) make use of one of the many puppet modules which are available on the web. For my own project, I made use of these modules:

• puppet-redis
• puppet-iptables
• puppet-module-nginx

You can include a top level root module directory in masterless mode by using the –modulepath command line option:

puppet apply --noop --modulepath modules manifests/example.pp

It goes without saying that your puppet manifests should be versioned, I prefer git for this purpose. It should be fairly apparent to you now how puppet might ensure that your environment is consistent and also to automatically fashion a new node into your preferred state by simply cloning my masterless recipe and then running puppet apply.

Further Reading

• Puppet on Ubuntu – Installing and configuring puppet on Ubuntu.
• Learning Puppet – The official documentation.
• Puppet at Opera – Puppet custom facts and masterless usage at Opera software.
• Puppet Resources – Puppet resources documentation.
• Puppet Examples – Puppet examples on github.

The challenge, to summarise, is to devise an application which can search through very large log files for lines which match the time (the name of the challenge will give away the purpose for those familiar with the grep tool). The challenge focuses specifically on HAProxy logs and although they’re not hard to generate yourself they do provide a sample perl generator. The main behaviour of the application is that a user should be able to provide either a time or time range and optionally a log path (with a default) and have the lines within the scope of the log file returned:

$ tgrep 8:42:04 

[log lines with that precise timestamp] 

$ tgrep 10:01 

[log lines with timestamps between 10:01:00 and 10:01:59] 

$ tgrep 23:59-0:03 

[log lines between 23:59:00 and 0:03:59]

For the purpose of this challenge, the log files can be assumed to have been rotated every 24 hours and you will therefore only need to worry about the time of a line and not a date changeover.

A successful application in the strict interpretation of this challenge would also need to work on Linux and would therefore likely need to be compiled with Freepascal.

What I like about this challenge is that there are many levels of solution which you could dive into. Clearly, as the challenge states, you are unlikely to achieve any real speed when searching through log files of many gigabytes if you perform a linear search. The size of the files involved will also cause problems any solution which isn’t careful about memory consumption.

For my first iteration I initially opted for a straightforward interpolative search, which turned out to be much more buggy than I’d originally thought however luckily, I had written my tests first even though I had to resort to a form of integration testing instead of using mock objects (Nick Hodges would not be happy!). It’s a shame that none of the current Mocking libraries appear to be compatible with Delphi 2009 so I thought I’d leave the mocks until a later iteration. My initial iteration works and although I haven’t performance tested it fully yet, I fully expect it to pale in comparison with some of the excellent tgrep entries made available via github. Of particular note is EvansBee’s C version, which in the few tests that I ran was very quick.

For my second iteration, I expect that it can be optimised in real-world conditions by taking account of the peaks and troughs in traffic that I would imagine reddit get during certain intervals of the day.

If anyone feels like they might like to give the tgrep challenge a go with some absurdly clever way of tackling this problem in delphi/fpc then feel free to comment/get in contact. It might be fun to put them all up on github so that we can all compare and contrast. If someone was really clever then they might even be able to hookup some kind of clever super project which builds a bunch of git submodules and tests them all.

Links

• tgrep challenge – the original job ad, explaining the details.
• A perl log sample generator – It is perl, but it works.
• github tgrep projects – Some of these contain some interesting ideas, if you’re stuck for an approach.

Holy cow, all those flashy words in a single description, that must mean this is some enterprisey buzzwordy new fangled thing, right? Nope. This is an extension of all those ideas that we know that we “should” and “could” have always been doing, and in fact some of you may already be doing it.

Continuous Delivery, in a nutshell, is about delivering your software in small, very frequent incremental updates through a huge quantity and quality of automation in build, testing and deployment. Traditionally, the release process for many teams culminates in a lockdown period for release followed by many hours or days of manual release process, followed by triage and observation. This is a broken and problematic approach and we all know it.

Instead, continuous delivery is the way forwards for delivering software and in future blog posts I will aiming to cover more of the implementation process, potential problems and cultural impact.

My interest was piqued when I attended a presentation by one of the authors of Continuous Delivery, which as the name would suggest is the currently definitive book on the subject. David and Jez developed the principles for Continuous Delivery at Thoughtworks, where Jez currently also works as the Product manager for Go (yes, neither this Go, nor this Go).

Continuous Delivery is about automating everything from build to deployment into production. This also means that methodologies such as Continuous Integration and Agile testing could be viewed as a specific subset of Continuous Delivery. I also recently attended ‘Evolving Continuous Delivery‘ with the London Continuous Integration Meetup where Chris Read gave us the quote of the night:

Until your code is in production making money or doing what it is meant to do, you have simply wasted your time

The Value proposition for Continuous Delivery is very simple: getting business value into production as quickly as possible. By repeatedly deploying to production in a controlled manner we also:

• Validate the business decisions which we’ve made more quickly by taking small steps towards our goal and gaining feedback before you’ve invested the full cost.
• Allow us to deliver our business value at lower risk because we make smaller changes at each step. Smaller changes are less risky than large packages of change.
• Encourages the culture that nothing is truly “done” until it is delivered to your users and more importantly proven before delivery.

Every build artefact should be considered a potentially releasable artefact, your comprehensive suite of tests are there to prove that it is not suitable. With each test passed, the build should be promoted through the pipeline to the next test. In order to be confident that your release is not risky, you need to be able to test everything in a release. Obviously, this means testing code changes, but also deployment processes and configuration. Being certain of your configuration also means testing your system configuration as well as your application configuration. We’ve all seen how “minor” OS patches can cause massive knock on effects to the performance or stability of an application unknowingly, it’s vital that you should be able to monitor and roll back. When you begin to consider what conditions you may want to roll back or draw attention to your release, you may need add specific monitoring and instrumentation to your application to monitor if you notice:

• Increased Latency
• Increased Load
• Lost Dependencies after a configuration change
• Poorer Business Performance
• Failed Basic Functionality of the application

Finally, deploying continuously may require cultural and process changes to the way you develop your products. In particular, your SCM and branching strategy may need to reflect the need to release partially implemented features as disabled (also known as feature toggles).  A presentation worth watching on this topic is Chuck Rossi on ‘How facebook releases software‘. Your application architecture may also need to be updated to allow continuous deployment in a safe manner.

Continuous Delivery isn’t just for Web Applications (although it, clearly has massive benefits to them) but as the Google Chrome team have demonstrated, can provide great benefits for Desktop applications (even Delphi ones!) too although requires different tools and approaches. Imagine being able to push a series of very efficient diffs to all the users of your application every time it successfully passes a full range of exhaustive tests.

Many companies currently use some form of Continuous Deployment to manage their operations, such as Flickr, Etsy, Netflix and more who I’ve forgotten. Continuous Delivery is a topic that impacts all areas of operations, development, marketing, and possibly any regulatory concerns. I intend on diving into detailed posts of various areas of the overall topic.

Further Reading

• Continuous Delivery by Jez Humble and Dave Farley – The Book
• Jez Humble at DevOps Days 2010 – A presentation by Jez on Continuous Delivery
• Push the Button – A recent talk by Sam Newman, a principle consultant at Thoughtworks on Continuous Delivery.
• The Value Proposition – An article about the value proposition on Continuous Delivery
• Continuous Delivery using Build Pipelines and Ant – A great end to end article on the Build Tools side by James Betteley, who I met at the London CI meetup.

I would like to thank my good friend Kingsley Davies for encouraging me to pickup this vein of blogging.

I had hosted my Delphi headers for the yajl json library on Github at Delphi-yajl, which were sadly in a broken state as I was unable to solve some of my conversion problems myself. I recently received a pull request on Github for my project from the ever active Delphi’er on github Jason T Penny. He very kindly forked my Delphi-yajl repo and submitted a set of changes which fixed up my botched conversion job and pushed it back. As a result, I’m pleased to announce that the Delphi yajl code is in working order and I now plan on adding a higher level wrapper which allows slightly more Delphi-like usage but if you need lightning fast SAX-style json parsing then feel free to try it out. I can imagine that it would be perfect for situations where you need to parse a stream of json, such as if you were streaming the Twitter Streaming API.

He also has some pretty cool other repos on his Github profile, including his GitForDelphi Project. GitForDelphi is a conversion of the libgit2 headers. libgit2 is portable, pure C implementation of the core git methods which allow you to effectively implement git functionality without having to call git command line (much cooler). The Delphi bindings are in a working state already and in order to return the favour, I do plan on helping him out by contributing some tests. I would encourage all Delphi developers to have a hunt around on Github as there are some interesting little gems to be found and it’s a very cool community to explore.

Further Reading

• An introduction to Git -The Github introduction to Git.
• A Bootcamp guide to Gtihub – The best guide to getting involved to github.
• libgit2 – The amazing native linkable implementation of git.
• Delphi-yajl – Delphi header translations for yajl
• yajl – The yajl (Yet Another Json Library) home.

The F# compiler and libraries were also recently released as Open Source under the Apache 2.0 license which should make for some exciting developments in the future, not to mention better support for Mono.

I also found F# a good way to understand the slightly functional constructs that appeared in Delphi Prism 2011. I’ve only just started beginning to learn F# but I thought I would highlight some of the particularly interesting differences that I’ve found compared to Delphi.

Identifiers and let bindings

Identifiers are the way which you assign names to your values so that you can reference them later, using the let keyword. To a Delphi programmer this may sound like the concept of variables, however there is one important difference in F#: Variables are immutable and therefore normally can’t be changed and once a value has been assigned to it. Consider the following:

open System

let x = 1
printfn "%d" x

let x = 2
printfn "%d" x

This will produce a compiler error where we attempt to reassign a new value to the identifier x. F# shares the type inference approach of C# 3.0 onwards so you do not need to explicitly declare the types of your variables as you do in Delphi (while Delphi Prism shares the C# approach). It should also be noted that as in Delphi, you can also assign identifiers to methods. For example, in Delphi you might do something like the following:

var
  i: integer;
  myAdder: TFunc<integer, integer, integer>;
begin
  myAdder :=
    function(a, b: integer): integer
    begin
      result := a + b;
    end;
  i := myAdder(1, 2);
end;

The equivalent F# code might look like this where F# Functions are values in their own right anyway:

// Could also be written in longform as:
// let myAdder = fun a b -> a + b

let myAdder a b = a + b
let i = myAdder 1 2

If you’re coming from Delphi, it is also worth noting that identifier names are case-sensitive.

Scope

F# defines scope, by default, using whitespace and indentation creates new scope, and the end of that particular scope is signaled by the end of the indented block. The following example demonstrates this, by reusing the identifier ‘z’ as a midway calculation half way through a block.

let z = 1

let y =
    let z = 3 * 2
    z / 2

printfn "%d" y

This code also demonstrates that like the Ruby language, you do no need to explicitly state a return value for a method. This can be quite freeing in many ways.
It should also be noted that you can redefine identifiers with the let keyword whilst in the scope of a function.

Pattern Matching

Another favoured building block for functional programming is pattern matching. A very simple example of this might be compared to series of case or if statements in Delphi. For example, if we look at an implementation of the FizzBuzz problem in Delphi:

	// if ((i mod 15 = 0)) then
        if ((i mod 3 = 0) and (i mod 5 = 0)) then
        begin
          WriteLn('FizzBuzz');
        end
        else if (i mod 3 = 0) then
        begin
          WriteLn('Fizz');
        end
        else if (i mod 5 = 0) then
        begin
          WriteLn('Buzz');
        end
        else
        begin
          WriteLn(i);
        end;
      end;

And then using pattern matching in F#:

// FizzBuzz Solution - @jamiei
open System

let FizzBuzz() =
    let nums = [1..100]
    let printbuzz x =
        match x with
        // | x when x%15=0 -> printfn("FizzBuzz")
        | x when x%3=0 && x%5=0 -> printfn("FizzBuzz")
        | x when x%3=0 -> printfn("Fizz")
        | x when x%5=0 -> printfn("Buzz")
        | _ -> printfn "%d" x;
    nums |> List.iter(fun x -> printbuzz x)
FizzBuzz()
let _ = Console.ReadLine()

Pattern matching must either be complete (i.e. must match all possible values of an identifier) or incomplete and include a wildcard (“_” in the above example) or the compiler will raise an exception at runtime. The compiler will also emit a warning for rules which will never be matched.

let rand = new Random()
let randNum = rand.Next(0, 100);
let coinToss =
    match randNum % 2 = 0 with
    | true -> "heads"
    //| false -> "tails"
printfn "%A" coinToss

Pattern matching may appear to be a simple tool but it’s an extremely concise way of controlling the flow of an application, particularly to get rid of what would otherwise turn into a spaghetti ball of if statements. Pattern matching can also be active, which means that they allow you to execute a function to determine the match.

Units of measurement

If you were listening carefully to the Podcast of Delphi.org when marc hoffman was talking to Jim McKeeth about the future of Delphi Prism, you’ll remember that he mentioned that this might be coming to Delphi Prism. This represents a pretty nifty way of marking the units of a value which makes you less likely to accidentally make a mistake when combining them with less effort than in Delphi where you would have to declare and create a class for each. The following code in F#, will throw an exception:

[<Measure>]type centimetre
[<Measure>]type inch
let distanceToMyThumb = 1<inch>
let distanceToMyOtherThumb = 2.5<centimetre>
let newVol = distanceToMyThumb + distanceToMyOtherThumb

Summary

F# is a very expressive language and the more I read and use it, the more I find that the expressions and constructs that it allows give me new ways to think about certain problems. It is also great that it is built upon the .NET Framework and can interact fully with the standard .NET libraries. My F# test modules can also interact with and be interacted by any Delphi Prism applications that I need it to. If your curiosity is piqued by functional programming and you are already familiar with the .NET framework then F# is a great way to try your hands.

Further Information

You can find more information on F# here:

• F# MSDN Home
• F# Community Samples
• An Introduction to Functional Programming for .NET Developers

MEF is a new library in .NET Framework 4 and Silverlight 4 that addresses the problem of easily extending and componentising applications by providing a nifty framework. Adding an easy ability to extend your application, whether it be for the purposes of componentising your application or allowing others to extend your application has always been available to Delphi users in the guise of frameworks like RemObjects Hydra, the TMS Plugin Framework or building your own homebrew solution but MEF is very nice framework.

MEF is incredibly powerful and I’ll only really be scratching the surface and introducing it here.

Imagine an application which runs a sequential series of processes on an imaginary input file, not exactly rocket science but this is only a basic introduction after all.

First I begin by defining an Interface in a separate assembly which all my plugins will adhere to:

  IFileAction = public interface
    method ProcessFile(path: String);
    property Name: String read;
    property Version: String read;
  end;

We then write our first plugin, which we’ll conveniently call the MyHelloPlugin. First create a new assembly and reference the base plugin assembly which contains our IFileAction interface and the System.ComponentModel.Composition assembly. We also need to ensure that we reference these in our uses clause of the unit containing our Hello Plugin class. The My Hello Plugin class looks like this:

  [Export(typeof(IFileAction))]
  MyHelloPlugin = public class(IFileAction)
  const
     MYNAME = 'The My Hello Plugin';
     MYVERSION = '1.0.1';
  public
    method ProcessFile(path: String);
    property Name: String read MYNAME;
    property Version: String read MYVERSION;
  end;

This should be pretty self explanatory but we have two properties for our name and version and a ProcessFile method which will be called by our host app, providing it with a path for the file that we’re processing. You’ll notice the Export attribute which denotes what we’re exporting. We’re specifying the type of the export but in many cases you may not even need to do this. MEF has a very powerful ExportMetaData system for providing information about what you’re exporting which is also worth reading up on.

Next, of course, we need to actually consume the plugins in our main application. To manage my collection of plugins, I’ll create a small, basic class to hold and manage them:

  PluginList = class
  public
    [ImportMany(typeof(IFileAction))]
    property FileActions: IList<IFileAction> read write;
    constructor;
  end;

Here I’m using the ImportMany attribute to indicate that I’m expecting to import many IFileAction instances and then providing a list with which to populate (which, as always, you must remember to instantiate before you attempt to fill it!).

Next I’ll load up my Hosting catalog and compose the parts of my application.

  var aggregateCat := new AggregateCatalog();
  // var catalog := new AssemblyCatalog(System.Reflection.Assembly.GetExecutingAssembly());
  // var catalog := new AssemblyCatalog('plugins\MyHelloPlugin.dll');
  var dirCatalog := new DirectoryCatalog('plugins');

  // Add our Catalogs here,
  aggregateCat.Catalogs.Add(dirCatalog);

  var container := new CompositionContainer(aggregateCat);
  // Create our plugin hosting object
  var pluginList := new PluginList();
  // Compose the parts.
  container.ComposeParts(pluginList);

You’ll note that I create an AggregateCatalog which isn’t strictly necessary in this case because I’m only adding one ComposablePartCatalog to it but in other uses this would also allow me to separately load single Assemblies or a collection of DirectoryCatalog’s. I then load the catalog into a composition container before creating my aforementioned PluginList class and finally pulling all the parts together. It really is that simple. You can test what has been loaded using our now populated PluginList.FileActions collection, like this:

  for each plugin: IFileAction in pluginList.FileActions do
  begin
    Console.WriteLine('Loaded ' + plugin.Name + ', version ' + plugin.Version);
  end;

You can now create a new and entirely different plugin assembly (or many, many more) and providing it conforms to our IFileAction interface and is either in our target directory or loaded manually, it will now appear in our list as if by magic.

Once you’ve begun using MEF, you’ll find out how powerful it can be and how easy it makes building modular applications.

If anyone is interested in me uploading the complete project set for this sample then please let me know in the comments below and I can upload it to github. If anyone else has any excellent samples of MEF usage that they can publish we could even start a sample repository.

Further Reading

• The MEF Architecture Overview
• The MEF Programming Guide
• The MSDN System.ComponentModel.Composition Guide

I still believe that participating in these communities and making visible “noise” does credit to the delphi community. Some of you may remember that I extolled the virtues of this point just over a year ago in another post, The Delphi Community: From the outside.

This is all particularly relevant in a month when, the largest community gathering, delphi.non-technical has been filled to the brim with little which might give a potential newcomer to the community any kind of welcome feeling what-so-ever. Also this month a set of stats generated from the GitHub API showed that Delphi was the 39th largest language on there, another online property that I advocated our further involvement with. Without saying that we should be much higher, I would say that I would like to see more Open Source Delphi projects on there. We could also do with more of a vocal presence in many other areas too but I guess this requires organic community growth (allbeit slightly prompted).

The recently unveiled previews of RAD Studio XE may not be the cross-platform upgrade that some people hoped it would be, it does represent the next product version of the Delphi we know and love. If what the Embarcadero staff are saying in the newsgroups of the quality this release is true then it will be a worthy addition to the family (if not as groundbreaking as we’d originally hoped). Indeed I liked the look of the new integrated tools for the Delphi IDE revealed today and will await the next previews but even if that isn’t your cup of tea it still leaves plenty to shout about.

What I’m advocating is that we retain a little perspective and focus on what is in the Delphi XE release and not what isn’t and use this as a conversation starting point to start making more noise about Delphi in “public places” with a view to reinvigorating the community. Which reminds me: what other communities/online properties could the Delphi community benefit from making better use of?

In other news: I’ve also got what I consider to be an exciting Delphi community based side-project which you may all potentially get to see very soon…

Visual Studio 2010 Support

I’ll start with the most obvious change: Delphi Prism now supports Visual Studio 2010 which was only released just over a month ago. The Visual Studio 2010 IDE looks great, feels faster than 2008 and most importantly, in my experience, is very stable.

Visual Studio 2010 with Delphi Prism

The shell is based on WPF and has had considerable work put into allowing it to be flexible and extensible which I understand has allowed a whole host of new plugins to flourish. It’s an extremely minor point but I’ve also found it much better in allowing themes for the code editor, springing forth sites like http://studiostyles.info/. My favourite scheme  is Son of Obsidian which works marvellously for C# code and works reasonably well for Delphi Prism code too. It’s great that we can take advantage of community extras like this, although it would also be great if we could tailor some of these themes specifically for Delphi Prism.

As marc mentioned back on episode #39 of the Podcast at Delphi there are a lot of free minor improvements that come with moving to Visual Studio 2010 (not financially free but that were inherent with the migration).

Other cool stuff includes support for Silverlight 4 and the “Paste C# as Oxygene” feature which allows you to translate code snippets on the fly. This release also includes MonoDevelop which includes support for Mac and Windows.

For, If and Case Expressions

For, if and case I hear you say? Haven’t we had those for as long as we can remember? Not like this we haven’t. Delphi Prism now supports them as expressions. This allows you to do operations like this:

  var sourceList: Array of Integer := [1 ,2 ,4 ,8 ,16 ,32 ,64 ,128];
  var sqList := for each num in sourceList yield num * num;

Now this sample could just as easily be done with a LINQ operation or a separate loop but this is another option available at your disposal and introduces a slightly Functional Programming feel. Delphi Prism now also supports If expressions and my favourite expression addition: case.

These expressions, as with so many language features hold the potential for misuse but they can also make things a little more concise:

method NotifierStr(newMails: integer): string;
begin
  result := 'You have ' + newMails.ToString() + ' new ' +
    case newMails of
      0: 'emails';
      1: 'email';
    else
      'emails';
    end;
end;

Please forgive the horrible string concatenation but I wanted to demonstrate it in this way because I thought it might help make it a little clearer.

Dynamic

Delphi Prism 2011 includes support for the (in some cases) contraversial .NET 4 dynamic type. To quote Scott Hanselman on dynamics: A dynamic is a statically typed dynamic type. Dynamics leave a lot more till run-time and allow a much cleaner interop with the Dynamic Scripting languages built upon the CLR.

For example, the ExpandoObject enables you to add and delete members at runtime, something might be useful when serialising and deserialising JSON or XML where you do not know the spec ahead of time:

dynamic tweetresult := new ExpandoObject();
tweetresult.Id = 1243535345;
tweetresult.Username = 'delphifeeds';

The Delphi Prism 2011 implementation has a required dependency upon a separate assembly, RemObjects.Oxygene.Dynamic.dll, which is freely re-distributable.

For a much better idea of what dynamics are and can be used for, I would refer you to Scott Hanselman’s whirlwind tour of dynamics and the MSDN reference on using dynamics.

Tail Calls

Tail Calls are another feature that could be seen as similar to a Functional Programming Language construct. They are essentially an exit statement followed by a call to a method. They have to be explicitly enabled and disabled for a method using the {$TAILCALL ON} flag. This allows you to code recursive functions, particularly useful for maths based methods. This concept will probably take a bit of getting used to and are something I’d like to cover in subsequent posts.

Tuples

A Tuple is an ordered list of Elements. They can contain 1 to 7 different references and types with an 8th reference being used to reference another Tuple if you need to hold more than 7. They are similar to an array and a class/record but unlike an array can contain more than one type and require considerably less declaration than a class/record. A Tuple in Delphi Prism might be used as follows:

  var telephoneList := new Dictionary<Tuple<string, integer>, String>();
  telephoneList.Add(Tuple.Create('Josh Moore', 1283843), '+0014158343131');
  Console.WriteLine(telephoneList[Tuple.Create('Josh Moore', 1283843)]);

They are particularly useful for returning more than one value from a method where you would otherwise need to explicitly declare a class or record to hold the data otherwise. Tuples are a good way to solve a specific problem and will be more familiar to those who have used them in other languages such as the Tuple support in Python.

Parallel Loops (not actually a new feature)

Parallel loops have been a part of the Delphi Prism (and formerly Oxygene) language for a long, long time. However what I hadn’t appreciated was that the Parallel Task Library upon which it relied was officially bundled into the .NET 4.0 Framework libraries. As such I felt it necessary to re-raise it and show just how nifty it is here even though it isn’t really new to Delphi Prism 2011.

Just to recap, with the parallel keyword you can parallelise a loop.

As a fully fledged .NET 4 language we can also use the AsParallel() method in order to return a ParallelEnumerable source capable of being queried with the .NET Parallel LINQ library.

The rough and ready example below demonstrate these two techniques:

class method ConsoleApp.Main(args: array of string);
begin
  var anagramletters := 'mite';
  var filepath := 'mydictionary.txt';

  var dictList := new List<string>();
  dictList.AddRange(System.IO.File.ReadAllLines(filepath));
  Console.WriteLine('Added ' + dictList.Count + ' lines to the file');

  var sortedList := new Dictionary<string, string>();

  for parallel i: Integer := 0 to dictList.Count - 1 do
  begin
    sortedList[dictList.Item[i]] := dictList[i].Sort();
  end;

  var matchingwords := sortedList.AsParallel().Where(w -> ((w.Value.Length = anagramletters.Length) and (w.Value = anagramletters.Sort()))).Select(w -> w.Key);

  Console.WriteLine('Found: ' + matchingwords.Count().ToString() + ' results');
  matchingwords.ToList().ForEach(x -> Console.WriteLine(x));

  Console.ReadLine;

end;

The code above reads a dictionary file in and then builds an index using a parallel loop before finding the words that are anagrams of the provided string using a Parallel LINQ query.

Extension Methods

I consider it a slightly more minor point as I’ve already briefly talked about the new method of defining Extension Methods, but this is now baked into Delphi Prism 2011 and is a pretty useful tool to have. I used the new Extension Methods syntax in the Parallel Example above to add an implementation specific Sort() method to my strings like so:

interface

extension method String.Sort : String;

implementation

extension method String.Sort : String;
begin
  var c := self.ToCharArray();
  Array.Sort(c);
  result := new string(c);
end;

Summary

Overall Delphi Prism 2011 is a pretty significant release and contains lots of new features that represent a lot of interesting new potential both on the language side and the IDE side, congratulations to the RemObjects team.

Further Reading

• What’s new in Delphi Prism 2011 – The Official Embarcadero page.
• New Features – The Delphi Prism Documentation Wiki page listing newly added features.
• Delphi Prism 2011 Release Notes – Is what it says on the tin.
• Using Type dynamic in .NET 4.0 – MSDN reference on the dynamic type.
• What is tail-recusion – StackOverflow answer on Tail Recursion which might help you to better understand Tail Calls in Delphi Prism.

WarbleCamp is a free unconference style event for the UK/Euro Twitter developer community which was held at the offices of the guardian. I was lucky enough to see an announcement notice before the first set of tickets had vanished in a very short space of time. We arrived at the very smart guardian offices where guardian staff were still working throughout the weekend (apparently there is some sort of big news unfolding or something?) and watched @jot and @kalv give the opening introduction to the event.

Session boards at Warblecamp (Photo from @raffi)

The day was broken up into Barcamp style sessions where anyone could propose a session simply by writing it onto a notecard/post-it note and finding a free timeslot and room on the wall board.

Session boards at Warblecamp (Photo from @raffi)

After several introductions and conversations over cups of coffee and a Strawberry and Banana oatmeal breakfast courtesy of moma foods we selected our first session. I was a little disappointment that there seemed to be a quite a few people who had registered for tickets and then simply never turned up.

Aral Balkan (@aral) – An Introduction to the Streaming API.

I felt very sorry for Aral as he’d bravely volunteered to move his session to one of the first of the day which meant that delays in starting was to be inevitable while people found rooms etc. It also amazes me how you can put 40-50 geeks into a room and then find that noone can work a projector.

After changing rooms we finally got onto the subject of the new Twitter Streaming API. The Streaming API is the 3rd Twitter API Subset which allows you to access various slices of public tweets. The Key differentiator is that unlike the other 2 twitter API services it actively pushes tweets to you in (sort of) real-time so that you don’t have to constantly poll Twitter’s servers to check for new information.

The Streaming API is an invaluable addition to the API Services and also hints at how many API services of this type of data model should actually be modelled. This session was cut rather short but one point that stuck with me is that Json is now the preferred response type and they’re considering XML return methods for depreciation.

Paul Mison (@blech) – Annotations vs Machine Tags

This session was a good discussion around the uses and power of Machine Tags and Annotations which focussed around Flickr Machine Tag usage as at this point in time Twitter doesn’t support adding metadata to Tweets (little did we know what was to come later in the day).

The basic concept is to decorate a resource with metadata in some way, this usually follows a simple model. This normally takes the form Namespace:key=value. For Example, on Flickr, One might tag a photo with Car:Manufacturer=Ferrari to indicate that it is a photo of a Ferrari car. This makes data about our photo more specific as a photo simply tagged “F357″ might be be any number of things.. a room, a camera, a car.

@aral started a good discussion around how you might “standardise” the namespaces of these tags and the hierarchy as it’s entirely possible that people may have different words for the same descriptor. There was popular consensus for an organic community driven approach.

@tommorris raised a good point: that even with a hierarchy of key-value pairs, is this really enough to actually associate a relationship perspective with an object? The example he gave within the context of Annotated Tweets is that you can tag a tweet with a book:title=”Treasure Island” but how can you tell whether this refers to a book I’ve read, a book I bought, a book I wrote?

Nick Halstead (@nickhalstead), CEO of Tweetmeme – #NoSQL

Nick gave us a brief overview of Tweetmeme‘s platform growth, they started with a well trodden track of a LAMP stack.

Tweetmeme services a fairly eye-watering amount of work: they store 4TB of data per day, index 500 million URLs and service 10,000 Req/s from about 25 servers. MySQL worked fine, following the usual Master / Slave division pattern until they got above a certain number of rows after which deletions started to become a pain point.

They have now moved to a Custom PHP job queuing system,  Nginx and HA_PROXY and use a tiered system of graceful degradation (similar to Twitter’s own system) which allows them to service requests differently according to the load on individual machines.

They first evaluated Tokyo and concluded that whilst it was very very fast  it was going to prove a difficult as it would mean carefully sharding data themselves.

They then evaluated Redis from which they were very impressed with the clever features to create Lists and Sets of data.

Finally they settled on Cassandra which provides eventual consistency and easy scaling across commodity hardware (a key advantage for a startup like Tweetmeme).

Christian Heilmann (@codepo8) – Geo Platforms

I’ve been a follower of Christian on Twitter for quite a while so I’m used to his enthusiasm for the Yahoo YQL platform but it was great to actually see his enthusiasm in person.

Whilst we were waiting for people to find their way to the room Christian revealed that some people are so into Geo that apparently they decide to walk around London on a particular trail in order to create a phallic route.. ergo gpsc**ks.com (conceivably NSFW) was born.

He first made the point that we’re beyond the point of it being novel to be able to display our GPS location on our mobiles and how we actually needed to do something useful with it. He covered how a few lines of Javascript we can use the HTML5 Geo API to locate a user (with varying degrees of accuracy). He pointed out that working with Geo locations is considerably more complicated than simply obtaining the Lat/Long of the current user. To help with this he suggested several resources:

• Geo Planet Explorer – A Yahoo web service that allows you to submit a WOE (Where on Earth) location and get back data about the neighbourhood and it’s neighbouring areas.
• YQL Geo Library – A JS Library which uses the YQL Library to provide services like geolocation, reverse geocoding, content analysis all in one easy library.

He then gave us a live demo of the YQL Service which which truly very powerful. It provides structured, consistent access to a whole load of different web services and datasets. This reminded me of what Ander Heijlberg said in his Technet 2010 keynote about programming for the what and not the how.

An example of a YQL queries might be: select * from geo.places where text=”san francisco, ca”

As someone who is used to working with web services the hard way, the YQL frees you from some of the effort of Authentication, gives you a consistent output format and does some caching on the server side.

Raffi Krikorian (@raffi) – Twitter Annotations

@raffi is Twitter’s Dev Lead for the API Platform (aside from being a generally nice chap) and had gotten permission just before he flew out to the UK to give us the first public draft of the upcoming annotations feature planned. You can check out his slides on the Twitter API Annotations and even watch the recording of the talk over on Vimeo.

The basic idea is that you will shortly be able to decorate tweets with up to 512 Bytes of data (excluding formatting and data classification structures). This could be used to append data about a book that the tweet refers to, a movie which it is talking about or any number of uses, including pure machine-to-machine communication.

This adds context to a tweet, something which twitter started doing when it added GeoLocated tweets and intends on doing more of in the future.

It is stressed that this is still a very early draft but that the Annotations may look something like the following:

{(type =>  (attribute => value),
               (attribute => value)}

And as an example:

[(“tv episode” => {“episode” => “The Vampires of Venice”,
		“series” => “Dr Who”,
		“air date” => “8 May 2010”}}]

The names of types and attributes will be allowed to grow organically from the community although they will have a broad set in use at the time of launch with support from a few partners. Attribute types and key name usage will be tracked and stats made available on dev.twitter.com to allow developers to browse existing usage implementations.

@codepo8 asked if they looked at RDF and microformats to see if they could adopt an existing standard for metadata and while they wouldn’t have been directly applicable, there is still a chance that they could reuse some of the taxonomy.

Lots of people asked if it would be possible to have metadata only tweets so that you could filter out all those annoying Foursquare checkin tweets and those “I’m listening to” tweets. This is now easily doable with the support of 3rd party client providers.

As with nearly all the previous new features released by Twitter, this may eventually be built into the twitter.com interface but it is initially planned as an API Platform feature.

Annotations will be made available to application developers  first as a preview within the next month sometime.

Others

There were several more sessions that I particularly enjoyed including Remy Sharp‘s (@rem) presentation on a conference dashboard he built he built for Chirp conference that works entirely with Javascript. Aral Balkan (@aral) also chaired what turned out to be an introduction and discussion of Twitter’s xAuth process (an adapted version of OAuth for Desktop Applications) which turned out to be a penny drop moment for me with the workflow process for OAuth Desktop applications.

I’m also keen to follow up on @ketan‘s proposal for a twitter:// protocol handler and to have a play around with the User Streaming API.

Thanks to the organisers, speakers and sponsors of warblecamp, I think it’s safe to say it was a well deserved great success. A special thanks to the primary sponsors, Guardian Open Platform, oneforty, Yahoo Developer Network, Paypal X and O2 litmus and to the micro-sponsors for supporting us with food, drink and entertainment.

]]> http://jamiei.com/blog/2010/05/warblecamp/feed/ 1 http://jamiei.com/blog/2010/03/yajl-for-delphi/ http://jamiei.com/blog/2010/03/yajl-for-delphi/#comments Wed, 31 Mar 2010 20:40:13 +0000 jamiei http://jamiei.com/blog/?p=575 Introduction

I have been toying with the idea of trying to convert the yajl parser bindings to Delphi in order to build a wrapper on top of the original C dll. yajl (Yet Another JSON Library) is a small fast SAX style JSON parser written and open sourced in C over at lloyd’s yajl GitHub page. I wanted to be able to use this library in particular because it’s very fast, small, portable and able parse from a stream. Plus, I had never really done any conversion of this nature on C header files before so I thought it might be a small, easy project to start with… I was a perhaps a little naive on this front!

I’ll leave you with a disclaimer early on: the code that I’ve been working on and have shared is not currently in a working state. This is the first part of my journey and I do intend on posting further as I learn from the major mistakes that I make in the process.

Approach

The 3 header files that I need to translate can be found in the src/api directory:

• yajl_common.h
• yajl_gen.h
• yajl_parse.h

I’m not sure what a *typical* process for conversion might look like but I started out by trying to use Dr Bob’s HeadConv tool as a way of getting me started and doing some of the legwork. However, the callback structure used by the parsing callbacks confused the tool and it never produced anything of use. After this setback I tried to install Rudy Velthuis‘s Conversion Helper Tool but I wasn’t able to load this into the Delphi 2009 IDE.

Having failed to get any of the conversion tools to do any of the work for me, I accepted my task and began to convert the headers from scratch. There were two resources that I found invaluable: the first is the Type Conversion table half way down the page on Dr Bob’s HeadConv page.  The second was the comprehensive Pitfalls of Conversion guide from Rudy.

As I began converting the headers I realised that although I thought I was translating the headers without a problem, the devil is in the detail and as my first major conversion project, I didn’t have the specfic knowledge necessary to correct my specific mistakes. As a result, please correct me if I have misunderstood any bits which I have picked up in the process.

Const parameters

One issue that I did notice is that there were quite a few parameters marked const in the header files, such as this declaration in yajl_parse:

   YAJL_API unsigned char * yajl_get_error(yajl_handle hand, int verbose,
                                            const unsigned char * jsonText,
                                            unsigned int jsonTextLength);

Rudy’s guide states that where possible Delphi programmers should ignore the const declaration as it’s easy to get wrong depending on whether the argument is passed by value or by reference, complicated by the issue that it means something subtly different in C. I translated this heading and simply ignored the const keyword:

    Tyajl_get_error = function(handle: yajl_handle;
                               verbose: integer;
                               jsonText: PChar;
                               jsonTextLegnth: Cardinal): PChar; stdcall;

Multiple Indirection Parameters

There was one declaration that I had to do a bit of searching for: it was in the yajl_gen file, in the declaration of yajl_gen_get_buf function:

YAJL_API yajl_gen_status yajl_gen_get_buf(yajl_gen hand,
                                              const unsigned char ** buf,
                                             unsigned int * len);

I had never seen the double asterisk before and found it surprisingly difficult to reliably Google for it. Luckily, this comprehensive page on C pointers was pretty useful in explaining that it is in fact a technique called multiple indirection which essentially creates a pointer to a pointer, in this case to a char.

    Tyajl_gen_get_buf = function (handle: Tyajl_gen;
                                 out buf: PChar;
                                 out len: Cardinal): yajl_gen_status; stdcall;

I wasn’t sure if this was indeed the correct translation but if you know otherwise then please let me know in the comments below.

Record Alignment and Enum size

When reading up on the perils of record alignment and enum sizes I found this blog post from Vlad loan Topan on Record Alignment and Enum sizes. Both problems are potentially applicable to the yajl header files. In his post, Vlad mentions that in order to line up the enum sizes you will need to use the {Z4} compiler directive to change the size from the Delphi default of 1 to the C compatible 4 as so:

type
  {$Z4}
  yajl_status = (
                  // no error was encountered
                  yajl_status_ok,
                  // a client callback returned zero, stopping the parse
                  yajl_status_client_canceled,
  ... snip ...

Reading through the Record alignment section of Rudy’s article I understood that I needed to search through the yajl project source to look for an Alignment setting. However, I couldn’t find a reference to the pragma directive anywhere in  the  source.

Next Steps

Well, it doesn’t actually work yet but I have put the Delphi-yajl project on GitHub where you can view my efforts so far, fork it and hopefully provide feedback or code contributions to show me where I’m going wrong. I have also setup a Delphi-yajl page here to serve as a placeholder for when it actually works!

Futher Reading

• yajl – The yajl project home page.
• Dr Bob’s Header Conversion Tool – The useful tool from the ever excellent Dr Bob.
• The Pitfalls of Conversion – Rudy Velthuis’s long but excellent article on commonly encountered problems when converting
• Conversion Helper Package – The Conversion package helper from Rudy but I’m uncertain as to how compatible it is any more.