With a compiled application we are able to make use of the 'EntryPointAttribute' to get access to the arguments passed in:

[<EntryPointAttribute>]
let main args =
    ShowFeedBurnerStats args
    0

Sadly this doesn't work with a script but it was pointed out on Hub FS that we can get access to all the command line arguments by using 'Sys.argv' or 'System.Environment.GetCommandLineArgs()' which seems to be the preferred choice of the compiler.

The problem is that with that method you get every single argument passed to the command line and there are some that we don't care about given the way you would typically call an F# script:

fsi --exec --nologo CreateFeedBurnerGraph.fsx -- "scotthanselman" "2009-03-01" "2009-07-14"

Results in the following arguments:

fsi
--exec
--nologo
CreateFeedBurnerGraph.fsx
--
scotthanselman
2009-03-01
2009-07-14

We care about everything after the '–' so I wrote a little function to just gather those values:

    let GetArgs initialArgs  =
        let rec find args matches =
            match args with
            | hd::_ when hd = "--" -> List.to_array (matches)
            | hd::tl -> find tl (hd::matches) 
            | [] -> Array.empty
        find (List.rev (Array.to_list initialArgs) ) []

I'm not sure this works for every possible case (if you put '–' in as an argument it wouldn't work as expected!) but it's doing the job so far.

An even better way of doing this which I came across while writing this is to use 'fsi.CommandLineArgs' which allows you to just get the arguments passed to the script. Even with this approach though the '–' is still counted as one of the arguments so the function above still makes sense.

GetArgs [|"--"; "scotthanselman"; "2009-03-01"; "2009-07-14"|]

val it : string array = [|"scotthanselman"; "2009-03-01"; "2009-07-14"|]

And from the script I have the following:

let programArgs = fsi.CommandLineArgs |> GetArgs
ShowFeedBurnerStats programArgs

These are some of my thoughts and our discussion of the paper:

• The first thing that was quite interesting was that the authors pointed out that if you just try and rewrite a part of a legacy system you are actually just writing legacy code yourself - we weren't sure exactly what was meant by this since for me at least the definition of legacy code is 'code which we are scared to change [because it has no tests]' but presumably the new code did have tests so it wasn't legacy in this sense. Rewriting the code doesn't really add any value to the business though as they point out since all that code might not even being used in which case it is just wasted effort. The idea of not rewriting is something that Uncle Bob advocates and Eric Evans also mentions the dangers of trying to replace legacy systems in his latest presentations.
• I thought it was interesting that the team didn't make use of automatic integration since they were frequently integrating on their own machines - I'm not sure how well this would work on a big team but certainly if you have a team of people fairly experienced with CI then I can imagine that it would work really well. Dean has written a post about serverless CI which covers the idea in more details.
• I liked the idea of putting up politics as user requirements on the story wall and then prioritising them just like any other requirement. More often the approach tends to be to try and address these problems as soon as they arise and then end up not really solving them and then getting burnt later on. This approach sounds much better.
• Another idea that I like is that the team didn't get hung up on process - the teams I've been on which worked the best weren't slaves to process and I've often heard it suggested that having a process is just a way of dealing with the fact that there is a lack of trust in the team. Jay Fields recently wrote about the idea of having more trust and less card wall and Ron Jeffries has a recent post where he talks about the light weight way that we should be making use of stories.
• Another really cool idea which I don't remember seeing before is having the whole team involved in major refactorings until the whole refactoring has been completed. Quite often with refactorings like this one pair might go off and do it and then when they checkin later there are a lot of changes and the other pairs have trouble merging and now have a lot of code which they are unfamiliar with.
• The idea of having a self selected team sounds like an interesting idea as you then only have people on the team who actually want to be on it and want to make things happen. I'm not sure how often this would actually happen but it is a nice idea.
• The importance of testing the system against a live database before putting it into production is emphasised and this goes beyond just using production data in a test environment. The team also made use of data verification testing to ensure that the new system and the current ones were working identically.

Although this paper is relatively short it's probably been the most interesting one that we've looked at so far. I think a lot of the ideas outlined can be used in any project and not just when dealing with legacy systems - definitely worth reading.

Stub objects

I use stubs mostly to control the output of depended on components of the system under test where we don't want to verify those outputs.

Most of the time I make use of the mocking library's ability to stub out method calls on these dependencies.

I find that it generally seems to require less effort to do this than to create hand written stubs although chatting to Dave about this he pointed out that one situation where it would make more sense to use a hand written stub is when stubbing out a clock/time provider. This is because there are likely to be multiple calls to it all over the place and most of the time you probably want it to return the same value anyway.

I actually quite like the fact that you need to specify all the stub calls that you want to make in each test - it helps you to see when you have too many dependencies and then hopefully you can do something about that.

On previous projects I worked on we decided the way to get around that problem was to define all the stub method calls in a setup method but that seems to lead to a world of pain later on when you forget that you've stubbed a method in the setup and now want to assert an expectation on it or (to a lesser extent) write a test which doesn't actually use the stub.

Fake objects

I often confuse fakes with stubs as seem to be quite similar to each other in their intent - the difference as I understand it is that with a stub we are controlling the output of a dependency whereas a fake just sits there and lets interactions happen with it. The values passed in earlier calls to the fake may be returned in later calls to it.

The most common use of this pattern is to replace a real database with a fake one for testing although on a recent project we were making use of a hand written fake session store to avoid having to refer to the real session in our test code.

We might have one call to the 'SessionFake' to store a value and then if a retrieve call is made later on we would return the value that we previously stored.

The approach Sarah describes for stubbing repositories seems quite similar to this as well.

Mock objects

I use mocks to replace depended on components of the system under test when I do care about the way that is is used i.e. we want to verify the behaviour of the dependencies.

If we see a mock object being created in a test then we should see a call to a 'verify' method later on to ensure that the expected methods are called on it.

I used to use these all over the place for just about every test where I wanted to control the way that a dependency acted until I realised how fragile and confusing that made the tests.

Now, after recently watching a presentation by Jay Fields, I try to ensure that I'm only setting up one expectation per test and use of the other test double approaches for any other dependencies that needs to be taken care of in that test.

Dummy objects

Most of the time when I pass dummy values into tests they tend to be strings and I prefer to pass in a value of 'irrelevantValue' rather than just passing in a null which may lead to difficult to locate Null Pointer Exceptions further down the line if the value which we thought was just a dummy starts being used.

We are generally only passing in these dummy values to satisfy the requirements of the system under test which may require values to be entered even if the particular piece of functionality that we are testing doesn't make use of them.

Overall

I think my current approach to testing leans more towards mockist rather than classicist although I think I am probably moving more towards the middle as I see the problems we can run into with over mocking.

With test doubles my current approach has minimising the effort required to create them as the most important aspect but I'm sure that will change given a different context. With all the test doubles I generally try and use test data builders where it's not overkill.

What did I learn?

• I decided that I wanted to parameterise the feedburner url so that I would be able to run the code for different time periods and against different feeds. In C# we'd probably make use of 'string.Format()' which has an equivalent in F# called 'sprintf'

  My initial thought was that I would be able to do something like this:

  let ShowFeedBurnerStats feed =
      let statsUrl = "https://feedburner.google.com/api/awareness/1.0/GetFeedData?uri=%s&dates=2009-01-01,2009-07-11"
      sprintf statsUrl feed |> GetXml
      // more code

  Which actually results in the following compilation error:

  The type 'string' is not compatible with the type 'Printf.StringFormat

  After a bit of searching I found a post by Robert Pickering where he explains that the format string needs to be next to the sprintf function to work as expected:

  let ShowFeedBurnerStats feed =
      let statsUrl = sprintf "https://feedburner.google.com/api/awareness/1.0/GetFeedData?uri=%s&dates=2009-01-01,2009-07-11"
      statsUrl feed |> GetXml
      // more code

  'statsUrl' therefore becomes a function taking in a 'string' and returning a 'string'.

• I'm still trying to work out the best way to decompose the code I write into functions which make sense in terms of the domain I'm working in.

  I often found myself splitting up a function along the boundary of where any I/O interaction was happening so that I could execute the I/O function and save the data before using it in another function which I would execute a lot more frequently (using F# interactive) while I was tweaking it.

• I still haven't come up with a completely satisfactory approach to coding these little applications - right now I'm finding that the feedback cycle is significantly quicker if I just write functions and then run them in F# interactive and then tweak anything which isn't working as expected.

  I didn't write any unit tests while coding this although I did find myself writing shorter functions than I originally did when writing my little twitter application. The problem of not writing the tests is that I lose the protection against regression that I would otherwise get.

• I still have a bit of a love hate relationship with tuples - I found myself making use of them early on when I was focused on getting the code to work and I could still understand the code easily.

  Originally I was only storing 'date' and 'circulation' in the tuple but once I added a third value to the tuple ('weeklyAverage') it became too confusing for me to understand so I decided to introduce the 'FeedBurnerStats' type to simplify things for myself.

• I ended up writing a function called 'Join' which is quite similar to 'Seq.zip' because I wanted to join two sequences together but only join items which had the same date (the 'string' value in the tuple).

  Therefore, if I had some data like this:

  'dailyStats'

  "2009-01,07", 200
  "2009-01,08", 222

  'weeklyAverages'

  "2009-01,07", 300
  "2009-01,08", 322

  I wanted the join of the two sequences to look like this:

  "2009-01,07", 200, 300
  "2009-01,08", 222, 322

  Which wasn't working as expected when I used 'Seq.zip' - the items that were getting matched together seemed to be quite random to me.

  let Join (dailyStats:seq<decimal*string>) (weeklyAverages:seq<decimal*string>) =
      dailyStats |> Seq.map (fun d -> { Date = d |> snd; 
                                        Circulation = d |> fst;
                                        WeeklyAverage = weeklyAverages |> Seq.find (fun w -> snd d = snd w) |> fst})

I've included the code is at the end of the post - there are some areas where I don't really like the way I've solved a problem but I'm not sure of a better way at the moment.

In particular:

• I wanted to make use of 'Seq.windowed' to find the rolling weekly average but I needed it to go back 7 days rather than forward 7 days which meant I needed to reverse the sequence. Right now I've done this by converting it to a list and using 'List.rev' to do so but this seems like a fairly inefficient way of doing this.

  The alternative seemed to be to write a function to change the order of the items in the sequence but again this doesn't seem like a great approach.

• What do you do with functions which are only used by one other areas of the code? For example 'ConvertToCommaSeparatedString' is only used by 'CreateGoogleGraphUri' so I defined it inside that function - I could then pull it to a function in its own right if other areas of the code need it. I did this to reduce the clutter of functions hanging around but it then makes 'CreateGoogleGraphUri' more difficult to read.

I decided to run it against some blogs I follow to see what the graphs, created using Google's Charts API, would look like:

ShowFeedBurnerStats "scotthanselman" "2009-03-01" "2009-07-11";;
ShowFeedBurnerStats "youdthinkwithallmy" "2009-03-01" "2009-07-11";;
ShowFeedBurnerStats "codinghorror" "2009-03-01" "2009-07-11";;

Interestingly you can actually see the points where feedburner for some reason counted a particular days circulation as being 0.

And here's the code:

open System.IO
open System.Net
open Microsoft.FSharp.Control
open System.Xml.Linq
open System
 
let downloadUrl (url:string) = async{
    let request =  HttpWebRequest.Create(url)
    let! response = request.AsyncGetResponse()
    let stream = response.GetResponseStream()
    use reader = new StreamReader(stream)
    return! reader.AsyncReadToEnd() }
 
let xName value = XName.Get value
let GetDescendants element (xDocument:XDocument)  = xDocument.Descendants(xName element)
let GetAttribute element (xElement:XElement) = xElement.Attribute(xName element)
 
let GetXml = downloadUrl >> Async.Run >> XDocument.Parse 
 
let GetDateAndCirculation (document:XDocument) = 
    document |> 
    GetDescendants "entry"  |> 
    Seq.map (fun element -> GetAttribute "circulation" element, GetAttribute "date" element)  |> 
    Seq.map (fun attribute -> Decimal.Parse((fst attribute).Value), (snd attribute).Value) 
 
let CalculateAverage days (feedStats:seq<decimal * string>) =
    let ReverseSequence (sequence:seq<_>) = sequence |> Seq.to_list |> List.rev |> List.to_seq
    feedStats |> 
    ReverseSequence |>
    Seq.windowed days |>
    Seq.map (fun x -> x |> Array.map (fun y -> y |> fst) |> Array.average, x.[0] |> snd) |>
    ReverseSequence    
 
let CalculateWeeklyAverage (feedStats:seq<decimal * string>) = CalculateAverage 7 feedStats
 
type FeedBurnerStats = { Date : string; Circulation: decimal; WeeklyAverage: decimal }
 
 
let Join (dailyStats:seq<decimal*string>) (weeklyAverages:seq<decimal*string>) =
    dailyStats |> Seq.map (fun d -> { Date = d |> snd; 
                                      Circulation = d |> fst;
                                      WeeklyAverage = weeklyAverages |> Seq.find (fun w -> snd d = snd w) |> fst})        
 
let GetFeedBurnerStats feed startDate endDate =
    let statsUrl = sprintf "https://feedburner.google.com/api/awareness/1.0/GetFeedData?uri=%s&dates=%s,%s"
    let allStats = GetDateAndCirculation (statsUrl feed startDate endDate |> GetXml)
    let weeklyAverages = allStats |> CalculateWeeklyAverage
    let dailyStats = allStats |> Seq.filter (fun x -> weeklyAverages |> Seq.exists (fun y -> snd y = snd x)) 
    Join dailyStats weeklyAverages   
 
let CreateGoogleGraphUri feed (stats:seq<FeedBurnerStats>) =
    let ConvertToCommaSeparatedString (value:seq<string>) =
        let rec convert (innerVal:List<string>) acc =
            match innerVal with
                | [] -> acc
                | hd::[] -> convert [] (acc + hd)
                | hd::tl -> convert tl (acc + hd + ",")          
        convert (Seq.to_list value) ""  
 
    let graphUrl = sprintf "http://chart.apis.google.com/chart?cht=lc&chtt=%s&&chco=000000,FF0000&chdl=WeeklyAverage|Daily&chs=600x240&chds=%s,%s&chd=t:%s|%s"
    let weeklyAverages = stats |> Seq.map (fun f -> f.WeeklyAverage.ToString("f0")) |> ConvertToCommaSeparatedString 
    let circulation = stats |> Seq.map (fun f -> f.Circulation.ToString("f0")) |> ConvertToCommaSeparatedString 
 
    let maximum = stats |> Seq.map (fun f -> f.Circulation) |> Seq.max
    let minimum = stats |> Seq.map (fun f -> f.Circulation) |> Seq.min
 
    new System.Uri(graphUrl feed (minimum.ToString("f0")) (maximum.ToString("f0")) weeklyAverages circulation)      
 
let ShowFeedBurnerStats feed startDate endDate = CreateGoogleGraphUri feed (GetFeedBurnerStats feed startDate endDate)

One thing which I found while doing this is that calls to the .NET base class library don't seem to fit in that well with the way that you would typically compose functions together in F#.

For example one of the first things I wanted to do was print the date and the circulation count to the console which I originally did like this:

open System.IO
open System.Net
open Microsoft.FSharp.Control
open System.Xml.Linq
open System
 
let xName value = XName.Get value
 
// GetXml is a function of type string -> string
 
let GetFeedBurnerStats url = 
    let feedBurnerXml = GetXml url |> XDocument.Parse
    feedBurnerXml.Descendants(xName "entry") |> 
    Seq.map (fun x -> x.Attribute(xName "circulation"), x.Attribute(xName "date")) |>
    Seq.iter (fun x -> printfn "%s %s" (fst x).Value (snd x).Value)

It's quite annoying that we need to store the XDocument as a value before being able to call one of the methods on it to get the data that we want.

I realised that if I created a function which took in the element whose descendants I wanted to find and the XDocument I could then call the 'XDocument.Descendants()' method inside that function:

1
2
3
4
5
6
7
8
9

let xName value = XName.Get value
let GetDescendants element (xDocument:XDocument)  = xDocument.Descendants(xName element)
 
let GetFeedBurnerStats = 
    GetXml >> 
    XDocument.Parse >> 
    GetDescendants "entry" >>
    Seq.map (fun x -> x.Attribute(xName "circulation"), x.Attribute(xName "date")) >>
    Seq.iter (fun x -> printfn "%s %s" (fst x).Value (snd x).Value)

Since we no longer need to store the intermediate step of creating the XDocument we can now just chain together the functions using the functional composition operator instead of the forward operator.

We can also do this with the calls to 'Attribute' in the 'Seq.map' function on line 9 which helps simplify the code around there.

1
2
3
4
5
6
7
8
9
10

let xName value = XName.Get value
let GetDescendants element (xDocument:XDocument)  = xDocument.Descendants(xName element)
let GetAttribute element (xElement:XElement) = xElement.Attribute(xName element)
 
let GetFeedBurnerStats = 
    GetXml >> 
    XDocument.Parse >> 
    GetDescendants "entry" >>
    Seq.map (fun x -> GetAttribute "circulation" x, GetAttribute "date" x) >>
    Seq.iter (fun x -> printfn "%s %s" (fst x).Value (snd x).Value)

It was roughly based around a CI Maturity Model which I recently came across although the intention was to find out what other teams were doing CI wise.

I became a bit more aware of how little I know about CI after listening to a Software Engineering Radio interview with my colleague Chris Read so I was keen to see how other teams are approaching this problem.

These were some of the most intersting parts of our discussion.

• Handling external dependencies - the idea here is that you don't have control over these end points and their reliability and it is very frustrating for the build to fail due to a problem with a dependency rather than a problem in your application.

  Several teams were testing integration of their application against these in a separate build something which we are also doing with a set of NUnit tests which run every 10 minutes directly against our service level end point.

  On my project we are also making use of an impersonator of our main integration point early in our build pipeline and then testing end to end integration at a later stage in the pipeline. We are not completely protected at the moment but we're getting there.

  The trade off here is finding the balance between making the impersonator way too complicated such that people can't understand it and we can't be sure that it actually works. It seems that if we are doing anything beyond a state machine with regards to handling requests & responses and are putting actual business logic inside the impersonator then we've probably gone too far.

• The importance of creating a single artifact if using a build pipeline and then running every unit, integration and functional test against that same artifact was pointed out.

  On my project we are recompiling our code at each stage of the build pipeline (from the same revision of the code in Subversion) as we haven't yet found a good way of putting the artifact somewhere where each build stage can pick it up from - some colleagues are making use of Ivy & Maven repositories to do this on Java projects.

  We also spoke about keeping the configuration of the application separate to the artifact so that we can easily deploy the application into different environments.

• We discussed the fact that keeping the build time very low by paying constant attention to it is one of the most important things to do to ensure quick feedback for the development team yet it is often allowed to slip to the point where we end up with build times exceeding an hour.

  Most of the teams had a pre commit build time of less than 10 minutes although only a couple had a build time of less than 10 minutes on their CI server.

  I've never seen it done but the idea of failing the build if it exceeded an agreed time was suggested forcing the team to address the problem rather than letting it gradually get slower and slower until the quick feedback cycle it originally provided has gone.

• The idea of running performance tests in the build was something which I found quite interesting - several colleagues pointed out that the point here is not to test the performance of the application when it's live but more to see the trend in how well our application is performing so that if it suddently dips we can address this straight away rather than waiting until later on to do this.

  We also spoke about including security tests - such as trying to inject evil SQL into every field - in the build although it was also suggested that these should be there anyway as part of the functional tests that we include. I think testing for XSS attacks might also fall into this area.

• Another interesting area of discussion was around whether it's acceptable for the build to break due to a change that we made. If we decide that it isn't then the only reason that it should fail is due to a problem with a depedency problems. To get to this stage it would need to be possible to run the entire build before checking in probably by having a grid of machines setup which we can assign local builds to run on. Dean Cornish has written an intersting post where he covers the idea of having continuous integration without a CI server.

  On all the projects I've worked on we haven't done this so in these situations I subscribe more to Stuart Halloway's thinking that it is actually ok to break the build in some circumstances. We are leaning on the CI server a bit to give us feedback but I don't think it's too bad to do this if it allows us to check in more frequently.

• We also spoke about the importance of getting the whole team to agree to a collection of CI practices if CI is really going to work on a project. These might include behaviours such as not checking in when the build's red, looking at why the build has broken if you're checkin broke it, rolling back your changes if you can't fix the problem quickly and so on.

I'm using the WebClient class to do this and the code looks like this:

let DownloadGraph (fileLocation:string) (uri:System.Uri) = async {
    let webClient = new WebClient()
    webClient.DownloadFileAsync(uri, fileLocation)}

Sadly this doesn't work when I run it from the client site where I have access to the build metrics as there is a corporate proxy sitting in the way.

I tried Googling how to do this but all the ways that I tried kept resulting in the following error:

407 proxy authentication required

Even though I was entering a user name and password!

I didn't succeed until my colleague showed me a way of getting past the proxy in C# which I could quite easily use in my code:

let DownloadGraph (fileLocation:string) (uri:System.Uri) = async {
    let webClient = new WebClient()
    webClient.Proxy <- new WebProxy("proxyName:port", true, null, new NetworkCredential("userName", "password", "corporateDomain"))
    webClient.DownloadFileAsync(uri, fileLocation)}

One thing I was doing wrong was putting a 'http' at the start of the proxyName which I think in my case was wrong as I later learn that the proxy isn't a HTTP one.

I'm also making use of asynchronous workflows in this example so that the actual downloading of the files will be done away from the main thread - this also gives me the option to download multiple files asynchronously if I want to.

After recommendations from Dean Cornish and Sam Newman on Twitter I decided to give the Google Graph API a try to do this and realised that I would need to create a comma separated string listing all the build times to pass to the Google API.

My initial thinking was that I could just pipe the sequence of values through 'Seq.fold' and add a comma after each value:

let ConvertToCommaSeparatedString (value:seq<string>) =
    let initialAttempt = value |> Seq.fold (fun acc x -> acc + x + ",") ""
    initialAttempt.Remove(initialAttempt.Length-1)

It works but you end up with a comma after the last value as well and then need to remove that on the next line which feels very imperative to me.

My next thought was that maybe I would be able to do this by making use of a recursive function which matched the sequence on each iteration and then when it was on the last value in the list to not add the comma.

I know how to do this for a list so I decided to go with that first:

let ConvertToCommaSeparatedString (value:seq<string>) =
    let rec convert (innerVal:List<string>) acc = 
        match innerVal with
            | [] -> acc
            | hd::[] -> convert [] (acc + hd)
            | hd::tl -> convert tl (acc + hd + ",")           
    convert (Seq.to_list value) ""

That works as well but it seems a bit weird that we need to convert everything in a list to do it.

A bit of googling revealed an interesting post by Brian McNamara where he suggests creating an active pattern which would cast the 'seq' to a 'LazyList' (which is deprecated but won't be removed apparently) and then do some pattern matching against that instead.

The active pattern which Brian describes is like this:

let rec (|SeqCons|SeqNil|) (s:seq<'a>) =
    match s with
    | :? LazyList<'a> as l ->
        match l with
        | LazyList.Cons(a,b) -> SeqCons(a,(b :> seq<_>))
        | LazyList.Nil -> SeqNil
    | _ -> (|SeqCons|SeqNil|) (LazyList.of_seq s :> seq<_>)

This doesn't cover the three states of the sequence which I want to match so I adjusted it slightly to do what I want:

let rec (|SeqCons|SeqNil|SeqConsLastElement|) (s:seq<'a>) =
    match s with
    | :? LazyList<'a> as l ->
        match l with
        | LazyList.Cons(a,b) -> 
            match b with
                | LazyList.Nil -> SeqConsLastElement(a)
                | LazyList.Cons(_,_) -> SeqCons(a,(b :> seq<_>))
        | LazyList.Nil -> SeqNil
    | _ -> (|SeqCons|SeqNil|SeqConsLastElement|) (LazyList.of_seq s :> seq<_>)

Our function to convert sequences to a comma separated string would now look like this:

let ConvertToCommaSeparatedString (value:seq<string>) =
    let rec convert (innerVal:seq<string>) acc = 
        match innerVal with
            | SeqNil -> acc
            | SeqConsLastElement(hd) -> convert [] (acc + hd)
            | SeqCons(hd,tl) -> convert tl (acc + hd + ",")           
    convert (value) ""

An example of this in action would be like this:

ConvertToCommaSeparatedString (seq { yield "mark"; yield "needham" });;
val it : string = "mark,needham"

I'm making use of the 'search' part of the API to return the metrics of all the builds run on a certain part of the pipeline and I then want to parse those results so that I can extract just the name of the agent that ran that build and the duration of that build.

The first part of this task is to parse the data and extract just the information I'm interested in.

The data is like this:

cruise_agent,cruise_job_duration,cruise_job_id,cruise_job_result,cruise_pipeline_label,cruise_timestamp_01_scheduled,cruise_timestamp_02_assigned,cruise_timestamp_03_preparing,cruise_timestamp_04_building,cruise_timestamp_05_completing,cruise_timestamp_06_completed\n
BuildAgentOne (Sydney, PersonOne),319,14052,Passed,2223,2009-06-25 12:14:01 +1000,2009-06-25 12:14:02 +1000,2009-06-25 12:14:02 +1000,2009-06-25 12:14:35 +1000,2009-06-25 12:19:54 +1000,2009-06-25 12:19:55 +1000\n
BuildAgentTwo (Sydney, PersonTwo),422,14084,Passed,2224,2009-06-25 14:13:57 +1000,2009-06-25 14:13:58 +1000,2009-06-25 14:13:58 +1000,2009-06-25 14:14:48 +1000,2009-06-25 14:21:49 +1000,2009-06-25 14:21:50 +1000\n

I first started off trying to do this extraction all in one regular expression but after a while realised that I'd probably have more success if I ran a regular expression over each line individually.

type CruiseData = { Agent: string; Duration: string }

1
2
3
4
5
6
7
8
9

let ExtractValues (item:string) =  
   let matchBuildDuration item = Regex.Match(item, "(.*\)),([0-9]+),") 
   Regex.Split(item, "\n") |> 
   Array.map (fun item -> 
        let m = matchBuildDuration item
        if(m.Success) 
        then { Agent = m.Groups.[1].Value; Duration = m.Groups.[2].Value } 
        else { Agent = ""; Duration = ""}  ) |>
   Array.filter (fun item -> item.Agent <> "" && item.Duration <> "")

I realised when I started writing the let statement inside the Array.map function on line 4 that I was thinking about this problem way too imperatively. I actually backed out at that stage and had another go but I decided it would be interesting to see what each iteration of the solution would look like if I had actually completed it.

An improvement on that would be to not set up an empty 'CruiseData' like we are doing on line 8 but instead to make use of the Option type to define when we do and do not have a value:

1
2
3
4
5
6
7
8
9

let ExtractValues (item:string) =  
   let matchBuildDuration item = Regex.Match(item, "(.*\)),([0-9]+),") 
   Regex.Split(item, "\n") |> 
   Array.map (fun item -> 
    let m = matchBuildDuration item
    if(m.Success) 
    then Some({ Agent = m.Groups.[1].Value; Duration = m.Groups.[2].Value }) 
    else None ) |>
    Array.filter (fun item -> item.IsSome)

It's still not great as we have imperative logic inside the Array.map function which looks pretty ugly.

At this stage I realised that I needed to excluded any lines which didn't match the regular expression so that I wouldn't have to care about them at all.

This was the next solution:

1
2
3
4
5
6

let ExtractValues (response:string) =  
    let matchBuildDuration item = Regex.Match(item, "(.*\)),([0-9]+),") 
    Regex.Split(response, "\n") |> 
    Array.filter (fun x -> (matchBuildDuration x).Success) |>
    Array.map (fun x -> (matchBuildDuration x).Groups) |>
    Array.map (fun group -> { Agent = (group.[1].Value); Duration = (group.[2].Value) } )

This is better although we are now calling the 'matchBuildDuration' function twice which is a bit wasteful.

Dave pointed out that if we run the data straight through the 'matchBuildDuration' function after splitting the new lines we can remove the need to call the function twice and then inline the function:

1
2
3
4
5
6

let ExtractValues (response:string) =  
    Regex.Split(response, "\n") |> 
    Array.map (fun x -> Regex.Match(x, "(.*\)),([0-9]+),"))  |>
    Array.filter (fun x -> x.Success) |>
    Array.map (fun x -> x.Groups) |>
    Array.map (fun group -> { Agent = (group.[1].Value); Duration = (group.[2].Value) } )

In all the functions we end up with the following data by executing this function:

[|{Agent = "BuildAgentOne"; Duration = "319"};
  {Agent = "BuildAgentTwo"; Duration = "422"}|]

My current thinking is that if I have more than one expression inside a function it's very probable that there's a better way of solving the problem and if I have conditional logic in there then I've gone very wrong.

I'd be interested to see if there's an even simpler way to solve this problem.

These are some of my thoughts and our discussion of the paper:

• One of the points suggested in the paper is that functional languages aren't used because of their lack of availability on machines but as Dave pointed out this doesn't really seem to be such a big problem these days - certainly for F# I've found it relatively painless to get it setup and running and even for a language like Ruby people are happy to download and install it on their machines and it is also pretty much painless to do so.
• Erik pointed us to an interesting article which suggests that functional programming can be very awkward for solving certain problems - I think this is definitely true to an extent although perhaps not as much as we might think. I am certainly seeing some benefit in an overall OO approach with some functional concepts mixed in which seems to strike a nice balance between code which is descriptive yet concise in places. I'm finding the problems that F# is useful for tend to be very data intensive in nature.
• Matt Dunn pointed out that an e-commerce store written by Paul Graham, which he later sold to Yahoo, was actually written in Lisp - to me this would seem like the type of problem that wouldn't be that well suited for a functional language but interestingly only part of the system was written in Lisp and the other part in C.
  

  Viaweb at first had two parts: the editor, written in Lisp, which people used to build their sites, and the ordering system, written in C, which handled orders. The first version was mostly Lisp, because the ordering system was small. Later we added two more modules, an image generator written in C, and a back-office manager written mostly in Perl.

• The article also suggests that it takes a while for Java programmers to come to grips with functional programs - I would agree with this statement to an extent although one of the things I found really hard when first reading functional programs is the non descriptiveness of the variable names. It seems to be more idiomatic to make use of single letter variable names instead of something more descriptive which I would use in an imperative language.

  I'm intrigued as to whether this will change as more people use functional languages or whether this is just something we will need to get used to.

• The author makes a very valid point with regards to the risk that a project manager would be taking if they decided to use a functional language for a project:
  

  If a manager chooses to use a functional language for a project and the project fails, then he or she will certainly be fired. If a manager chooses C++ and the project fails, then he or she has the defense that the same thing has happened to everyone else.

  I'm sure I remember a similar thing being said about the reluctance to make use of Ruby a couple of years ago - it's something of a risk and human nature is often geared towards avoiding those!

• I think the availability of libraries is probably very relevant even today - it helps F# a lot that we have access to all the .NET libraries and I imagine it's also the same for Scala with the Java libraries. I don't know a lot about the Lisp world but I'm told that people often end up rolling their own libraries for some quite basic things since there aren't standard libraries available as there are in some other languages.
• Another paper pointed out as being a good one to read was 'Functional Programming For The Rest Of Us' - I haven't read it yet but it does look quite lengthy! Wes Dyer also has a couple of articles which I found interesting - one around thinking functionally and the other around how functional programming can fit in a mixed programming environment

I think in general a lot of the points this paper raises have been addressed by some of the functional languages which are gaining prominence more recently - Erlang, F# and Scala to name a few.

It will definitely be interesting to see what role functional languages have to play in the polyglot programming era that my colleague Neal Ford foresees.