Robert Pickering's Strange Blog

The twits - Yet another Twitter app

Robert Pickering — Sat, 04 Jul 2009 12:26:35 GMT

Taking its name from twitter, with a large nod towards Roald Dahl's cautionary tale, the twits is a twitter app that lets you visualize you network of friends on twitter. It’s available to try here. Before we go any further you should know that this software is in alpha status, i.e. not for the fait hearted. Here’s an image of the results below.

As you can see it does a reasonable job of separating other developers I follow from people at the guardian I follow (the coloured lines & text are added afterwards).

The app is designed to run local and uses .NET 3.5, F#, and WPF, if you’re running vista or win7 it’ll probably work okay out of the box, any other else and you’ll need install .NET 3.5 etc. Caveat: if you have more than about 50 friends (people you follow) it’s unlike to be able to map them all at once, first there’s the twitter API limits, then there’s the fact the algorithm is exponentially expensive in processing time and also you just won’t be able to find them on the screen. I may consider doing a Silverlight port, if it becomes popular, to make it more accessible. The main idea of the app is to show off some of Collective Intelligence work, this app is another example of hierarchical clustering and also uses optimization. There’s also some interesting use of async workflows. If you want to try the source it’s available on git hub. I've lots of ideas for future version - but not much time to develop them. Licence is GPL 2.0, if you'd like that to change, lobby me. If you like it let me know!

UPDATE: The download was broken because of a missing dll, this is fixed now. Thanks to @samjiman for noticing.

Videos to Watch

Robert Pickering — Fri, 19 Jun 2009 10:42:41 GMT

So this blog has been quite lately, well it never was the most active in the world. Anyway the good news is there are lots of other good F# resources on the web at the moment. Particually Lang.NET was a great source of F# videos, I was there myself, but I’ve spoke with a few people who were, and indeed there they told me there was lots of interest in F#.

A couple of videos I really liked were:

Luke Hoban on F# Productization: This was an interesting talk, not because it when in depth into F# the language, but because Luke gave a peak behind the scenes of how you run a major F# project - the productization.

http://www.langnetsymposium.com/2009/talks/25-LukeHoban-FSharpProductization.html

Amanda Laucher on F# Concurrency: This a very interesting talk from the charismatic Amanda “Pandamonial” Laucher. Amanda presents a case study of a real world project she’s been working on in the insurance domain. Her mission was to use the concurrency features of F# speed up the companies risk calculations, and the results she achieved are quite amazing. She presents what I see as very compelling reasons for using F#:

http://www.langnetsymposium.com/2009/talks/21-AmanderLauter-FSharpConcurrency.html

Of course I can’t not mention the “Language Oriented Programming in F#” talk by Roger Castillo at Lang.net’s sister event “DslDevCon”. I was briefly involved with the organization of this talk, though to be fair it was Chance Coble and Roger who did the hard work. I was hoping to make it to the event to co-present the talk, but alas, Paris is a long way from Seattle:

http://msdn.microsoft.com/en-us/oslo/dd727739.aspx

Other note worthy videos are Luke Hoban on F#, who makes some very interesting points about how F# programming differs from programming in C#. I like this video because it builds on what Brian McNamara said in this blog post.

Phil Trelford, did a nice session for the skillsmatter.com guys and gals, and also has another planned.

There’s also this series of 4 web casts from Tomáš Petříček, which to be honest I haven’t had chance to watch yet, but knowing Tomas’ work I’m sure they’ll be very high quality.

Past Speaking engagements

Robert Pickering — Fri, 19 Jun 2009 10:04:23 GMT

Back in February this year I spoke at TechDays Paris 2009, I’ve always enjoyed TechDays Paris so a big thanks to Eric Mittelette and his team for inviting me to speak. Also, many thanks Julien Laugel who help me review the presentation slides and also shared some his experiences using F# with the audience. A screen cast of this presentation is now available. The slides for this presentation are available on the slideshare.net site and the code from the slides is available here, except from the collective intelligence part which is available in its github project.

(aside: I’ve not been actively working on CI project for a while now, but will pick it up sooner or later. I’ve also been talking to Matthew Podywsocki, Joel Pobar, and a few others, who have similar interests in F# and CI, if you’re interested in contributing please contact me)

I was also lucky enough to be invited by Scott Bellware to give a tutorial at the “Progressive .NET Tutorials” at London, in partnership with skillsmatter.com. The podcast of this tutorial is now available on line, but its 3 hours long so not really for one sitting! The code form the tutorials in available here.

I currently have no concrete future speaking engagements, but have a few things in the offing. Feel free to contact me if you’d like me to present or run a tutorial for you.

F# in Beta1 of Visual Studio 2010

Robert Pickering — Tue, 26 May 2009 19:43:27 GMT

F# is now available as part of the beta1 of Visual Studio 2010. I felt a strange sense of pride when I could select F# as the default language in Visual Studio:

Find a summary of where to get it here: http://research.microsoft.com/en-us/um/cambridge/projects/fsharp/release.aspx

“Beginning F#” is “Foundations of F# 2nd Edition”

Robert Pickering — Sat, 04 Apr 2009 16:07:32 GMT

A few people have noticed that a new book “Beginning F#” by me is available for pre-order on Amazon. I wanted to make it clear that this is the new title for the second edition of “Foundations of F#”. The publisher wanted the title change to make it clear that this was a complementary rather title to “Expert F#” rather than competing with it. It also reflects that during the rewrite that I’m focusing a lot on making the book more accessible to people with no functional programming experience.

Interesting performance Consequences of Seq.map

Robert Pickering — Sat, 28 Mar 2009 18:06:27 GMT

It’s fairly well know that Sequences or “seq”, the short hand for IEnumerable, are lazy. This has some interesting performance consequence I had not considered until recently. When we execute a line like:

let lotsOfInts = Seq.map (fun x -> x + 1) (seq { 1 .. 1000000 })

The command executes almost instantaneously, despite the fact we’re creating a list of a million integers (Real: 00:00:00.001, CPU: 00:00:00.000 on my PC). This is because everything is lazy; no actual work is done till the list is enumerated. Executing a command like:

let lotsOfInts' = List.of_seq lotsOfInts

Takes a significant amount of time (Real: 00:00:01.107, CPU: 00:00:01.076 for me), because we turn the lazy collection into a concrete collection. This is often pretty much what we want, not to do any work until we need to enumerate the collection. The thing to beware of is that we do this work every time we enumerate the collection. This may be desirable for some times of collection, for example say something that reads from a file of data base that’s like to change between enumerations. But there’s also a class of problem were this is highly undesirable. Consider the following code:

let rec loop seq iteration =

let timer = new System.Diagnostics.Stopwatch()

timer.Start()

let seq' = Seq.map (fun x -> x + 1) seq

let list = List.of_seq seq'

printfn "Interation: %i time: %i" iteration timer.ElapsedMilliseconds

if iteration < 10 then

loop seq' (iteration + 1)

loop (seq { 1 .. 1000000 }) 0

Each iteration will take longer than the last, as each time “List.of_seq seq” is execute the sequence is recursively reiterated. I get the following results on my computer:

Interation: 0 time: 953

Interation: 1 time: 919

Interation: 2 time: 1151

Interation: 3 time: 1857

Interation: 4 time: 1528

Interation: 5 time: 2041

Interation: 6 time: 1820

Interation: 7 time: 2341

Interation: 8 time: 2300

Interation: 9 time: 2747

Interation: 10 time: 2673

It’s trivial to fix this problem, simply use the “Seq.cache” function:

let rec loop' seq iteration =

let timer = new System.Diagnostics.Stopwatch()

timer.Start()

let seq' = Seq.cache (Seq.map (fun x -> x + 1) seq)

let list = List.of_seq seq

printfn "Interation: %i time: %i" iteration timer.ElapsedMilliseconds

if iteration < 10 then

loop' (seq' :> seq<_>) (iteration + 1)

loop' (seq { 1 .. 1000000 }) 0

Which gives the following results:

Interation: 0 time: 1014

Interation: 1 time: 2337

Interation: 2 time: 2303

Interation: 3 time: 2294

Interation: 4 time: 2506

Interation: 5 time: 2343

Interation: 6 time: 2449

Interation: 7 time: 2075

Interation: 8 time: 2242

Interation: 9 time: 2370

Interation: 10 time: 2107

(Of course in this case it’s probably easier to password the concrete list we’ve already created, but in most places you’ll want to use “Seq.cache”)

Progress on the “DataTools” Project

Robert Pickering — Sat, 28 Mar 2009 17:32:44 GMT

I note on in a previous blog post that I’ve started creating an open source project for manipulating data. For the moment it’s mainly based around the idea of “Collective Intelligence”, but I hope to draw on influences from other sources as it evolves. I’ve reorganised the source so it’s a bit clearer and added some other concepts, notably a tool for accessing books from “Project Guttenberg” and the work I did on generic algorithms.

For the moment it’s licensed under GPLv2. I’m not a licensing expert but I believe this gives protection to the source while allowing it to remain open. If you need it under different licensing terms, contact me.

Collective Intelligence and the Guardian Data-Store

Robert Pickering — Thu, 19 Mar 2009 18:33:54 GMT

I’ve been interested in collective intelligence and machine learning for a while now. These too related fields centre round using statistical tools on large sets of data to make measurements and predictions. So when the UK’s Guardian newspaper announced their “Data-store”, a collection of data set open to the public I felt it was time to apply some of what I’ve learned to the data they were offer.

I choose to apply hierarchal clustering to the data on world health. The idea of hierarchal clustering is to measure how similar data sets are then pair off the similar data sets to build a binary tree that will relieve groups of similar data. I used the pearson correlation to compare the data sets and the resulting data is drawn in a dendrogram, a way of showing the distances between the various clusters that emerge from our clustering algorithm.

The code I’ve used is available on github.com, it’s packaged in an F# project called gdata.fsproj. For a direct link to the project click here. (There’s also a demonstration on hierarchal clustering with word counts from blogs from TechDays Paris 2009 talk).

Anyway, I’m not going to dig too deeply into the code, at least for this post, so let’s have a look at the results. First I clustered by county using the following statics to form my vectors:

Hospital beds per 1000
Nursing and Midwifery Personnel per 1000
One-year-olds Immunised with diphtheriatetanustoxoidandpertussisdtp
One-year-olds Immunised with hepatitis b
One-year-olds Immunised with hibhib3vaccine
Adolescent fertility rate (%)
Births attended by skilled health personnel (%)
Infant mortality rate (per 1 000 live births) both sexes
Maternal mortality ratio (per 100 000 live births)
Neonatal mortality rate (per 1 000 live births)
Life expectancy at birth (years) both sexes
Life expectancy at birth (years) female
Life expectancy at birth (years) male
Deaths among children under five years of age due to HIV/AIDS (%)
Per capita recorded alcohol consumption (litres of pure alcohol) among adults
Population with sustainable access to improved drinking water sources (%) total
Population with sustainable access to improved sanitation (%) total.

The statistics were chosen mainly because they were the most complete; it is only possible to compare countries using this technique if all statistics are available. The resulting dendrogram can be seen below:

There’s no great surprises from the stats, there appears to be two distinct clusters, one of poor countries towards the bottom of the diagram and one of richer countries towards the top, with the 1^st world countries being located towards the top of this cluster (absolute position doesn’t matter much is the diagram it’s more who your close to). There are perhaps a few surpises, maybe we wouldn’t have expected to find Cananda quite so close to the Ukraine or perhaps not the Czech Republic so closed to Germany. It may be worth going back to the underlying statistics to find why this is.

Perhaps a more interesting analysis is to reverse the matrix so we are no comparing which conditions are related to each other:

Again, the diagram does show some obvious relations. Male and female life expectancies were always going to statically similar to overall life expectancy, but it does appear that this is closely related to infant mortality rates. In turn is closely correlated to births attend by medical professions and access to clean water and sanitations. While this is fairly logical I think it’s good that we can show, statically speaking at least, that access to clean water and sanitation will improve infant mortality rates and life expectancy.

While these first steps in analysing the Guardian Data didn’t perhaps turn up anything we didn’t already know, I feel it’s shown that if you spend a bit of time working with public available data you can start to find interesting patterns. I shall definitely be looking at how I can further these experiments.

Feature Speaking Engagement – F# Tutorial at the Progressive .NET Tutorials, May 11-13th, London

Robert Pickering — Thu, 19 Mar 2009 14:50:34 GMT

I will be giving a half day F# tutorial at the “Progressive .NET Tutorials” organised by Skills Matter. This will be an excellent 3 daylong event with 2 tracks featuring half day and full day Tutorials by Gojko Adzic, David Laribee, Hammet, Ian Cooper, Mike Hadlow, Scott Belware and Sebastien Lambla.

My will be giving a half day tutorial on Wednesday May 13th (the last day of the event). I will be presenting 'F# Tutorial ', which will aim to give delegates the building blocks for using F# productively and to start having fun with it.

For the full programme and description of my tutorial, and all other Progressive .NET tutorials, check out: http://progressive-dotnet.com

Special Community Discount: Book on or before March 31st and pay just £500!

Skills Matter has given me a promotion code that will entitle you to a substantial discount off the Tutorial Fees. Simply book on or before March 31st, quote SM1368-622459-33L (in the Promo Code field) and pay just £500 (normal price £1000). Offer is valid until March 31st only, and tickets are going fast, if you would like to secure a place and claim your discount – you’d better get a wriggle on.

The code to use is: *SM1368-622459-33L* and must be entered in the box provided when booking online at https://skillsmatter.com/register-online/conf/280

Full details of the event can be found at http://progressive-dotnet.com

ALTi

Robert Pickering — Thu, 19 Mar 2009 10:11:10 GMT

(Sorry I’ve been a bit quiet recently; this is the first of several posts I’ll be making this morning)

I decided a little while ago that I’d like to change direction in my career and go back to consulting, after interviewing around a bit I decided to join ALTi. It was my first day on Monday, and so far I’m enjoying my first week, although obviously I’m just getting settled in. The thing I like most about the company so far is that they seem quite open to suggestions and seem will to let you develop your career in the direction you want. I’ll be predominately working on .NET projects in there .NET practice, so I’m interested in finding any projects with an F# slant out there (although F# won’t be my exclusive focus). I’m also hoping to develop the training and speaking side of my career. So if you have some F# work or are interested in having an F# presentation or tutorial, do not hesitate to drop me a line: Printf.sprintf "%s@%s.%s" "robert" "strangelights" "com"