Rob Blackwell

Are You On My Scale Unit?

2013-01-26T00:00:00+00:00

Clemens Vasters has a really interesting video called Environments and Scale Units and gives some good insight into how to architect high scale cloud services and how Service Bus works on Windows Azure.

Clemens talks about the benefits of a consistent naming scheme for the components within a scale unit and how CNAME aliases can be used to direct subscribers to particular scale units. Here is his whiteboard diagram showing how your service bus namespace gets mapped to a cloudapp.net via the DNS.

]

I wonder how many scale units exist in Windows Azure and how the naming scheme works in practice.

Obviously Clemens can't give away too many details, so I thought it would be fun to poke around and see if I could find anything interesting.

Service Bus

I created new Service Bus namespaces, one in each of the datacentres and looked at the CNAME records in the DNS.

E.g. if robne1 is in Northern Europe

 $ host robne1.servicebus.windows.net
 robne1.servicebus.windows.net is an alias for ns-sb2-prod-db3-001.cloudapp.net.
 ns-sb2-prod-db3-001.cloudapp.net has address 94.245.88.192

Here is a table of my results

Region	DNS
East Asia	ns-sb2-prod-hkn-001.cloudapp.net
West Europe	ns-sb2-prod-am2-001.cloudapp.net
North Europe	ns-sb2-prod-db3-001.cloudapp.net
East US	ns-sb2-prod-bl2-001.cloudapp.net
Southeast Asia	ns-sb2-prod-sin-001.cloudapp.net
North Central US	ns-sb2-prod-ch1-001.cloudapp.net
West US	ns-sb2-prod-by1-001.cloudapp.net
South Central US	ns-sb2-prod-sn1-002.cloudapp.net

What can we deduce?

ns presumably means namespace?

sb2 could be Service Bus version 2; we know that there has been at least one big re-architecture / rewrite because that's also alluded to in the video.

prod = production.

Presumably hk = Hong Kong, am = Amsterdam, db = Dublin, bl = Blacksburg??, si = Singapore, ch = Chicago, by = Berkeley or Bay??, sn= San Antonio?

I have heard separately that there are three datacentres in Dublin, so perhaps db3 means the third datacentre? This might suggest that there are two Amsterdam datacentres and two in East US.

If 001 refers to the scale unit number, then could it be that there are only one or two scale units in each datacentre at the moment? Perhaps the individual scale units can actually handle a large amount of traffic?

Storage

Lets look at storage.

Region	DNS
Southeast Asia	blob.sg1prdstr03a.store.core.windows.net
East Asia	blob.hknprdstr03a.store.core.windows.net
North Central US	blob.ch1prdstr05a.store.core.windows.net
North Europe	blob.db3prdstr02a.store.core.windows.net
West Europe	blob.am2prdstr05a.store.core.windows.net
West US	blob.by1prdstr02a.store.core.windows.net
East US	blob.bl2prdstr02a.store.core.windows.net

Storage is using a slightly different format, but it appears to be based on similar principles; Singapore is sg not si.

prd = production, str = storage?

Perhaps 03a is the name of the scale unit?

Media Services

How about the newly launched Windows Azure Media Services? Probably not many users yet, but I can see this growing pretty rapidly. It might be fun to try tis gain in six month's time.

Region	DNS
Southeast Asia	wamssinreg001orig-hs.cloudapp.net
East Asia	wamshknreg001orig-hs.cloudapp.net
North Europe	wamsdubreg001orig-hs.cloudapp.net
West Europe	wamsamsreg001orig-hs.cloudapp.net
West US	wamsbayreg001orig-hs.cloudapp.net
East US	wamsblureg001orig-hs.cloudapp.net

wams = Windows Azure Media Services

sin = Singapore, hkn = Hong Kong, dub = Dublin, ams = Amsterdam, bay = San Francisco Bay area?, blu = Blacksburg??

So again, some similarities as well as inconsistencies. Assuming scale units are being used, then this would suggest that there is just one in each datacentre.

Conclusions

A lot of this is guesses based on public information and speculation, so if you know better, please let me know.

So how could we estimate how many scale units there are for the various Windows Azure services?

Feel free to record your scale unit DNS names in the comments below, or tweet them with hashtag #AzureScaleUnit . Let's see how many scale units we can find between us.

Programming Languages and Natural Languages

2012-07-20T00:00:00+00:00

Is choosing a programming language like choosing a natural language?

Recently I've been running a series of workshops in and around Europe. Whilst I can understand a little French and a few words of German, sadly my monolingualism has meant that all the sessions have had to be conducted in English.

At times I've found this a little embarrassing and I've certainly wondered about the wisdom and efficiency of a room full of Norwegians having to speak English on my account, but it has lead me to ponder the comparison of natural languages to programming languages.

English seems to be a lowest common denominator language in the same vein as most ALGOL like pseudo code. It's not ideal, but it seems to be your best bet for communicating an idea to a wide variety of people. Most people seem to have a smattering of English and most programmers seem to be able to understand block structured imperative languages. Many people have been "forced" to learn English. A bit like students have to learn BASIC or Java perhaps?

At the start of a new software project there is often a debate about the choice of programming language. Sometimes this borders on religious fervour. I wonder whether people argue about natural languages in quite the same way.

Natural language choice seems to be more about maximising the understanding of the communication. People seem much more willing to use their second language if it gives them access to new information. Equally they seem comfortable in breaking out in their native tongue amongst cliques if it furthers the aims of the wider meeting. Perhaps mixing programming languages is not as much of a problem as we might think.

In general, I think it's probably best to choose computer languages based mostly on the skills and needs of the team. It would be crazy to bring together a group of literary authors and ask them to write a book in an unfamiliar language, yet we do this in Computing in the name of progress.

If we'd stuck with say, FORTRAN, Common Lisp and C, instead of pursuing the likes of Java and C#, would we be any worse off today? What about the innovation that we have seen in languages like Haskell, Clojure and Erlang?

Just as better dictionaries and word processors have helped natural language writers, I'm encouraged that advances in programmer tooling such as Light Table, will take hold in the coming years.

Natural languages are borne of different cultures, so are many programming languages; consider the origins of Ruby, Clojure and Node for example. Consider their communities.

Domain specific languages can be powerful. The Saami People have a rich vocabulary for snow and ice that is ideal for living in the Arctic region. Similarly the programming language R is probably a good choice if you want to work with statistics.

Japanese Haiku don't seem to translate readily into English and programming languages have different expressive qualities. I imagine that taking a well written Haskell program and trying to convert it to C might similarly cause the loss of some expressive elegance. The Rubaiyat of Omar Khayyam is a beautiful collection of poetry in English but the original texts are in Persian. Must we learn Persian to truly appreciate it?

Truly multilingual people seem to be able to change the language in which they think. I believe that programming languages are also thinking tools and I often find myself designing programs in Lisp even if they end up having to be written in C#. Of course Lisp programmers tend to extend and change the language to suit the program. Natural languages evolve similarly with new phrases, and new words get added to the lexicon each year.

I suppose that as you learn more programming languages, you are able to express a wider variety of ideas and reason from different perspectives. Perhaps your ability to communicate with other programmers improves too?

Getting Started with the Windows Azure Command-Line Tools on Linux

2012-06-11T00:00:00+00:00

Big news. Windows Azure now supports Linux Virtual Machines and you don't need a Windows box to deploy them.

The new Windows Azure Portal no longer uses Silverlight, it's all HTML5 and works just fine from a Linux hosted web browser. Microsoft's new command line tooling allows you to manage your cloud deployments from a local Linux box via a Bash prompt.

You can sign up for a free trial account here.

The tools are written in Node.Js so you need to have a recent Node build to get things working. The official documentation suggests adding the Debian Sid repository to your apt config - that sounds pretty scary to me. Fortunately, installing Node manually isn't hard.

I'm using Ubuntu Linux 12.04. Things might be slightly different for your distro, but here goes:

You can download the Node source code from the Node website.

Unpack like this

tar -zxvf node-v0.6.9.tar.gz
cd node-v0.6.9

You need to install some dependencies

sudo apt-get install build-essential libssl-dev

But then you can do the usual configure, make, make install routine:

./configure
make
sudo make install

Then go and grab the Azure Command Line tools from Windows Azure Downloads

Beware, contrary to Unix etiquette, unpacking the compressed tar file will dump it's contents directly into the current directory rather than creating a single subdirectory, so you might wish to proceed like this:

mkdir azure
tar -zxv -C azure -f azure-2012-06.tar.gz

There's a configure file and a Makefile, so you're good to go.

cd azure
./configure
sudo make install

Now you can try it out:

azure

.. and this is where you first spot that the team probably hasn't been exposed to very much UNIX philosophy in their Microsoft careers. The output is very wordy and lines are prepended with tags like "info:" and "help:" which makes it hard to pipe output to other processes.

It doesn't feel very UNIXy and I am reminded of The Rule of Silence. "When a program has nothing surprising to say, it should say nothing."

(If anybody on the team is reading this, I suggest buying a copy of Eric Raymond's book "The Art of UNIX Programming". Actually I think it should be required reading for every programmer on every platform).

Anyway, let's get things configured with our Windows Azure credentials:

azure account download

This gives you a URL that you can use to download your account publishing settings. Having logged in with your Live ID in your favourite web browser, the file can be downloaded and imported.

azure account import my.publishsettings

If you have multiple accounts attached to your Live ID, then you'll need to set the correct account, something like this:

azure config set subscription "My Windows Azure Subscription"

And from there you can explore

azure site list

It's a great pity that the output is occluded by so much unnecessary line noise, but it's a good start. I guess the fact that it's open source means I should delve in and fix it. Now if only there were more hours in the day ..

For more information, see

www.windowsazure.com/en-us/manage/linux/other-resources/command-line-tools/

and

www.meetwindowsazure.com/DigitalChalkTalks.

Exploring Windows Azure Network Latency

2012-05-13T00:00:00+00:00

I wanted to understand the latency between the various Windows Azure data centres, so I set up six deployments, one in in each of the principal locations:

europe-north
europe-west
us-north
us-south
asia-east
asia-southeast

I installed an out-of-the-box copy of AzureRunMe using a small VM. I started IIS on each VM using:

> net start w3svc

We can't use ping to test latency because the Windows Azure VMs don't respond to the Internet Control Message Protocol (ICMP). Even if we turn off the firewall on the VM, ICMP is (quite rightly) blocked at the Windows Azure boundary.

So I started to experiment with tcping, a console app that uses TCP rather than ICMP.

tcping -i 5 -n 12 xxx.cloudapp.net 80

This runs twelve tests with an interval of five seconds; I took the average.

I conducted my tests during the early afternoon on Sunday 13 May 2012.

On each host, I ran the following test.bat file:

start /b tcping.exe -i 5 -n 12 europe-north.cloudapp.net 80 > europe-north.txt
start /b tcping.exe -i 5 -n 12 europe-west.cloudapp.net 80 > europe-west.txt
start /b tcping.exe -i 5 -n 12 us-north.cloudapp.net 80 > us-north.txt
start /b tcping.exe -i 5 -n 12 us-south.cloudapp.net 80 > us-south.txt
start /b tcping.exe -i 5 -n 12 asia-east.cloudapp.net 80 > asia-east.txt
start /b tcping.exe -i 5 -n 12 asia-southeast.cloudapp.net 80 > asia-southeast.txt

I occasionally saw a big spike in the results (one of the timings being about four times higher). I couldn't explain these results, so I confess; I just ignored that data and re-ran the test.

Here are my results to the nearest millisecond:

\	europe-north	europe-west	us-north	us-south	asia-east	asia-southeast
home	45	45	126	140	311	296
europe-north	16	31	105	125	311	346
europe-west	32	15	110	125	313	345
us-north	105	110	15	46	221	250
us-south	131	126	47	15	205	235
asia-east	311	313	220	204	15	47
asia-southeast	333	343	250	234	47	15

As you would expect, the latency between data centres within the same region is small - 45 ms for USA and Asia, 30 ms for Europe. I wonder why the European backbone seems fastest?

The time to ping a deployment from itself seems to be about 15 ms for all data centres. I'm seeing just less that 15 ms when I run the same test between two machines on my own gigabit ethernet switch. That suggests that the hyper-visor, VLAN and load balancer are not a big overhead.

My two nearest data centres are Dublin and Amsterdam; they seem to be neck-and-neck in terms of latency.

I thought it would be fun to play with tracert, but of course we soon run into problems with blocked ICMP. Modern Unix traceroutes have TCP and layer four options, but alas not Windows.

Instead, I installed NMAP, which now supports TCP based traceroute like this:

 nmap -sS -p 80 -Pn --traceroute europe-north.cloudapp.net

From home, I get this

TRACEROUTE (using port 80/tcp)
HOP RTT      ADDRESS
1   31.00 ms homeportal (192.168.1.254)
2   31.00 ms homeportal (192.168.1.254)
3   46.00 ms 213.123.107.186
4   46.00 ms 213.123.107.161
5   46.00 ms 213.1.69.66
6   46.00 ms 217.41.168.35
7   46.00 ms 217.41.168.107
8   46.00 ms 109.159.249.110
9   31.00 ms core1-te0-7-0-6.faraday.ukcore.bt.net (109.159.249.19)
10  31.00 ms 213.131.193.54
11  32.00 ms peer1-xe1-1-0.telehouse.ukcore.bt.net (109.159.254.136)
12  32.00 ms 195.99.126.42
13  32.00 ms ge-3-3-0-0.lts-64cb-1a.ntwk.msn.net (207.46.42.18)
14  32.00 ms xe-0-0-0-0.lts-96cbe-1a.ntwk.msn.net (207.46.42.168)
15  31.00 ms xe-8-2-0-0.db3-96c-1b.ntwk.msn.net (207.46.42.190)
16  ... 18
19  47.00 ms 65.52.76.146

But you can see that hops 16 to 18 are missing. It looks like those clever Windows Azure administrators have configured things so that you can't find any details about the infrastructure. Indeed this is confirmed if you try NMAP from the Windows Azure VM itself.

TRACEROUTE (using port 80/tcp)
HOP RTT     ADDRESS
1   ... 4
5   0.00 ms 65.52.76.146

The details f the hops might be missing, but at least we know the number of hops:

\	europe-north	europe-west	us-north	us-south	asia-east	asia-southeast
europe-north	5	9	11	13	13	16

Another thing I noticed was that if I used NMAP to portscan my xxx.cloudapp.net host, the RDP connection dropped. Looks to me like there are quite a lot of smarts in the Windows Azure security and denial of service attack protection ;-)

NMAP also now comes with Zenmap, but I'll leave the pretty pictures for another day.

The cURL Test for REST APIs

2012-04-06T00:00:00+00:00

REST APIs arguably have a much lower barrier to entry than SOAP APIs for client library implementers. The notion of simple HTTP verbs executed against resources makes it easy to build a client in any language that has a TCP/IP library.

But as soon as you consider security, up goes the complexity again. There are many choices including basic authentication, client certificates, custom signature schemes, OAUTH WRAP, OAUTH2.0 etc. Client library implementers now have to hope there are good security and cryptography libraries for their platform.

REST security standards are still evolving. OAUTH2.0 is still in draft at the time of writing. So how do we balance the needs of simplicity and security? I propose the cURL test:

If you can’t demonstrate and document your REST API using straightforward cURL samples, it’s probably too complicated.

cURL is a command line tool for transferring data to and from servers using web protocols. It’s designed not to require a web browser or any user interaction. If you use a UNIX like platform (Linux or MAC OS X), then you’re probably already familiar with its utility. If you’re on Windows I suggest you install it via Cygwin.

A simple GET request with no authentication can be accomplished like this:

 $ curl http://feeds.bbci.co.uk/news/rss.xml

If you use client certificates, you can specify the certificate using –cert. Here is an example of calling the Windows Azure Service Management API to list hosted services. Note the use of the x-ms-version custom header.

   $ curl --cert robblackwellmanage.pfx.pem:mypassword --header x-ms-version:2011-02-25 https://management.core.windows.net/MYSUBSCRIPTION ID/services/hostedservices

Calling the Windows Azure Service Bus API is a two-step process. First we have to call a Web Resource Access Protocol (WRAP) endpoint to obtain a Simple Web Token (SWT). Then we can use that SWT to present claims to the Service Bus API.

Here is the WRAP call:

 $ curl --data-urlencode 'wrap_scope=http://robblackwell.servicebus.windows.net' --data-urlencode 'wrap_name=owner' --data-urlencode 'wrap_password=c6V4LUpWKhXx1BxQGNQgJinMfpy3MuA83PuBUlufSS4=' https://robblackwell-sb.accesscontrol.windows.net/WRAPv0.9/

This returns a simple web token that looks something like this:

 wrap_access_token=net.windows.servicebus.action%3dListen%252cManage%252cSend%26http%253a%252f%252fschemas.microsoft.com%252faccesscontrolservice%252f2010%252f07%252fclaims%252fidentityprovider%3dhttps%253a%252f%252frobblackwell-sb.accesscontrol.windows.net%252f%26Audience%3dhttp%253a%252f%252frobblackwell.servicebus.windows.net%26ExpiresOn%3d1333573514%26Issuer%3dhttps%253a%252f%252frobblackwell-sb.accesscontrol.windows.net%252f%26HMACSHA256%3dEBxtUWnSD5tMFY9gpPpmIlB%252b7QeGUwrOEqgS7AWlWXE%253d&wrap_access_token_expires_in=1199

As you can see, the token is URL-encoded, so we need to URL decode it before we can use it. Let’s try that gain, with Perl to the rescue:

   $ curl --data-urlencode 'wrap_scope=http://robblackwell.servicebus.windows.net' --data-urlencode 'wrap_name=owner' --data-urlencode 'wrap_password= c6V4LUpWKhXx1BxQGNQgJinMfpy3MuA83PuBUlufSS4=' https://robblackwell-sb.accesscontrol.windows.net/WRAPv0.9/ | perl -pe 's/%([0-9a-f]{2})/sprintf("%s", pack("H2",$1))/eig'

This gives us something like this:

 wrap_access_token=net.windows.servicebus.action=Listen%2cManage%2cSend&http%3a%2f%2fschemas.microsoft.com%2faccesscontrolservice%2f2010%2f07%2fclaims%2fidentityprovider=https%3a%2f%2frobblackwell-sb.accesscontrol.windows.net%2f&Audience=http%3a%2f%2frobblackwell.servicebus.windows.net&ExpiresOn=1333660384&Issuer=https%3a%2f%2frobblackwell-sb.accesscontrol.windows.net%2f&HMACSHA256=B0Yc7g7ktGxXKvYETwyjjTxKHJXm9N5xsV1hDo9y3nw%3d&wrap_access_token_expires_in=1200

And we can copy and paste the SWT into a new command line like this and use it to authorize a request to list all the Queue resources on a service bus endpoint.

$ curl --header 'Authorization:WRAP access_token="net.windows.servicebus.action=Listen%2cManage%2cSend&http%3a%2f%2fschemas.microsoft.com%2faccesscontrolservice%2f2010%2f07%2fclaims%2fidentityprovider=https%3a%2f%2frobblackwell-sb.accesscontrol.windows.net%2f&Audience=http%3a%2f%2frobblackwell.servicebus.windows.net&ExpiresOn=1333660384&Issuer=https%3a%2f%2frobblackwell-sb.accesscontrol.windows.net%2f&HMACSHA256=B0Yc7g7ktGxXKvYETwyjjTxKHJXm9N5xsV1hDo9y3nw%3d"' 'https://robblackwell.servicebus.windows.net/$Resources/Queues'

This returns an XML / AtomPub feed with the list of queues. XML is arguably harder to read than JSON when returned on the command line, but at least it works.

The Service Bus example above is a bit torturous, but just about passes the cURL test.

Accessing Windows Azure Storage currently requires a reasonably complicated request signing process that would be very difficult with cURL, so I think we should regard that as failing the cURL test.

Of course you could always build a custom proxy server that translates simple cURL requests into the format and protocol that your particular complex API security demands, but that’s cheating!

Guest Lecture at Essex University 2012

2012-02-02T00:00:00+00:00

It was fun to talk about what it's like working in a software company at Essex University today. This year group seems like a lively lot!

Here are my slides.

My thanks to Keith Primrose.

Windows 8 on Acer Iconia Tab

2011-09-19T00:00:00+00:00

#BLDWIN envy got the better of me, so I downloaded the Windows 8 developer preview bits this weekend.

Running Windows 7 on the Acer Iconia has been, well, crappy to say the least. It's just not been in the same user experience league as my iPad, so I figured that I had nothing to lose by wiping it.

First I plugged in my Keyboard and DVD drive via USB (that's something you can't do on an iPad) and having worked out how to boot to the BIOS (Power + Windows key together) got it to boot. The installation was completely painless. Only the Bluetooth driver was missing, everything else just worked out-of-the -gate.

My initial reaction? This is what Metro is made for. Readers will know that I never really "got" the Windows Phone 7 thing with its text falling off the edge, but Metro looks fantastic on the tablet. It's simple, intuitive and it works. Most of the time you forget that it's Windows under the covers. (That is until you click something that's not yet implemented and you get dropped back into the old style desktop).

It's disappointing that the prerelease seems to be missing some of the keynote demo material (notably the Windows Live integration).

It's given the Iconia a whole new lease of life (in terms of performance and usability), and I've been surprised how many hours I've spent with it this weekend. I'm really taken with the joined up thinking - the device and the cloud really working together. This is the release of Windows that will properly integrate Windows Live rather than making it seem like an afterthought.

The Iconia hardware (and all the Intel tablets I've seen) seem a bit clunky (too thick, too heavy), but I really think that ARM will cure that problem next year. I hope that this is the bridge to ARM that will hasten the decline of power hungry Intel chips.

I'm not ready to give up my iPad yet, but I am inspired to try my hand at writing a Metro application. I've seen enough to suspect that I'll be investing in a serious slate at Windows 8 launch time.

This is a platform that's worthy of developer time and investment.

Microsoft Windows Azure Development Cookbook

2011-09-14T00:00:00+00:00

Packt very kindly sent me a review copy of Neil Mackenzie's Microsoft Windows Azure Development Cookbook.

The problem with Windows Azure is that it doesn't stand still for long enough to document it; even MSDN has been sadly lacking at times. Anybody who takes on a book project like this has to be either brave or mad, but writing a book with such broad and accurate technical information demands respect.

The Windows Azure Platform has a large set of features, but this book seems to cover them all. Everything from Compute and Storage through to AppFabric. There is even practical advice on AppFabric Caching and Windows Azure Connect (both relative newcomers to Windows Azure).

It was refreshing to see the book get quickly into detail and code. This book isn't for newbies; you won't find the usual "why cloud computing" space fillers here, just useful well-considered recipes that are clearly borne of real world experience.

Some of the material will start to date quickly; only today Microsoft launched the Windows Azure SDK 1.5, but Neil has tried to pick important, relevant and useful problems to solve, so I think that the book will fare well over time. I hope he will continue to update and refine it in further editions.

Yes there are one or two structural rough edges - the Index feels a bit computer generated, the AppFabric Caching information is awkwardly split across the first and the last chapters but that's nit picking; the content itself is first class and it's easy to dip into whenever you have a problem or need advice. This is a book that I'm hanging onto.

If you are a Windows Azure Developer or hands on Architect, then it's well worth the money - it consolidates a great deal of Windows Azure experience and wisdom in one place, it will save you googling time and you're bound to learn something new!

Why Node.js is Important for Windows Azure

2011-09-01T00:00:00+00:00

All the cool kids seem to be playing with Node.js, but why is Microsoft so interested?

Scott Hanselman is doing it.

Steve Marx is doing it.

Microsoft is even funding it.

Building web applications with JavaScript on the server has a nice symmetry with JavaScript on the client, but JavaScript has a reputation as an odd language. What's going on?

I think it's because Node.js will become a great way of building modern cloud applications. Perhaps it will even become de facto.

JavaScript is not so bad. It has all the interesting traits of a dynamic language whilst still having a familiar looking block structured syntax. Programmers don't freak out in the same way as they do when they first see Lisp or even Ruby. Some people even think that JavaScript is an acceptable Lisp. (It might have a REPL, JSON is almost homoiconic, but it would need a macro system to pass the Lisp test for me).

In the cloud, horizontal scale out matters. If you really want cloud economics, then it's better to have a large number of small virtual machines than a small number of large virtual machines; you can start small and scale more granularly. Ideally you want to run those VMs hot to get best value for money.

On Windows Azure, a small VM is a single 1.6 Ghz CPU with 1.75 GB of RAM. That's less than a crappy Netbook from PC World. The point is that small is beautiful if you can architect your software to make use of it.

We've all been told that Moore's law is coming to an end; that we won't be getting faster processors, but we will be getting more processors; that functional programming is the only way to cope with multithreading. Building multithreaded applications is hard, but if you're on a single CPU, maybe you don't need to bother? Perhaps there's a better way to keep that CPU busy, and that's where I think that Node comes in.

Node is inherently single threaded. Node gives you great performance by making the I/O asynchronous. If you aren't blocking on I/O then you have a much better chance of maximising CPU load.

The Node runtime is small and highly efficient, being based on the Google V8 JavaScript Engine. It compiles JavaScript to native code. There's still a way to go before we get a stable port for Windows, but the early signs are encouraging.

It turns out that cloud applications are typically I/O bound. A Web Role is waiting for web requests and a Worker Role is often shuffling data in and out of storage. These are ideal candidates for Node, which should give us better throughput on modest hardware. Node gives us a way to build small, fast, efficient, cost-effective, scalable cloud components.

I think it's a safe bet that we'll be seeing more of Node.js on Windows Azure. Watch this space.

In the meantime, feel free to check out node-azure.

Determining the Windows Azure SDK Version at runtime.

2011-05-29T00:00:00+00:00

I have had a few issues with people using old versions of the Windows Azure SDK with AzureRunMe (This isn't helped by the fact that there are at least three different versions of 1.4 in existence).

I wanted to display the Windows Azure SDK version when AzureRunMe starts, but it turned out to be more work than I'd expected. This information isn't currently available from any of the APIs.

There is a file called RoleModel.xml that gets written to the %RoleRoot% that has an attribute with the SDK version.

So here is the code

public string GetWindowsAzureSDKVersion()
{
    try
    {
        // Warning this is undocumented and could break in a future SDK release
        string roleModelFile = Environment.GetEnvironmentVariable("RoleRoot") + "\\RoleModel.xml";
        XmlDocument xmlDocument = new XmlDocument();
        xmlDocument.Load(roleModelFile);

        return xmlDocument.GetElementsByTagName("RoleModel")[0].Attributes["version"].Value;
    }
    catch (Exception e)
    {
        return "unknown";
    }
}

So now when AzureRunMe starts, you get something like:

Information RD00155D362047: 2011-05-28 09:20:47Z AzureRunMe 1.0.18.37253 on Windows Azure SDK 1.4.20407.2049

I know that this is unsupported and will probably break in some future version of the SDK, but needs must.

Windows Azure, the good the bad and the ugly.

2011-05-28T00:00:00+00:00

I really enjoyed speaking at UK tech.days about my experiences with the Windows Azure Platform.

I talked a bit about running non dot net applications and even managed to show some Common Lisp! I know it's pretty niche, but if you are interested in more details please see cl-azure.

You can download my slides from here.

The latest AzureRunMe code is available from GitHub.

The session video is now available.

Some interesting questions and lots of interest in running legacy code - particularly C and C++.

My thanks to Eric Nelson, David Gristwood and Paul Lo.

View this video as a WMV

Six reasons why MVPs should explore Windows Azure.

2011-04-13T00:00:00+00:00

I was speaking to Claire Smyth recently and we couldn't understand why there weren't more Windows Azure and SQL Azure MVPs in the UK. Here's why I think the MVP community and developers in particular should be interested:

Whatever your expertise, Windows Azure will touch your world.

Many Microsoft products are moving towards Software as a Service offerings: SharePoint, SQL Server, CRM and even Office. Most products will start to touch the cloud in some way, even if it's just for storage.

Many .NET applications are straightforward to migrate to Windows Azure, so cloud is particularly relevant to developers.

Exciting projects.

You only need to glance at the Windows Azure Case Studies to see the range of cool projects out there. From satellite communication systems to financial calculations and web sites promoting global brands, this is a chance to take your career to a new level and get involved in something big.

New opportunities and innovation.

Imagine having access to a large amount of compute power on demand. What could you build?

Perhaps you would implement a massively parallel high performance compute algorithm?

How about a new search engine or social network?

Perhaps you have the world's next big online idea?

Windows Azure gives you access to compute resources at reasonable cost, on demand. It's leveling the playing-field and allowing lone developers and start-ups to compete with big, established players. You no longer need a huge capital investment to get started.

There is a skills shortage.

There aren't many people around with hands-on Windows Azure expertise and less than a handful of MVPs in the UK.

With "Cloud Power" posters across London and major European transport hubs, itís hard to miss the marketing push. Despite, and perhaps because of, the global recession, people are migrating to the cloud and the momentum is building.

You don't need to be a rocket scientist, but cloud architecture is a different mindset.

Cloud is arguably a paradigm shift.

We've seen the mainframe era, the client-server era and the web era. Now we're embarking on the Cloud era.

It's easy and free to get started.

Grab yourself a free 30 day pass.

Come and see us at TechDays 2011.

Azure Northern Europe is Dublin and Western Europe is Amsterdam.

2011-04-12T00:00:00+00:00

Windows Azure has two European datacenters, Northern Europe and Western Europe located in Dublin and Amsterdam respectively.

Unfortunately, this is a cause of much confusion because, judging by Bing Maps, Dublin is much more westerly than Amsterdam and only slightly further north.

It's time to clarify this once and for all.

Wikipedia's definition of Western Europe is as follows:

Microsoft's slide from the Windows Azure Platform Training Kit is consistent, other than perhaps suggesting that Amsterdam is somewhere in northern Italy?

Let's try to back this up with a practical proof.

Northern Europe

Deploy AzureRunMe to Northern Europe as northereurope.cloudapp.net.

northerneurope.cloudapp.net resolves to 94.245.105.138.

When I RDP in and bring up a browser, bing.com reports United Kingdom.

That would all seem to be consistent with Dublin.

Western Europe

Deploy AzureRunme to Western Europe as westerneurope.cloudapp.net.

westerneurope.cloudapp.net resolves to 65.52.153.40.

When I RDP in and bring up a browser, bing.com reports Netherlands.

That would seem to be consistent with Amsterdam.

So there you have it, Northern Europe is Dublin and Western Europe is Amsterdam. QED.

Technology Enabling Business Event.

2011-02-10T00:00:00+00:00

Enjoyed speaking at The Framlingham Technology Centre today about Cloud Computing and Windows Azure.

Here are my slides.

My thanks to Jenny Stockman and Ian Chapman.

Guest Lecture at Essex University 2011.

2011-01-18T00:00:00+00:00

It was fun to talk about Software development, cloud computing and Windows Azure at Essex University today.

Here are my slides.

My thanks to Iain Langdon.

Making Azure more appealing to Java and Open Source Developers.

2011-01-06T00:00:00+00:00

Microsoft is increasingly reaching out to developers who use third party languages and application stacks with initiatives and announcements like Microsoft plans to make Java a first-class citizen on Windows Azure and PHP on IIS7.

Many developers are surprised when they see the extent of the Interoperability Bridges and Labs Center.

With the worldwide recession, consolidation, migration and evolution are the new watch words. Not many people can afford File-New and people are squeezing value from their legacy applications. I think its an interesting time for Microsoft. As we move to the Cloud and Platform as a Service, the operating system is becoming less and less relevant. It's value for money and cost saving that matters now.

Ironically, many non .NET applications are arguably easier to port to Azure because they have less Windows specific dependencies. They don't write to the registry or need elevated privileges or sophisticated installation. Many Java apps are very self contained with a clear abstraction over their operating environment.

But what would it take to make Azure more appealing to a wider development community? Here are some initial thoughts:

1 Lower Cost of Entry

Yes there are deals, incentives and extra small instances, but it's still too expensive to make a long term commitment to hosting a simple blog on Azure. Extra small instances are still in Beta.

Developer Fabric isn't a sufficient starting point and isn't cross platform. Developers need to use the cloud for real.

If you appeal to geeks through their side projects and spare time hacks they'll transfer their skills back to their day jobs.

Azure needs a "loss leader".

2 More TCP Ports

Five ports really isn't enough. Java people use sockets for everything from AJP, JDPA, LDAP, Web, SSL etc. If you want to use the new Remote Desktop facilities too, you have even less to play with.

3. SSH

Telnet and FTP aren't secure and RDP is overkill for quick systems admin tasks. SSH access is fast, flexible and secure - why isn't it included as an option with Windows?

4. A Good Porting Layer

I know we have Cygwin, MingW and VC++ but whatever happened to Microsoft's Services for Unix and POSIX support?

    ./configure
    make 
    make install

The above works seamlessly on most Unixes including Linux and Mac OS X - so why does the build have to be complicated for Windows?

5. A Package Management System

apt-get install

on Linux and

 brew install

on OS X - what about Windows?

Whatever happened to Garrett Serack and CoApp?

A package management system is not the same thing as an App Store!

Java JDBC to SQLAzure Connection Drop Workaround

2010-12-02T00:00:00+00:00

UPDATE: The Microsoft JDBC Driver 4.0 for SQL Server is now available in CTP. The information below is now officially documented on MSDN.

A while ago, we encountered some problems with using Java JDBC to access SQL Azure. In a nutshell, connections were getting forcibly closed after about one minute, preventing any kind of sensible connection pooling strategy and hurting performance. The problem seemed to be caused because TCP keepalive messages were not being sent.

Anyway, there is now a workaround.

For on premises services, you need to change the registry and then reboot for the changes to take effect. You can use regedit or simply run this BAT file:

REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v KeepAliveTime /t REG_DWORD /d 30000
REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v KeepAliveInterval /t REG_DWORD /d 1000
REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v TcpMaxDataRetransmission /t REG_DWORD /d 10

With a little messing around, you can apply the same fix to a Windows Azure instances too, but you need the recently released Windows Azure SDK 1.3 with its elevated startup task to be able to change the registry.

Add a StartUp task to your service definition file, something like this:

<Startup>
    <Task commandLine="Startup.cmd" executionContext="elevated" taskType="simple">
    </Task>
</Startup>

Then add a Startup.cmd file to your Web or Worker Role project. Make sure that the "Copy to Output Directory" setting is set to Copy always.

The Startup.cmd should look something like this.

if exist startup.txt goto skip
time /t >> startup.txt
REM Workaround for JDBC to SQL Azure
REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v KeepAliveTime /t REG_DWORD /d 30000 >> startup.txt
REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v KeepAliveInterval /t REG_DWORD /d 1000 >> startup.txt
REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v TcpMaxDataRetransmission /t REG_DWORD /d 10 >> startup.txt
shutdown /r /t 1
:skip

The trick here is the reboot - The first time through this script, the startup.txt file won't exist, so the registry update block runs followed by the shutdown command. On subsequent invocations, the startup.txt file exists, so the code block isn’t run, preventing further reboots.

Advanced Technology Workshops Cloud Computing

2010-12-01T00:00:00+00:00

I really enjoyed speaking at http://www.oerc.ox.ac.uk/events/cloudworkshop yesterday.

Dennis Gannon was particularly inspiring!

My slides are available here

Java on Windows Azure Roundup

2010-11-06T00:00:00+00:00

Introduction

I've been working with a number of customers, predominantly independent software vendors (ISVs) helping to migrate Java applications to Windows Azure.

There were some interesting Azure announcements at the Microsoft Professional Developers Conference (PDC) last week, so I thought it was time to give my perspective on the current state of Java on Azure and share some thoughts on what's coming.

The Story Today

My interest in Java on Azure started with Steve Marx and his Does Windows Azure Support Java? blog post and conference talks (still well worth exploring).

I haven't seen many "File > New > Java Azure" applications being written. Java people usually have an existing stack that they want to migrate and extend for the cloud. Often the application represents hundreds of man years of development and is tested on platforms ranging from Solaris through to AS/400 and Windows. Being cross-platform, these applications aren't about to start accumulating Azure specific features (at least not without a plugin architecture). I think that platform compatibility and bootstrapping tools are more important than developer tools per se.

The good news is that most Java applications are, by their nature, XCOPY deployable. Getting them packaged and uploaded to Azure is usually straightforward. Running them is the fun part.

Microsoft still seems to be promoting the Windows Azure Tomcat Solution Accelerator, but in the immortal words of Harry Enfield, "You don't want to do it like that." AzureRunMe is arguably a better starting point. It's general purpose and extensible; you don't have to keep uploading wedges of JVMs and app servers every time you want to change something.

The Windows Azure platform is well-designed, open, standards-based and interoperable from just about any platform with a TCP/IP stack. If you want to use Windows Azure specific features, then the Windows Azure SDK for Java is a good choice betrayed by a confusing web site. You might like to jump straight to the Sourceforge download.

There is an Appfabric SDK for Java Developers.

We have a Log4j appender for Windows Azure.

You might want to look at the Windows Azure tools for Eclipse, but do yourself a favour and buy Cloud Storage Studio too.

What are the Snags?

If you want to use JDBC to SQL Azure, then good luck. You might get away with it, but there are problems with connection timeouts and some reports of bugs due to SQL Azure / SQL Server differences.

You're only allowed 5 ports in your service definition. Many Java stacks use ports liberally for everything from JMX, JCA, clustering etc. Configuring these ports at runtime, whilst achievable with a bit of Sed Fu, is a bit of a pain.

If you're using non-blocking IO with java.nio, then you're probably going to hit trouble. For example, Apache DS and Jetty are both problematic, depending on their configuration.

You can't use the loopback adapter, so you need to beware when configuring IP addresses.

It can be a little challenging to build hybrid on-premises/cloud solutions although PortBridge is your friend.

The Java News from PDC10

The headline: "Improved Java Enablement: Microsoft plans to make Java a first-class citizen on Windows Azure. This process will involve improving Java performance, Eclipse tooling and client libraries for Windows Azure. Customers can choose the Java environment of their choice and run it on Windows Azure."

Virtual Machine VM Role will be a catch all solution so that we can say, "If it runs on Windows it'll run in the cloud". I don't advocate Infrastructure as a Service, but for pesky applications that violate the Azure Platform as a Service model, it's a nice fall-back position. It will also solve problems with unhelpful vendors who won't support their software in Azure because it's not a recognised build of Windows.

I'm more interested in Elevated Privileges than the VM Role because it should allow us to do installs, registry hacks and file system permission changes without sacrificing our Platform as a Service ideals. Presumably we will be able to use this to make registry setting changes to reconfigure the JDBC driver and work around problems or install a specific version of the C++ runtime for example. It's not yet clear to me whether this will enable us to run Windows Services, but I can't see any reason why not.

Extra Small Compute Instance should be a great way for cash-strapped developers to experiment with Azure. I hope that my own monthly Azure bill will now be lower than my hotel bills on future expenses claims.

Our WindowsTelnetDaemon was only ever meant to be a tactical fix, I'm looking forward to deprecating it in favour of Remote Desktop access. I hope that debugging Azure apps will become a lot less like remote debugging a Mars Rover.

The ability to dynamically take instances off the load balancer for debugging will be useful.

The improved Azure Platform Management Portal looks clearer and there is support for multiple users and roles. I'm disappointed that it appears to be Silverlight only which limits the Java / open platform story. I'll no longer be able to control my Azure instances from my iPad.

Azure Connect (formerly Project Sydney) will probably deprecate PortBridge for Azure scenarios and will make it much easier to build hybrid on-premises/ cloud solutions. I expect it to open up many enterprise consolidation scenarios.

Open in the Cloud

Vijay Rajagopalan gave an interesting talk Open in the Cloud: Windows Azure and Java and demoed the Fujitsu InterStage Java app running on the forthcoming Windows Azure SDK 1.3. There were some interesting clues in the demo.

The Eclipse tooling and packaging support looked good, but it's not a high priority for me. I think it will be tough to support a wide range of application servers and versions this way. Java developers are usually smarter than the average bear and will want to do their own thing, so having them manually ZIP up their frameworks and supporting tools for upload to Blob Storage isn't a big deal.

It appears that the CSPKG file still contains the Java Runtime and App Server making deployment across low bandwidth (rural home broadband!) connections too time-consuming. Dynamic web site updates through Tomcat looked promising.

Support for remote debugging looks good too, although that's pretty easy to achieve yourself using JDWP. You also need to be very careful about security (you don't want Script Kiddies attaching their debuggers to your app!). Again PortBridge is your friend in the short term, and Azure Connect will be the strategic answer.

Windows Azure 1.3 SDK will fix the Java NIO issues and enable the loopback adapter.

Looking at the Fujitsu configuration file, it appears that you'll be able to specify a "localPort" attribute to an EndPoint, so we won't have to worry about port remapping to 20000 20001 etc. in the future. It appears that the 5 port limit has been relaxed too?

I'm looking forward to checking out the support for the Windows Azure Management API in the new Windows Azure SDK for Java.

Other Thoughts

Unfortunately there was no official news, as far as I could tell, on JDBC to SQL Azure fixes?

I didn't hear much about the developer fabric which is too different to production reality to be useful for Java development scenarios. I'd still like a Virtual-PC-like solution that mirrors what's really running in Azure.

I've been playing with HyperV in anticipation of custom VM role, but it's more suited to servers than developer machines and laptops (issues with driver support for graphics cards in Windows Server 2008 etc). It would be nice if there was a more developer friendly solution using Virtual PC, but as far I can tell that's 32 bit only and therefore won't be suitable for building Azure custom VMs? VMWare support would be helpful, but I guess that's pretty unlikely.

Summary

There is now real commitment by Microsoft to Java on Azure. Some of the dates are a bit hazy, but the future looks bright, easier and more certain.

Java on Azure can work very well, and this is proving to be a more popular scenario than many of us would have guessed.

There will still be a role for AzureRunMe and we're aiming to continue to maintain, extend and support it.

Thanks Microsoft, we spoke, you listened and responded.

Of course this is also good news for Clojurians.

Migrating from iPhone to Windows Phone 7 and Back

2010-11-03T00:00:00+00:00

UPDATE: Paul Foster has been in touch, posted a response and kindly offered to come and help me!!

I picked up my new HTC Trophy Windows Phone 7 on Monday, hooked up my SIM card and got it onto the wireless network. Before long I was picking up email from our Exchange server. Good.

I wanted to try to buy some applications. The London Underground application is quite useful on the iPhone so I decided to start with that. I selected Marketplace then Applications, then searched for "London Underground". It appears that the search isn't context sensitive, so no applications were listed:

I don't understand this thing about fonts being too big for the screen and text getting cropped, but I hear that's just me, fair enough I can learn to live with that.

After some browsing through the marketplace menus, I found the application under travel.

It was $0.99. I expected the price to be localised to GB pounds for me, but never mind.

I selected buy. Then from confirm purchase, I needed to add a credit card. Rather surprisingly it shelled me out into the browser and asked me to sign in with my Live Id, before asking me to enter my credit card details. After entering the details, I got an error:

"We can't authorise the payment method. Please make sure the information is correct, or use another payment method. If you continue to get this message, please contact your financial institution."

It helpfully deleted all the card details I had already entered. I tried again. No luck. I tried another card, still no luck. I've tried this on several occasions and even had somebody look over my shoulder to make sure that I'm not doing something stupid, but still no luck. Re entering credit cards is a pain without cut and paste.

I managed to install some free applications, but was surprised that Twitter didn't integrate with the People Hub / Social Networking feature. Maybe I missed something?

Before the journey home, I switched on Bluetooth and paired the device with the audio system in my Volvo V70 car. Now whenever I start the engine, the phone reboots.

UPDATE: We tried Richard's new phone too. Every time that it connects via Bluetooth to the car it reboots; "Boom Baby" indeed.

Later, at home, I thought I'd install Zune and add some music. The Zune installer wanted a restart. Okay. Windows crashed on shutdown. I powered off and back on and started Zune.

I'm not familiar with Zune but I expected it to be similar to iTunes, so I had a click around and found a get started wizard:

It seemed that I needed to log in, but that failed (See below).

I logged onto Windows Live from my laptop browser and checked the settings. My profile suggests that I am in the UK. There doesn't appear to be an option for GB? My laptop regional settings are all UK too.

I went through and reset all the options that seemed to involve a country, but still no luck. I don't really want an entirely new Live ID when all my other Microsoft assets including Azure, use this one.

Back in Zune, I decided to try the Get Started wizard again.

Somebody suggested that the error message might well have been "check your musical tastes and try a more recent decade."

UPDATE: Somebody else suggested that the Mark Knopfler and Volvo V70 references in this article really do betray my middle-agedness. Thank you.

Sorry Microsoft, I really tried. I've been a Windows MVP twice and I really like working with Windows Azure. I know that some of my colleagues really like Windows Phone 7, but I'm afraid you've lost me on this one.

Perhaps I'll come back when you're a few versions in, but in the meantime, my iPhone 3 with its cracked screen seems like a preferable option.

Advanced Debugging on Windows Azure with ADPlus

2010-10-27T00:00:00+00:00

Let's say you have a C++ program that works locally, but doesn't work in Windows Azure compute - How do you go about debugging it?

The Windows Azure Diagnostics crash dump feature only works if the worker role itself dies, not if a spawned process dies.

Fortunately, ADPlus is now XCOPY deployable, so we can use it with Azure.

Let's assume that you are using AzureRunMe and the WindowsTelnetDameon to get onto Azure. (See my earlier blog posts ).

You need to have a 64 bit dev machine. Download and install the Debugging Tools for Windows (x64) which is part of the Windows 7 SDK.

Then ZIP up the "C:\Program Files\Debugging Tools for Windows (x64)" directory and upload it blob store. Configure AzureRunMe to load the zip package.

I have a very simple C program, FailTest.exe that divides by zero:

   #include "stdafx.h"

   int _tmain(int argc, _TCHAR* argv[])
   {

    printf("Hello world");
    int x = 10;
    int y = 0;
    printf ("Answer %d", x / y);
    return 0;
}

Okay, so then Telnet onto Azure and

  > adplus -crash -o . -sc FailTest.exe

This generates you a crash dump. You can drag it back to your dev machine by copying it to a cloud drive and downloading the VHD file.

You are then free to use crash dump analysers like WinDbg or VS2010 to see what's really going on!

My thanks to Neil Kidd for suggesting this.

If we can help with your Azure migration project, please contact us.

Telnet to Windows Azure

2010-09-23T00:00:00+00:00

UPDATE: AS OF NOVEMBER 2010, THE NEW AZURE SDK 1.3 IS AVAILABLE THAT SUPPORTS REMOTE DESKTOP - A MUCH MORE SECURE AND PREFERABLE SOLUTION!

Debugging Windows Azure applications can be time consuming, particularly if you make a mistake and have to redeploy. If you've done any serious work you'll know that it's easy to write an app that runs in the developer fabric but won't work in production.

Sometimes you just want to log on to the box and have a poke around. We have a rudimentary way of doing that using Telnet...

First install the Windows Telnet client - Control Panel > Programs and Features > Turn Windows features on or off > Telnet Client.

Now you need to install a telnet server in Azure. We've written a rudimentary telnet daemon that's available on GitHub . You can download and compile it with Visual Studio and test it out on your development box.

To get it running on Azure, grab a copy of AzureRunMe

AzureRunMe is a kind of bootstrap program that makes it easy to run EXEs, BATch files and many Windows apps that are copy deployable in Azure.

Upload AzureRunMe.cspkg upto your blob storage account (I recommend CloudStorageStudio for this). I use a container name called "packages".

AzureRunMe sets up some nice environment variables for you, including %ipaddress% and %telnet% which are the IP Address and telnet port respectively. Using that information, create a runme.bat file as follows:

Telnetd.exe %ipaddress% %telnet%

Zip up the two files - runme.bat and telnetd.exe together as telnetd.zip - upload that to blob store too.

Now you need an AzureRunMe configuration file, something like this:

You need to fill in your various Azure details of course, and then save as telnetd.cscfg, upload this file to Blob storage too.

Now log onto the Windows Azure portal and provision a new service. Browse for the CSPKG and CSCFG files from blob store. Run the new role.

Optionally you can use the TraceConsole from AzureRunMe to watch the boot-up process.

When the instance is up (give it 10 minutes or so), you should be ready to telnet in

> telnet claptrap.cloudapp.net

And away you go. Try typing SET, DIR etc.

Note that the terminal interaction is rudimentary. In particular the delete key doesn't work. It's not really secure enough for production servers like this, but if you need a more robust solution, feel free to hire us.

UK AzureNET Slides

2010-05-28T00:00:00+00:00

Last evening I had the opportunity to give a talk about “Interoperability on Windows Azure” at the UK AzureNET User Group meeting at Microsoft, Cardinal Place in London.

Here are my slides

My thanks to Marcus Tillett for organising the event.

Slides from QCon2010

2010-03-13T00:00:00+00:00

Here are my slides from QCon yesterday.

My thanks to Matt Deaon , Simon Thurman and Simon Davies.

QCon 2010

2010-03-09T00:00:00+00:00

I’m off to The QCon Conference tomorrow to help man the Microsoft stand.

We’ve built a couple of interesting web sites:

ukinterop.cloudapp.net – A Java / Restlet web site running on Windows Azure and
rubyukinterop.cloudapp.net – A Ruby on Rails site (Work in Progress!)

Both these apps use azurerunme.codeplex.com which makes it easy to run third party languages and technologies on Windows Azure.

On Friday I’ve been invited to join Microsoft’s Simon Davies and Simon Thurman for The Interoperable Platform session. I’ll talk a bit more about how we built the apps.

You can follow the event via Twitter.

David Gristwood and AWS talk Azure

2010-02-18T00:00:00+00:00

MSDN Flash Podcast 020

David Gristwood interviews Richard and me about our work with the Windows Azure Platform.

Guest Lecture at Essex University 2010

2010-01-24T00:00:00+00:00

Here are the slides from my lecture at Essex University on Friday.

My thanks to Iain Langdon for the invitation.

If you are a University student looking for a summer placement, please send me email.

Web Designer Issue 166

2010-01-16T00:00:00+00:00

Tip 88 in the HOT 100 section features me talking about Cloud Computing and Windows Azure.

Look out for Tricky and Ben too.

It’s unusual for me to be in the company of so many trendy people.

Haiti

2010-01-16T00:00:00+00:00

I’ve just been doing some preparation for a lecture I’m giving next week.

One of the demo’s is a mash up that takes data from the U.S. Geological Survey Earthquake Hazards Program in XML. It runs a simple clustering algorithm and overlays the data on a map using the Microsoft Bing Maps API.

It’s a demo that I’ve been doing for a couple of years, but this time the results were more shocking..

You can make a donation here https://www.ircuk.org/haitiappeal

VS2010 Beta 2 Problem on a Win7 MacBook

2009-11-30T00:00:00+00:00

I wasted far too much time trying to install Visual Studio 2010 Beta 2 on my MacBook this morning.

The installation just kept failing with this error message.

Microsoft .NET Framework 4 Beta 2:
Error code 5 for this component means "Access is denied.
Microsoft .NET Framework 4 Beta 2: 
Component Microsoft .NET Framework 4 Beta 2 returned an unexpected value.
Microsoft .NET Framework 4 Beta 2: 
Return from system messaging: Access is denied.
VS70pgui:
DepCheck indicates Microsoft .NET Framework 4 Beta 2 is not installed.

I tried to install the Framework separately and it failed too, suggesting that it couldn’t extract its files!

Anyway, to cut a long story short, it seems there is a problem with .NET Framework 4 Beta 2 that tries to write to the Mac OS X partition. Removing that drive letter fixes the problem.

Thanks to Kevin McNeish.

PDC 2009 Wrap Up

2009-11-23T00:00:00+00:00

I thought the keynote started slowly – I didn’t understand why the world needs a new Twitter client, but after some patience, it was good to see the whole Azure thing really come together.

Modern data centres are built using containerised servers like this one:

It was fun to be shut inside while they turned up the air conditioning ;-)

I’m intrigued by Dallas and PinPoint and the democratization of data. Zach gave me a key, so I’m looking forward to experimenting. I’m wondering how it will play in the semantic web space and whether the metadata will be rich enough to enable discovery and thus dynamic mash-ups. I want to be able to link concepts across data sets and I think RDF and Linked Data is probably the way to go.

I went to an interesting session on Microsoft Semantic Engine – It seems like WinFS on steroids. I think it has a lot of relevance to Intranets and our own research on Infoplaza. Unfortunately the PM spoke for so long that the engineer only had a few minutes to show us under the covers.

On Wednesday, Richard and I were invited to demonstrate some of our work with the .NET Service Bus (Lessons Learned: Building On-Premises and Cloud Applications with the Service Bus and Windows Azure). The video and slides are available here : http://microsoftpdc.com/Sessions/SVC31. There is an accompanying case study.

I’d heard about StreamInsight from Mark, but had been put off by the SQL Server badge. It turns out to be a very clever piece of kit that looks great for dealing with time series data in close to real-time.

The announcements about AppFabric (the new home of the Service Bus) seems to make a lot of sense. WIF, WRAP and ADFS V2 will surely change the way we build identity and security into web applications. Project Sydney looks like it will make it easier to mix cloud and on-premise applications.

Great to see Java and PHP running win Windows Azure and the launch of the Soyatec libraries for accessing Azure Storage. I think these are important cross platform and integration messages and it’s good to see Microsoft taking this seriously.

The file system over blob store technology (X drive?) is apparently coming soon, and I got the impression that it wasn’t just an Azure compute thing, but that you might be able to run it from your laptop.

Some interesting talks on High Performance Computing, multi-core and parallel etc. There is certainly innovation with f# but unfortunately I don’t find the syntax particularly elegant so I’m reluctant to use it in anger.

VS2010 and .NET 4 both demonstrate innovation. System.Dynamic is great news for obscure-language-compiler-developers like me!

I also got to chat for a few minutes with Patrick Dussud in the speaker work room. Patrick is a past contributor and inspirator to LSharp. He didn’t seem to have lost is enthusiasm for Lisp and he did a pretty good job of rekindling my own!

We were fortunate enough to be invited to the Azure VIP party. Great company, as well as blue illuminated cocktails.

Service Bus Talk

2009-11-15T00:00:00+00:00

Richard and I are heading off to the Microsoft Professional Developers’ Conference in LA where we’ll be presenting as part of Todd Holmquist-Sutherland’s session :

Lessons Learned: Building On-Premises and Cloud Applications with the Service Bus and Windows Azure.

We’ll be demonstrating some interesting aspects of the RNLI Sea Safety system which we now have running in the Windows Azure cloud. Specifically, Richard has written a cool SNMP library which allows remote management via the service bus.

Come and see us at 515A on Wednesday at 1:00 PM

AWS in the Daily Telegraph

2009-11-08T00:00:00+00:00

Active Web Solutions is a finalist in the National Business Awards for The Orange Best use of Technology category.

The Daily Telegraph included a Special Report on the National Business Awards this Saturday. The categories and finalists were listed, and there was a piece about AWS.

The winners will be announced on Tuesday at the award ceremony to be held at London’s Grosvenor House Hotel.

WindowsAzure4J

2009-10-29T00:00:00+00:00

Microsoft has released a Windows Azure Storage SDK for Java Developers – Windowsazure4j

http://www.windowsazure4j.org/

Source code is apparently coming soon, but we successfully used the JAR file to upload some Blobs to the Azure cloud today, so it seems a good alternative to JAzure.

cl-opencalais : A Common Lisp Wrapper for the OpenCalais API

2009-09-15T00:00:00+00:00

The Thomson Reuters Calais web service takes unstructured text and returns Resource Description Framework (RDF) formatted results identifying entities, facts and events within the content.

I’ve been playing with Calais and evaluating it for some upcoming project work. I thought it would be fun to build a Common Lisp wrapper to facilitate some exploratory programming. It turned out to be very easy using Drakma.

I’m getting to really like clbuild and I’ve added cl-opencalais and a number of other things to my wnpp-projects projects file, like this:

# Rob's own projects
cl-opencalais get_git git://github.com/blackwre/cl-opencalais.git

So it now seems really easy to mix and match libraries and just play!.

It was also my first try at using GitHub . I have to say it’s a joy.

cl-opencalais is available from http://github.com/blackwre/cl-opencalais

Profile of the 21st Century Researcher

2009-07-15T00:00:00+00:00

Yesterday I spoke briefly, and participated in a panel discussion at an event at the University of Essex.

Here are my slides.

SocketShifter

2009-07-06T00:00:00+00:00

Richard and I had a great time in Redmond a couple of weeks ago. While we were there, we put together a code sample that showed how to use the Windows Azure Services .NET Service Bus to connect any two arbitrary TCP endpoints irrespective of any intermediate firewalls.

We’ve cleaned up the code and released it on Codeplex

It turns out to be a really useful tool for setting up remote desktop connections from behind firewalls or managing remote SQL Servers using SQL Management Studio.

Our full blog post is available here

Exploring Windows Azure with LSharp in Full Trust

2009-06-18T00:00:00+00:00

I finally got around to updating lsharp.cloudapp.net to use Full Trust (it was just a case of adding enableNativeCodeExecution="true" to the ServiceConfiguration.csfg file, rebuildng and uploading).

Full Trust gives you access to a whole lot more than reported in my earlier article Poking around in Windows Azure

Access to the file system is now allowed

>(new "System.IO.DirectoryInfo" "c:\\windows")
c:\windows

And we can enumerate all the disks

>(reference "System.Management")
(seq (.get (new "System.Management.ManagementObjectSearcher" "SELECT * From Win32_LogicalDisk ")))
(\\RD00155D301BA6\root\cimv2:Win32_LogicalDisk.DeviceID="A:" \\RD00155D301BA6\root\cimv2:Win32_LogicalDisk.DeviceID="C:" \\RD00155D301BA6\root\cimv2:Win32_LogicalDisk.DeviceID="D:" \\RD00155D301BA6\root\cimv2:Win32_LogicalDisk.DeviceID="E:")

All the interesting stuff seems to be on the D drive

>(map (fn(x) (.ToString x)) (.GetDirectories (new "System.IO.DirectoryInfo" "d:\\")))
("$Recycle.Bin" "boot" "Debuggers" "Documents and Settings" "inetpub" "OSdiagnostics" "Packages" "PerfLogs" "Program 

Files" "Program Files (x86)" "ProgramData" "rdsources" "System Volume Information" "Users" "Windows")

We can now get to the registry and find out what kind of processors are being used.

>(reference "Microsoft.Win32")
(.GetValue (.OpenSubKey ( Microsoft.Win32.Registry.LocalMachine) "HARDWARE\\DESCRIPTION\\System\\CentralProcessor\\0") "VendorIdentifier")
"AuthenticAMD"

Have a play and let me know if you discover anything interesting!

Java, Restlet and Windows Azure

2009-05-19T00:00:00+00:00

There’s an interesting scenario for Windows Azure Services which hasn’t been talked about much: inter-enterprise application integration.

Connecting suppliers with customers can be tricky – Who hosts the web services? Who’s going to buy the server? How will we configure the firewalls? Azure could make all that pain disappear – just put an endpoint in the cloud and communicate securely. The endpoint could be a Table, a Queue, Blob store or the Services Bus.

Key to getting this scenario to work is Java support; there’s still a lot of legacy Java out there running on UNIX systems and we have to be able to work with that.

A quick Google for “Windows Azure Java” threw up only very little material:

Steve Marx’s blog http://blog.smarx.com/posts/programming-language-interoperability-in-windows-azure – Inspiring, but I struggled to get the code sample working and it’s a little messy.
and
Schakra’s Java SDK for .NET Services which, as far as I can tell only supports the Service bus.

Neither was what I was really looking for.

Restlet seemed like the way to go, so I contacted the folks at Noelios and Thierry Boileau responded with a code sample showing how the Microsoft Shared Key scheme was supported in Restlet 1.2 Milestone 2

See comments under

http://blog.noelios.com/2009/02/20/microsoft-selects-restlet-to-show-rest-interoperability/

Before long, I was listing queues, doing PutMessage to queues etc etc and I’ve started to build out a simple library that allows me to do things like:

String endPoint = "http://account.queue.core.windows.net";
String accountName = "XXXX";
String accountSharedKey = "YYYY";

QueueStorage queueStorage = new QueueStorage(endPoint,accountName, accountSharedKey);

Queue q = queueStorage.getQueue("foo");
q.putMessage("Hello World 2");

Is anybody interested in helping with or sponsoring a CodePlex project to build out a Java version of the StorageClient sample?

SemSearch 2009

2009-04-20T00:00:00+00:00

Dyaa and Udo will be at SemSearch 2009, at the WWW Conference in Madrid this week presenting our paper:

“Managing Collaboration Projects using Semantic Email”

Here is The Poster

I really wish I was going, but work commitments unfortunately don’t allow ..

Getting Started With ClojureCLR

2009-04-10T00:00:00+00:00

I heard that Clojure had been ported to the CLR and I was keen to give it a go. You need to grab the source and build it yourself, but here’s what worked for me:

I use subversion and git installed as part of Cygwin.

I installed everything in c:\opt

First you need to get the code for Clojure-contrib from Kevin O’Neill’s repository

git clone git://github.com/kevinoneill/clojure-contrib.git

Then change into the clojure-contrib directory

cd clojure-contrib

Then you need to get hold of the Dynamic Language Runtime. At the time of writing, ClojureCLR doesnt build against the latest DLR, so I had to explicitly grab revision 10606. The required version is documented in C:\opt\clojure-contrib\ClojureCLR\versions.txt

svn co -r 10606 https://dlr.svn.codeplex.com/svn .

You also need the Visual J# Redistributable to provide BigInteger and BigDecimal. Download and install from here:

http://www.microsoft.com/downloads/details.aspx?familyid=f72c74b3-ed0e-4af8-ae63-2f0e42501be1&displaylang=en

Locate ClojureCLR.sln (C:\opt\clojure-contrib\ClojureCLR\Clojure) and open in Visual Studio 2008.

Its a pretty big solution, so you may have to wait a bit while Visual Studio does its thing.

At this stage, the whole solution probably won’t build, unless you have Silverlight, NUnit , Rhino etc – but don’t worry – You only need to build Clojure.Console – Right click on Clojure.Console and select build. Right Click again and say Debug, Start New Instance. And off you go ..

If you’ve read this far, you probably want to buy Programming Clojure by Stuart Halloway. It’s available in beta.

Document Version Control Surface Style

2009-04-09T00:00:00+00:00

Another day another Surface demo. This time a POC for how a Surface could interact with a document management system to ensure that the latest versions of documents are used at meetings.

Document Version Control Surface Style from Rob Blackwell on Vimeo.

Microsoft Surface Arrives!

2009-04-08T00:00:00+00:00

A Microsoft Surface box recently arrived at AWS and we were up and running within an hour or so.

Maritime Charts on Microsoft Surface

2009-04-08T00:00:00+00:00

Richard Prodger leads the satellite tracking and sea safety projects at AWS. Here he is talking about how we ported our .NET based UKHO Admiralty charting system to Surface.

Maritime Charts on Surface from Rob Blackwell on Vimeo.

Interestingly, users of paper charts are already used to having a chart table, so there are some nice parallels here with Surface.

CAD Drawings On Surface

2009-04-08T00:00:00+00:00

Owen and I have been working on how to view Computer Aided Design drawings on Surface.

AutoCAD on Microsoft Surface from Rob Blackwell on Vimeo.

We have several customers in the construction sector and being able to call up drawings from their document management systems could be a killer application.

You could imagine Surface being used as part of the design process in client meetings.

Surfacing in Reading

2009-04-04T00:00:00+00:00

For the last few days, I’ve been at the Microsoft Technology Centre at Reading, locked in a room with 4 Microsoft Surface Boxes, Dr Neil and a bunch of other interesting people.

It’s pretty easy to get a WPF app running on a Surface (I managed to get the AWS Admiralty Charts running).

Building intuitive, seamless, social, 360 degree interfaces is a whole different ballgame though …

I don't do Email

2009-04-02T00:00:00+00:00

On Thursday 26th March I met with David Lammy, MP for Tottenham and Minister for Higher Education.

We showed him some of the great research that we’ve been doing with Essex University that promises to revolutionise the way we run collaboration projects by appying natural language processing techniques to email.

Unfortunately David admitted that he didn’t do email.

Mix09

2009-03-22T00:00:00+00:00

Just got back from Las Vegas where Richard and I attended Mix09.

Lots of interesting stuff – Here are some notes:

Launch of Internet Explorer 8. ie8addons.com , IE8 Tools versus Firebug
ASP.NET MVC 1.0 – NerdDinner sample application
Availability of a Mobile Device BRowser file on Codeplex
Microsoft Translator
Silverlight 3 CTP
Blend 3 Sketchflow
Bizspark initiative for start ups
350 Million installs of Silverlight
IIS Media Services and Smoth Streaming
Support for SFTP – finally!
WIndows Web App Gallery
Webplatform Installer

Guest Lecture at Essex University 2009

2009-02-24T00:00:00+00:00

Today I had the opportunity to talk to some second year Computing students about Cloud Computing, and to demonstrate some of the work we’ve been involved with at AWS.

Demos included Virtual Earth, Photosynth and a Facebook application.

We also talked about Windows Azure and the amazing opportunities that cloud computing mashups afford.

Here are the slides.

My thanks to Iain Langdon and Simon Lucas for the invitation.

If you are a University student looking for a summer placement – please call the AWS Office on 01473 834560 and speak to Lucy, or drop me an email.

Serving VE Mapcruncher Tiles from Windows Azure

2009-02-10T00:00:00+00:00

Microsoft Mapcruncher is a tool that allows you to overlay your own custom maps onto Virtual Earth. It works by “Crunching” your map into tiles which you then make available via a web server so that they can be overlaid in VE using map.AddTileLayer. Depending on the complexity of your map, and the number of zoom levels, Mapcruncher can generate quite large amounts of data. Mapcruncher already has a facility to render its data directly to Amazon S3, but now you can copy those tiles to Windows Azure Blob Storage instead.

First crunch your data – There are a number of online tutorials and good instructions with Mapcruncher, so I wont go into detail here. Let’s assume you’ve crunched your data to a local folder, say c:\ve

Now install PowerShell and CloudDrive (CloudDrive is in the samples directory of the Windows Azure SDK – follow the instructions you find there).

Using PowerShell and CloudDrive, change directory into Blob storage:
> cd blob:
Create a new public container:
> new-item -public tiles
Change into that container:
> cd tiles
Copy all the tiles from your local drive to the cloud:
> copy-cd c:\ve\Layer_NewLayer\* .
Now all these tiles should be publicly available.

If you are running against development blob storage, then URLS will look something like this
http://127.0.0.1:10000/devstoreaccount1/tiles/0313131311133.png
For production cloud storage, they’ll look like this
http://myaccount.blob.core.windows.net/tiles/0313131311133.png
Now we need some HTML to try it out – I grabbed the sample from the Virtual Earth Interactive SDK and hacked it around a bit to suit my particular map. The main thing to change is the path to the tiles., something like this:
var tileSourceSpec = new VETileSourceSpecification("lidar", "http://127.0.0.1:10000/devstoreaccount1/tiles/%4.png");

or this

var tileSourceSpec = new VETileSourceSpecification("lidar", "http://myaccount.blob.core.windows.net/tiles/%4.png");

Once you’ve created your HTML, you can upload it as before:
> copy-cd C:\ve\index.html .
Now browse to your storage end point
http://127.0.0.1:10000/devstoreaccount1/tiles/index.html
or
http://myaccount.blob.core.windows.net/tiles/index.html
and away you go!

Poking around in Windows Azure

2009-02-09T00:00:00+00:00

Now that I have my first app running in the real Windows Azure cloud, and that app is a scripting engine, I wonder what I can discover about the underlying platform?

Running this:
(reference "System.Net") (using "System.Net")

(let ip (first (.AddressList (Dns.GetHostByName (Dns.GetHostName)))) (prn ip) (prn (.HostName (Dns.GetHostByAddress ip))))
We get
10.113.121.84 RD00155D302603.reddog.microsoft.com
The IP Address is a private IP address. So, as you might expect, there is network address translation going on. The hostname reveals the Red Dog name which was the code name for the project prior to PDC.

If we ping lsharp.cloudapp.net we get
70.37.9.89

A quick port scan (purely in the interests of research!) shows that only port 80 is up for that address.

The 70.37.9.89 address is indeed registered to Microsoft Corp and DNSStuff reports the city as Temecula, California – is that where the data centre is I wonder?

We seem to be able to make arbitrary outbound HTTP connections like this:
(.OpenRead (new "System.Net.WebClient") "http://www.aws.net")
But not to the localhost
(.OpenRead (new "System.Net.WebClient") "http://127.0.0.1:80")
and not to other addresses on the local network
(.OpenRead (new "System.Net.WebClient") "http://10.113.121.85")
In both cases we get a
System.Net.WebException: Unable to connect to the remote server --->

System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions

The publc IP Address works fine
(.OpenRead (new "System.Net.WebClient") "http://70.37.9.89:80")
Can we access the registry ?
(reference "Microsoft.Win32") (.OpenSubKey ( Microsoft.Win32.Registry.LocalMachine) "HARDWARE\\DESCRIPTION\\System\\CentralProcessor\\0")


System.Security.SecurityException: Request for the permission of type ’System.Security.Permissions.RegistryPermission,

mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’ failed.
Not at all suprising really!

Can we access the file system ?
(new "System.IO.DirectoryInfo" "c:\\")


System.Security.SecurityException: Request for the permission of type ’System.Security.Permissions.FileIOPermission,

mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’ failed.
How about using System.Management ?
(reference "System.Management") (.get (new "System.Management.ManagementObjectSearcher" "SELECT * From Win32_LogicalDisk "))

System.Security.SecurityException: That assembly does not allow partially trusted callers.

So, as you would expect, some smart people have put an awful lot of thought into platform security and there don’t seem to be any chinks in the armour! Can anybody think of anything else that might reveal more info about the platform?

LSharp in the Cloud Video

2009-02-07T00:00:00+00:00

Howard van Rooijen from EMC (formerly Conchango) made a video of Adrian Jakeman and me talking about LSharp in the Windows Azure Cloud.

Great Job – many thanks!

LSharp in the Cloud from Howard van Rooijen on Vimeo.

LISP in the Windows Azure Cloud?

2009-02-05T00:00:00+00:00

So here I am in London learning about the Microsoft Windows Azure Services Platform

In an idle few moments, I thought it would be fun to try to run LSharp in the cloud – I expected that the partial trust model would probably prevent the reflection and code gen stuff from working, but to my surprise it seems to work well! Try it out at http://lsharp.cloudapp.net/

I’m just pushing the source code up to Subversion if you want to take a look ..


svn co https://lsharp.svn.sourceforge.net/svnroot/lsharp/v2 lsharp

You can run it locally using the dev fabric (I’m using the January CTP build).

The rss.ls sample runs unmodified, as does fib.ls and factorial.ls (look in the samples directory)

One interesting thing is using LSharp to have a dig around in the Azure runtime


> (System.Environment.OSVersion)
Microsoft Windows NT 6.0.6002 Service Pack 2, v.223
> (System.Environment.ProcessorCount)
1
> (System.Environment.Version)
2.0.50727.3069

Special thanks to Simon Davies and Alistair Beagley who managed to get me an Azure account in record time.

I’ve no idea whether it will stay up for long, but have a play and let me know what you think …

LSharp Progress

2008-12-23T00:00:00+00:00

Back in September, I decided to have another look at L Sharp, with a view to building a compiler. My intention was to try to port it to the Dynamic Language Runtime – More by luck than judgement, I stumbled upon LINQ expression trees and then I just kept going …

Early on, I was influenced by Paul Graham’s Arc, but as time went by, I also discovered Clojure – I’ve stolen ideas from both.

By early December, things had stabilised enough to start sharing the source code.

svn co https://lsharp.svn.sourceforge.net/svnroot/lsharp/v2 lsharp

There are still rough edges, but it feels like a major step forward over version 1.

There have been a few discussions on the Google Group .

Daniel has done an AutoCAD plugin and Roman has managed to get it working in Silverlight. Roman is in the UK in January, so I hope to incorporate his changes into subversion.

Anyway, if you have time over the Christmas break, check it out www.lsharp.org.

L Sharp Version 2 Coming Soon

2008-10-17T00:00:00+00:00

It’s been a while, but I’m working on L Sharp again! I hope to have a version 2 beta released by Christmas 2008. Here are some highlights ..

Major rewrite which exploits .NET Framework 3.5.
New liberal license terms – Probably BSD or Perl Artistic.
Much Faster! Compiles to LINQ Expression trees (and from there to IL).
Breaks compatibility with version 1 in favour of closer alignment with Paul Graham’s Arc language.
Dot notation for .NET calls (console.writeline “Hello World”).
Smaller, better structured, more extensible code base.

2008 Microsoft MVP Award!

2008-07-01T00:00:00+00:00

I’ve just got an email saying that I’ve received a Microsoft Most Valuable Professional Award for 2008 for my work with Windows Live Services.

This is the second year that I’ve been awarded an MVP and I really didn’t expect it this time around.

Light Up the Web

2008-05-31T00:00:00+00:00

My article on Microsoft Silverlight Streaming by Windows Live was published by CoDe focus a couple of weeks ago to coincide with the Mix conference in Las Vegas. The magazine was in the delegate pack. Now it’s available online too.

Guest Lecture at Essex University

2008-02-23T00:00:00+00:00

Yesterday I had the opportunity to talk to some second year Computing students about “Software plus services”, and to demonstrate some of the work we’ve been involved with at AWS.

Demos included the Windows Live Quick Applications, Virtual Earth and a Facebook application.

Here are the slides.

My thanks to Iain Langdon for the invitation – I really enjoyed it.

Alphabetical List of Arc Functions

2008-02-08T00:00:00+00:00

I’m still trying to get a handle on Arc syntax so I wanted a list of all the Arc functions.

In arc.arc, I modified def like this:


(set *functions* nil)
(set def (annotate 'mac
            (fn (name parms . body)
              `(do (sref sig ',parms ',name)
	      	   (set *functions* (cons ',name *functions*))		
		(safeset ,name (fn ,parms ,@body))))))

Now, every time a function gets defined, its name gets put onto the functions list.

I modified as.scm to comment out the loading of libraries, so that just arc.arc is loaded.

;(aload "libs.arc")

Now when I start Arc I can say:


(sort (fn (x y) (< (coerce x 'string) (coerce y 'string))) *functions*)

And I get a list of all the Arc functions defined in arc.arc:


(<= >= abs acons adjoin alist all alphadig alref andf assoc atend atom avg before best bestn caar cache cadr carif caris cddr commonest compare consif copy count counts date dedup deq dotted downcase ellipsize empty enq enq-limit ensure-dir ero even expand-metafn-call expand= expand=list fill-table find firstn firstn-that flat idfn insert-sorted inst intersperse isa isnt iso join keep keys last list listtab load load-table load-tables map map1 mappend maps max median mem memo memtable merge mergesort metafn min mismatch most multiple no nonwhite nthcdr number odd only ontree orf pair parse-format plural pos positive pr prall prn prs punc qlen qlist queue rand-string random-elt range read read-table readall readfile readfile1 readline readstring1 reclist recstring reduce reinsert-sorted rem rev round roundup rreduce safe-load-table saferead save-table setforms since single some sort split splitn string subseq sym tablist temload temloadall templatize temread testify to-nearest trav tree-subst trues tuples upcase vals warn whitec write-table writefile1)

Now we just need to write some documentation…..

UPDATE

Thanks to contributors on the arclanguage.org forum, it turns out you can get better results without hacking the source – Try this:


(each key (sort < (keys sig))
      (prn key " " (sig key)))

Arc Development using Emacs on OS X

2008-01-31T00:00:00+00:00

In case you hadn’t noticed, Paul Graham has released Arc. I think it’s a great start. I enjoyed hacking LSharp, but Arc has to be a great way forward for innovative Lispers. Maybe somebody will port it to Scheme on .NET one day.

This is how I’ve set myself up to start developing Arc programs on my MacBook.

I have Xcode tools and MacPorts already installed.

First I created a directory for all my Arc stuff.

mkdir arc cd arc

Then I downloaded and built MzScheme


curl -O http://download.plt-scheme.org/bundles/352/mz/mz-352-src-unix.tgz
tar -zxvf mz-352-src-unix.tgz
cd plt/src/
./configure
make
make install

Back to my arc directory

cd ../..

Then I grabbed the code for Arc itself.

curl -O http://ycombinator.com/arc/arc0.tar tar -xvf arc0.tar

Then I created myself a little shell script in the arc0 subdirectory called arc.sh

#!/bin/sh cd ~/arc/arc0 ~/arc/plt/bin/mzscheme -m -f as.scm

Then I could run an interactive Arc session


rob-blackwells-macbook:arc0 reb$ ./arc.sh
Use (quit) to quit, (tl) to return here after an interrupt.
arc> (prn "hello world")
hello world
"hello world"
arc>

Great, but I really need an editor and a way of interacting with the REPL. Enter Emacs and Inferior Lisp Mode.

I put the following in my ~/.emacs file

(show-paren-mode t) (setq inferior-lisp-program "~/arc/arc0/arc.sh")

Hey Presto, now when I start Emacs I can type:

M-x run-lisp

I can edit an Arc program in another buffer and set it to lisp-mode

M-x lisp-mode

I get bracket matching and colour highlighting.

I can send expressions from my edit buffer to Arc running as the inferior Lisp process using

C-x C-e

Happy days!

FTSE 100 Web Servers

2008-01-19T00:00:00+00:00

I wanted to do some independent research about the web server technologies employed by some of the top UK companies.

I used curl to send an HTTP HEAD request to each of the sites for companies in the FTSE 100, and looked at the server header returned. Results were as follows:

40% Microsoft
32% Apache
18% Unspecified
5% IBM

My guess is that several of the unspecified servers are really Apache, but it’s hard to say.

Here is the gory detail (pipe separated fields).

www.3igroup.com/|
www.abbeynational.co.uk/|Server: IBM_HTTP_Server
www.alliance-leicester.co.uk/|Server: Microsoft-IIS/6.0
www.alliance-unichem.com/|Server: Apache
www.allieddomecqplc.com/|Server: Apache/1.3.27 (Unix)
www.amersham.co.uk/|Server: Apache/2.0.52 (CentOS) mod_perl/1.99_16 Perl/v5.8.5 DAV/2 mod_python/3.1.3 Python/2.3.4 mod_ssl/2.0.52 OpenSSL/0.9.7a
www.amvescap.com/|Server: Apache
www.angloamerican.co.za/|Server: Microsoft-IIS/5.0
www.abf.co.uk/|Server: Microsoft-IIS/6.0
www.astrazeneca.com/|Server: Microsoft-IIS/5.0
www.aviva.com/|Server: Microsoft-IIS/6.0
www.baa.co.uk/|Server: Apache
www.baesystems.co.uk/|Server: Microsoft-IIS/6.0
www.barclays.co.uk/|Server: Apache
www.bgplc.com/|Server: Apache/1.3.26 (Unix)
www.billiton.com/|
www.boc.com/|Server: Microsoft-IIS/5.0
www.boots.co.uk/|Server: Apache
www.bpamoco.com/|Server: Web Server
www.bbg.co.uk/|
www.british-airways.com/|Server: Apache
www.bat.com/|Server: Lotus-Domino
www.britishland.com/|Server: Microsoft-IIS/6.0
www.sky.co.uk/|Server: IBM_HTTP_SERVER/1.3.19 Apache/1.3.20 (Unix)
www.bt.com/|Server: BT Web Server
www.bunzl.com/|Server: Apache/1.3.33 (Debian GNU/Linux) mod_throttle/3.1.2 PHP/4.3.10-16
www.cadburyschweppes.com/|Server: Microsoft-IIS/6.0
www.canarywharf.com/|Server: Microsoft-IIS/6.0
www.capitagroup.co.uk/|
www.centrica.co.uk/|Server: Microsoft-IIS/6.0
www.granada.co.uk/|Server: Apache/1.3.27 (Unix) (Red-Hat/Linux)
www.dmgt.co.uk/|Server: Apache/2.0.52 (Red Hat)
www.diageo.com/|Server: Microsoft-IIS/6.0
www.dixons.com/|Server: Apache/2.0.59
www.emap.com/|Server: Microsoft-IIS/6.0
www.exel.com/|Server: Microsoft-IIS/6.0
www.friendsprovident.co.uk/|Server: FPWS/1.0
www.gallaher-group.com/|Server: Microsoft-IIS/6.0
www.gknplc.com/|Server: Microsoft-IIS/6.0
www.gsk.com/|Server: IBM_HTTP_Server
www.granadamedia.com/|Server: Microsoft-IIS/6.0
www.gusplc.com/|
www.hanson.co.uk/|Server: Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.2.3 PHP/5.1.2-1
www.hays.co.uk/|
www.hbosplc.com/|Server: Microsoft-IIS/6.0
www.hiltongroup.com/|Server: Apache/1.3.33 (Debian GNU/Linux) PHP/4.4.2-0.dotdeb.1 mod_perl/1.29
www.hsbc.com/|Server: IBM_HTTP_Server
www.ici.com/|Server: Microsoft-IIS/6.0
www.imperial-tobacco.com/|Server: Microsoft-IIS/6.0
www.invensys.com/|Server: Microsoft-IIS/6.0
www.matthey.com/|Server: Apache/1.3.29 (Linux/SUSE) PHP/4.3.4 mod_python/2.7.10 Python/2.3.3 mod_perl/1.29 mod_jk/1.2.15
www.kingfisher.co.uk/|Server: Microsoft-IIS/6.0
www.landsecurities.co.uk/|Server: Microsoft-IIS/5.0
www.legal-and-general.co.uk/|Server: IBM_HTTP_Server/6.0.1 Apache/2.0.47 (Win32)
www.libertyinternational.uk.com/|Server: Apache-Coyote/1.1
www.lloydstsbgroup.com/|
www.mangroupplc.com/|Server: Microsoft-IIS/6.0
www.marks-and-spencer.com/|Server: Server
www.mm02.com/|Server: Apache/2.0.49 (Unix) PHP/4.3.9
www.morrisons.plc.uk/|
www.ngc.co.uk/|
www.next.co.uk/|
www.northernrock.co.uk/|Server: Microsoft-IIS/6.0
www.oldmutual.com/|Server: Apache-Coyote/1.1
www.p-and-o.com/|Server: Oracle-Application-Server-10g/9.0.4.1.0 Oracle-HTTP-Server OracleAS-Web-Cache-10g/9.0.4.1.0 (N)
www.pearson.com/|Server: Microsoft-IIS/6.0
www.prudential.co.uk/|
www.reckitt.com/|Server: Microsoft-IIS/6.0
www.reed.com/|Server: Apache/2.2.6 (Fedora)
www.rentokil-initial.com/|Server: Apache
www.reuters.com/|Server: Apache
www.rexam.co.uk/|
www.riotinto.com/|Server: Microsoft-IIS/6.0
www.rolls-royce.com/|Server: Apache
www.royal-and-sunalliance.com/|Server: Apache-Coyote/1.1
www.royalbankscot.co.uk/|Server: Microsoft-IIS/6.0
www.sabmiller.com/|Server: Microsoft-IIS/6.0
www.safeway.com/|Server: Microsoft-IIS/5.0
www.sage.com/|Server: Microsoft-IIS/5.0
www.sainsburys.co.uk/|Server: Microsoft-IIS/6.0
www.schroders.com/|Server: Microsoft-IIS/6.0
www.scottish-newcastle.com/|
www.scottish-southern.co.uk/|Server: Microsoft-IIS/6.0
www.scottishpower.plc.uk/|
www.severn-trent.com/|Server: Apache
www.shell.com/|Server: Sun-ONE-Web-Server/6.1
www.shiregroup.com/|Server: Apache-Coyote/1.1
www.sixcontinents.com/|
www.smith-nephew.com/|Server: Apache/2.0.49 (Unix) mod_perl/1.99_13 Perl/v5.8.4 mod_ssl/2.0.49 OpenSSL/0.9.7d DAV/2 mod_jk2/2.0.2
www.smiths-industries.com/|Server: Microsoft-IIS/6.0
www.stanchart.com/|Server: Apache
www.tesco.co.uk/|
www.tomkins.co.uk/|
www.unilever.com/|Server: Microsoft-IIS/5.0
www.unitedutilities.com/|Server: Microsoft-IIS/6.0
www.vodaphone.com/|Server: Apache/2.0.55 (Unix)
www.whitbread.co.uk/|
www.wolseley.com/|Server: Microsoft-IIS/6.0
www.wpp.com/|Server: Microsoft-IIS/6.0
www.xstrata.com/|Server: Apache

Arc Likely To Be Open-Sourced This Winter

2008-01-15T00:00:00+00:00

“Btw, will Arc ever be open-sourced?
I’d love to just sudo apt-get install arc in this lifetime.”

“For once, I will answer this question: yes, soon. This winter, I hope.”

This one line from Paul Graham is possibly the most interesting tech news so far this year

Let’s hope it lives up to the massive expectation.

Build a Silverlight Media Player

2008-01-10T00:00:00+00:00

My article Build a Silverlight media player is live on RegDeveloper.

It explains how to use Expression Blend and Visual Studio to build a simple, but realistic Silverlight application and host it in the Silverlight Streaming Service by Windows Live.

Thanks to Helen Yousaf for permission to use her copyright material in the application.

My brother Peter Blackwell is a session musician; he plays guitar on the track.

Have you got a Map Mate?

2007-12-17T00:00:00+00:00

Richard’s fatastic MapMate application is now live on Facebook.

It’s also the featured site on Via Windows Live.

MapMate is a new way to update your Facebook status. You can plot your current location and set your status anywhere using Microsoft’s Virtual Earth.

Try it now http://apps.facebook.com/mapmate/

Volta: Are the Materials in the Room Good Enough?

2007-12-09T00:00:00+00:00

I had a quick look at Microsoft Live Labs Volta today.

Described as â€œweb development using only the materials in the roomâ€?, it allows you to build multi tier web applications using a consistent programming model; only later do you say how the application is split across the tiers and exactly what kind of client technology you want.

Volta includes some very interesting demos, most notably the â€œVirtual Earth SDK – Volta Styleâ€?. All the code is written in C# (not JavaScript!) and it seems like a really interesting abstraction. Performance seems to be a problem on my rural broadband link â€“ start up time seems slow for some reason.

I went on to read Erik Meijer’s OOPSLA 2007 Paper, “Democratizing the Cloud” and that got me thinking about where all this is all going…

Erik is absolutely right about current programming models being “Gratuitously hard” and he raises some good points:

We need a common language and libraries across all the tiers.
Tier partitioning decisions need to be reversible and easy to change.
Just like Visual Basic democratised Windows programming, it would be great to democratise web programming for “ordinary programmers”.

I think there are some practical problems on the client tier though – Silverlight apps and AJAX apps have very different capabilities â€“ how can we avoid lowest common denominator interfaces?

I still think that AJAX + lots of JavaScript is a short to medium term hack, which we ought to find a way of stopping â€“ weâ€™ll regret it in a few years.
ASP.NET Web forms already tried to democratise web development by allowing Windows forms programmers to put down buttons and define event handlers on web pages â€“ it doesnâ€™t really work. Can Volta really abstract away pesky issues like the page lifecycle?
Should we really be using JavaScript as a compiler target?

Also, if weâ€™re going to choose a consistent language and model across all the tiers, is C# and CLR the right choice? Shouldnâ€™t we be looking for something with better support for distribution and parallelism? A functional programming model, maybe? Something that would allow us to use multiple processors / cores in each of the tiers too?

I think we need to recognise that whilst the Web browser is ubiquitous, it is also “broken”. Itâ€™s just not designed for the kind of rich, interactive applications we want to build today. Conventional wisdom seems to be that we have to work with what we have and hack around it. I’m not so sure. There has been huge and rapid adoption of other web-oriented applications including iTunes, Flash Player and Silverlight. Maybe we do have a second chance on the client?

If we did have a clean sheet of paper, what would the solution look like then?

TechEd Developers 2007 Summary

2007-11-10T00:00:00+00:00

I didn’t make it to Barcelona until Tuesday so I wasn’t at the Keynote. By all accounts I didn’t miss much. The good news was that Microsoft will release Visual Studio 2008 and the .NET Framework 3.5 by the end of November 2007.

As is often the case, TechEd was as much about catching up with people as the sessions themselves. It was good to go out on the Sidra with, amongst others Richard Griffin, Una Wlash, Jennifer Forsythe and Johannes Kebeck.

Lots of interesting stuff, but here are some thoughts from my homeward bound plane journey:

ASP.NET MVC will be out as a CTP soon and is likely to be rolled into the mainstream product by middle of next year. I think that having a very extensible Model View Controller pattern will allow for much more of an engineering approach to ASP.NET apps. I’ve always thought that Web Forms is the wrong abstraction. I think this framework will fix all that, and lead to more maintainable and extensible web apps. A big issue is how to support existing web controls. To be honest I think they should ditch those and start again. Anybody considering the jump to Ruby on Rails might like to take a look at this and reconsider.

I’ve done some fun things with XSLT in the past, but building XSLs doesn’t come naturally to me. Dave McMahon’s talk has inspired me to think about it some more. To be honest the talk started out looking like it was going to be rather dull, but those attendees that persevered were rewarded with some very cool demos and code. I didn’t know that Visual Studio could debug XSLT and I was surprised at the opportunities for hacking XSLT from .NET. Generating XAML using XSLT makes an awful lot of sense.

Andy Wigley gave us a comprehensive tour of Windows Mobile 6 from a programmerâ€™s perspective. The tool support in VS2008 all seemed to be much improved. Iâ€™m tempted to do more with Mobile, but also reluctant to invest too much time and effort. Surely someday soon, we’ll get full blown .NET CLR . The hardware is getting rapidly more powerful. So is it worth waiting rather than struggling with missing features? I guess I’m just frustrated that I can’t port LSharp to the mobile platform because of missing Reflection features. Sun seem to have dropped their Mobile Java in favour of full blown JVM. When will Microsoft do the same?

The .NET Micro Framework looks very cool. I like the idea that a software guy like me could start prototyping hardware devices without much recourse to a soldering iron.

Silverlight seems to be getting all the glory, but maybe we’ll see more WPF apps following the fantastic demos by the Blendables guys.

WCF seems to bring a whole lot of simplicity to the communications stack and now, with support for REST and for AJAX end points I think we’re going to be using more of it.

If time was more plentiful Iâ€™d certainly be thinking about building a LISP or LSharp IDE using Visual Studio Extensibility.

Pat Helland gave an interesting talk entitled “The Irresistible Forces Meet the Moveable Objects”. He discussed his thoughts on the future and touched on the many-core (multi core) problem. He made some interesting predictions about future hardware (Disk is Tape, Flash is disk, Tape is dead) and about future data centres. I liked the fact that he tried hard to measure, and provide statistics for, what he was talking about. (E.g. only 15% of the cost of a data centre is the building – 40% is power). It was the first time I’d heard about Sun’s data centre in a 20-foot shipping container. I still think multi core is a huge problem. There are lots of things going on at Microsoft to look at it â€“ Transactional Memory, Joins library, F# etc. but I didn’t find anybody who had the big picture.

LINQ is undoubtedly powerful, but I canâ€™t help wondering about how C# and VB are evolving to add new language features. I was interested in Claudio Russo’s talk on C Omega and joins etc, but here we are hacking the language again. Perhaps we should be looking at the original programmable programming language (LISP) and seeing if there is a way to add Macros to C#?

Sagrada Familia Magic Square

2007-11-10T00:00:00+00:00

Did any TechEders notice this magic square at La Sagrada Familia in Barcelona this week? It’s on the wall to the left of the front entrance.

Anybody know the significance of 33?

Get started with Silverlight

2007-11-06T00:00:00+00:00

My first article for Reg Developer entitled “Get started with Silverlight” has just been published here.

I’m hoping to do some more tutorial style pieces about Microsoft related technologies over the coming months.

My thanks to Gavin Clarke for his encouragement.

Redmond Racers

2007-11-04T00:00:00+00:00

It wasn’t all work on my trip to Microsoft last week.

Actually, it was a great opportunity to catch up with various folks from Windows Live, Virtual Earth, Photosynth and ASP.NET.

I think Mix 08 is going to be pretty exciting …

Microsoft Windows Live Quick Apps Campaign

2007-10-24T00:00:00+00:00

Microsoft UK has launched a new campaign to raise awareness of the Windows Live Quick Applications.

See http://msdn2.microsoft.com/en-gb/live/bb880951.aspx

It’s certainly having a big impact on downloads!

OSI Approves Microsoft License Submissions

2007-10-16T00:00:00+00:00

The OSI has formally approved the Microsoft Public License (MS-PL).

Perhaps now I wont keep getting “corrected” by Microsoft people when I say that Windows Live Quick apps are available as open source. ;-)

Slides from Mix UK 07

2007-09-15T00:00:00+00:00

My slide deck from Mix UK 07 is available for download here – Out of interest, the PowerPoint file is hosted on Windows Live SkyDrive Beta .

Here are the resources / links that I referred to in the talk:

dev.live.com/quickapps
www.codeplex.com/liveinabox
www.popfly.com
www.easygps.com
www.sportsdo.net
www.conchango.com
www.liveside.net
www.viawindowslive.com

I understand that some follow up materials will be posted to www.microsoft.com/uk/mix07/

Great conference – worth attending just to see Simon Peyton Jones tackling one of the most important problems in Computer Science today.

Mix:UK 07

2007-09-06T00:00:00+00:00

I’ve been working on last minute preparations for Mix UK, London next week.

If you haven’t already booked, you are probably too late – the event is now sold out. If you are coming, you are probably in for a treat.

My session “Building Web Solutions using Windows Live Quick applications” is on Wednesday at either 14:00 or 15:15 depending on which particular Microsoft information you happen to be reading – presumably, all will become clear on the day.

If you are a hardcore Windows Live developer, I recommend that you skip my session and go and see Danny Thorpe and Angus Logan’s two part “Building next generation web applications using Windows Live Services”.

If you want to see some Windows Live demos and simple code samples to get you started, then Live Quick Apps is for you.

Una Walsh from Conchango will be joining me for the demos. Una was responsible for the design, user experience and much of the creativity on the Quick Apps.

We’re also hoping that Jonathan Greensted will show some code from his TrackMe product … based on Windows Live Quick Apps.

For more information , see http://www.microsoft.com/uk/mix07

Energy Trading Robots

2007-09-03T00:00:00+00:00

I’ve just finished a really interesting project for a big player in the UK Energy sector, working with the guys from Planlogic .

Hamish is an expert in trading strategy and in particular, Trayport GlobalVision and the RORC trading exchange from Reuters – he came up with the clever algorithms, which I then helped to codify. We’ve now got some nice abstractions and what seems to be a good, reusable framework implemented in .NET.

Basically we built an automated trading system where a trader can set up a price book with a variety of limit orders and associated trading rules. These prices are hidden from the market, but the software we wrote constantly monitors the market changes and hits prices if the rules match. Obviously, time-to-trade is a key measure so there is a real challenge in balancing the needs of readable / maintainable / bugfree code versus high performance and fast execution. The code had to be good too – mistakes could be costly, so we’re using some automated testing based on NUnit.

Energy trading is still relatively young in the UK (complete liberalisation didn’t occur until the late nineties) and so there are still lots of opportunities for decision support technology. I think we’ll see lots more “robots” in the coming years and I wonder how that will play out. Will energy trading just become a competition to see who can build the fastest or cleverest robot?

Somehow, I dont think we’ll ever replace human beings – there are still many things that we cant capture in code – common sense and gut-feel being good examples. Also, I think that humans are still probably the best market makers.

Angus gets slashdotted!

2007-08-17T00:00:00+00:00

Announcement of the new Windows Live Id Web Authentication API is on Slashdot. http://developers.slashdot.org/article.pl?sid=07/08/17/0042224

Windows Live Quick Applications

2007-07-10T00:00:00+00:00

Greetings from the Microsoft Worldwide Partner Conference, here in Denver Colorado.

Today, Microsoft is announcing the first two Windows Live Platform Quick Applications, Contoso University and Contoso Bicycle Club. (This is the thing that’s been keeping me busy lately!)

More information here

The sites themselves are available at

http://contosouniversity.mslivelabs.com/

and

http://contosobicycleclub.mslivelabs.com/

Dont forget to install Silverlight first.

Basically these applications demonstrate how you can use Windows Live Services in your own web applications. The great thing is that Microsoft is releasing them under an open source license (MSPL). Source code available on Codeplex

I’m speaking as part of ISV011 at 16:30.

A Microsoft statement about GPLv3

2007-07-07T00:00:00+00:00

http://www.microsoft.com/presspass/misc/07-05statement.mspx

“Microsoft remains committed to working with the open source software community” as long as they arent using the GPL.

Actually I really wanted to be a lawyer but it turned out that I wasnt smart enough so I had to go into software R&D instead ;-)

Microsoft Most Valuable Professional

2007-07-02T00:00:00+00:00

I’ve just heard that I’ve received a 2007 Microsoft MVP Award for “Windows – Virtual Earth/MapPoint” !

Looks like I’m buying the beers at lunchtime …

MSR MapCruncher with Commercial Use Terms

2007-06-27T00:00:00+00:00

Microsoft has released a new version of MapCruncher that doesnt have the “for non commercial use only” clause in the license.

At the time of writing, it’s pretty hard to find from either Live Search or Google, so here is the link.

http://research.microsoft.com/research/downloads/details/66596bbd-0444-4ab1-93bc-e41f010a237d/details.aspx

RNLI Video on Silverlight Streaming

2007-06-25T00:00:00+00:00

Last year, Microsoft commissioned a video of our work with the RNLI on a fishing safety system now known as MOB Guardian.

Iâ€™ve uploaded the video to the Microsoft Silverlight Streaming Service as a test and you can check it how here.

You need to install Silverlight if you haven’t already done so.

Convert GPX to GeoRSS using XSLT

2007-06-24T00:00:00+00:00

I have a Garmin eTrex Legend handheld GPS. It’s been in the drawer for the last year or more and I’ve lost the software and manuals.

I need to record a track log for an upcoming project, export the track log to the PC and then visualise the route using Microsoft Virtual Earth. This turned into an interesting Sunday afternoon hack ..

1. eTrex instructions easily available from the Garmin site – why cant all hardware manufacturers do this?

2. EasyGPS comes to the rescue for transferring the tracklog – It’s easy to install and set up and within a matter of minutes I have a GPX file (GPS eXchange format).

3. I need to turn GPX into GeoRSS. Both are XML formats, so an XSLT stylesheet comes to the rescue. gpx2georss.zip

4. Apply the stylesheet to the GPX file. On a UNIX box or under Windows Cygwin, you can use xsltproc.

% xsltproc gpx2georss.xsl mytrack.gpx > out.xml

I also discovered that you can do it on a Windows command line with the MSXSLT parser available here.

d:> msxsl.exe mytrack.gpx gpx2georss.xsl > out.xml

5. Take the out.xml file and plug it into the Virtual Earth control using the modified GeoRSS sample, available from the Virtual Earth Interactive SDK here.

Here are some dog walks as proof that it works …

develop/discover/design

2007-06-01T00:00:00+00:00

My Microsoft Virtual Earth tutorial article has been published in this month’s .net magazine.

http://www.netmag.co.uk/zine/latest-issue/issue164

Revolutionising the Property Market with Online Mapping Technology

2007-05-17T00:00:00+00:00

I’m quoted in the following Microsoft press release about Virtual Earth and its applications.

http://www.microsoft.com/emea/presscentre/pressreleases/VirtualEarthTS_952007.mspx

Microsoft takes on the free world

2007-05-14T00:00:00+00:00

Two weeks ago, I was at Mix07 in Las Vegas at Microsoft’s invitation, thinking that the world had changed:

Hearing about the importance of being cross platform;
Seeing things like Ruby apps and Apple Macs in a keynote;
Talking to people who think that “Microsoft is now less evil than Google”;
Watching the Pussycat Dolls courtesy of Microsoft.

Then this article comes along to bring me back down to earth.

How the birds-eye imagery is done

2007-04-23T00:00:00+00:00

The Gadget Show recently had a clip talking about Live Search and how the birds-eye imagery is done.

Accessing WMS from Virtual Earth

2007-04-16T00:00:00+00:00

My article Accessing WMS from Virtual Earth has just been published on www.viavirtualearth.com.

It shows how to access Web Map Server (WMS) servers from Virtual Earth using a custom tile server.

My earlier article Roll Your Own Tile Server has some minor updates too.

ILC07 Underway

2007-04-01T00:00:00+00:00

The International Lisp Conference 2007 kicked off in Cambridge this morning with an excellent tutorial on Hunchentoot and Friends by Edi Weitz.

Edi was using LispWorks on Windows. He has a Starter pack which explains how to get up and running. The LispWorks folks were giving away free evaluation licenses.

Edi used Hunchentoot to develop a sample application that allowed users to enter their name and age. He then extended the application to show various features such as authentication and cl-who etc.

The Ajax demo was impressive.

In terms of production, Edi mentioned delivering to Linux with SBCL and detachtty. Also using LispWorks to produce a stand alone EXE including the web server.

Hunchentoot-test is really a suite of demos.

Performance looks to be good – there may be some performance issues because of flexi-streams. I’m a bit concerned about what would happen if a hunchentoot server experienced the slashdot effect – would it just keep spawining worker threads? Good to hear that Edi has already used Hunchentoot to deliver a number of serious commerical sites.

One of the big attractions for me is the ability to bring up a toploop onto a running web server and start debugging and tweaking without bringing the web server down. Also, of course, the amazing productivity that CL provides!

You may also be interested in The Ilc07 Group Blog.

Virtual Earth Partner Day

2007-03-22T00:00:00+00:00

Yesterday I spoke at the Microsoft Virtual Earth Partner Day at Sonning. Lot’s of enthusiastic people doing some very cool projects.

Was good to catch up with Dr Neil and Tricky again – Dr Neil also talked about Windows Live Services at an MSDN event in London today.

Microsoft Campus, Redmond

2007-02-17T00:00:00+00:00

Just been on a whirlwind trip to Microsoft, Redmond with Simon Davies , Ian Moulster and Alistair Beagley.

It felt a bit like shooting a channel 9 video – walking around Microsoft Campus, knocking on office doors, but we were fortunate enough to meet some really interesting people, including Tim Sneath ,Stephen Lawler, Jon Lam, Martin Maly, John Dietz,John Montgommery, Ankur Chavda, Aric Weiker, Koji Kato and Catherine Heller.

Cant say much due to NDA, but suffice it to say that there are some very smart people doing some very cool things. I think the mash-up approach has the opportunity to revolutionise the way that we build software.

Essex University

2007-02-09T00:00:00+00:00

This morning I gave a guest lecture about Web based GIS and Virtual Earth at the Computer Science dept, Essex University .

Slides should shortly be available on your intranet.

If anybody from the audience is interested in working with AWS on a summer placement, please email us.

Building Mashups with Virtual Earth and Windows Live services #2

2007-01-23T00:00:00+00:00

Back by popular demand, Simon and I will be talking about Windows Live Services and Virtual Earth in London on 8 Feb.
More Information >>

BCS Awards 2006

2006-12-09T00:00:00+00:00

On Thursday, Active Web Solutions won three British Computer Society Awards for our work on the RNLI Sea Safety System.

Here’s a picture of me picking up the Mobile Computing Award:

Here are some related links:

http://www.computerweekly.com/Articles/2006/12/12/220408/bcs-awards-technology-to-the-rescue.htm

http://www.computerweekly.com/Articles/2006/12/12/220409/bcs-awards-rnli-nets-win-on-the-move.htm

Mashups - Slides and Code

2006-12-01T00:00:00+00:00

Following on from the “Building Mashups with Virtual Earth and Windows Live Services” event in London on Tuesday, here are the slides and the demonstrations from my talk:

Demos.zip

Mashups281106.ppt

Demos.zip includes the tile server demo.

For more information about tile servers, see my MSDN article.

Thanks to everybody who attended. There were several GIS people there and I really enjoyed the discussions in the interval and after the event.

Thanks to Simon, Ian and all the Microsoft folks who organised the event.

Anglia IT Managers Forum

2006-12-01T00:00:00+00:00

On Wednesday I gave a talk about Web based mapping at the Anglia IT Managers Forum.

Here are my PowerPoint slides.
(Approx 7Mb).

Building Mashups with Virtual Earth and Windows Live services.

2006-11-21T00:00:00+00:00

Building Mashups with Virtual Earth and Windows Live services – London, 28 November

I’ve been invited to talks at this event. I’ll cover building simple VE apps, business challenges, MSR MapCruncher and custom tile servers.

More info and registration.

LSharp and Windows Presentation Foundation, WPF.

2006-11-16T00:00:00+00:00

Although time has been limited, I did manage to do some LSharp hacking whilst at TechEd Europe last week.

Microsoft released the .NET Framework Version 3.0 which includes Windows Presentation Foundation (formerly Avalon) technology.

Anyway, to cut a long story short, I’m now running and developing LSharp under Windows Vista RC2. Both Emacs and LSharp seem to run without any problems. The OS is reported as Microsoft Windows NT 6.0.5744.0.

The WPF example is included in subversion under the download directory. See wpf.ls and calc.xaml. You need to compile and run the latest LSharp from Subversion and also install the .NET Framework 3.0 runtime redistributable.

One issue was around the use of STAThread. It seems that WPF requires a single threaded apartment model, so I’ve modified spawn to make this an optional argument.


(reference "presentationcore")
(reference "presentationframework")
(reference "windowsbase")
(reference "system")
(reference "system.xml")

(using "system.windows")
(using "system.io")
(using "system.windows.markup")

;; Load and instantiate objects from a XAML file
(defun load-xaml (filename)
      (with (f (new filestream filename (Open FileMode))
	element (call load xamlreader f))
	(close f)
	element))

(defun main()
  ;; Define a new application
  (= app (new application))

  ;; Define a window based on a XAML Description
  (= win (the window
	   (load-xaml "d:\\code\\lsharp\\lsharp.org\\download\\calc.xaml")))

  ;; Run the application with the new window
  (run app win))


;; WPF Requires  [STAThread]
(spawn
 (main)
 (sta system.threading.apartmentstate))

Virtual Earth API Version 4

2006-11-09T00:00:00+00:00

The New Virtual Earth API Version 4 launched this week with a new Active X control which supports 3D Maps. It’s very cool and allows Google Earth like features with the advantage that the control is browser hosted.

The good news for mapcrunchers and custom tile servers is that you get 3D for free on your own maps just by changing a few lines of code.

Assuming you are using
tileSourceSpec.TileSource = "http://mytileserver/..."

Then update the script refrence to use the Version 4 API "http://dev.virtualearth.net/mapcontrol/v4/mapcontrol.js" .

Then map.SetMapMode(VEMapMode.Mode3D); once you’ve loaded your map.

Here is an Admiralty chart in 3D delivered from one of my custom tile servers:

We had dinner last night with Jennifer, Johannes and Jaap who were manning the VE stand at TechEd in Barcelona.

Changes to the Virtual Earth API

2006-09-15T00:00:00+00:00

Microsoft has made some changes to the Virtual Earth API. Cool new features include support for custom tiles and for coloured polygons.

Unfortunately, these changes have broken the example from my ariticle Roll Your Own Tile Server

I’ve updated the ve.zip file so it’s back working. If you’re interested, I suggest you take a look atthe revised veapi.js file.

Thanks to Gordon Hooton for pointing this out.

LSharp Discussion Group

2006-08-12T00:00:00+00:00

Following coverage on Reddit and recent emails, I’ve created a Google Group for LSharp.

BCS Medalist

2006-07-25T00:00:00+00:00

We went to the British Computer Society in London today to hear the press annnouncement that Active Web Solutions and the RNLI Sea Safety Project are medalists in this years awards.

Lunch with Dr Neil

2006-07-24T00:00:00+00:00

We had lunch with Dr Neil of Via Virtual Earth fame, Pete Williams from Microsoft and Tricky today. We’re hoping to work together on some interesting projects.

Roll Your Own Tile Server

2006-07-23T00:00:00+00:00

My article Roll your own Tile Server is now live on the Via Virtual Earth site.

MEDC Europe 2006

2006-06-09T00:00:00+00:00

This is me on the Microsoft MapPoint stand at the Microsoft Mobile and Embedded Developers Conference 2006 in Nice, France.

We were demonstrating our GeoPoint location based services platform which uses Microsoft Virtual Earth in a mash up with data from satellite transponders and GPS enabled mobile phones.

If you haven’t already done so, check out the Virtual Earth technology at Windows Live Local.

If you’re a programmer take a look at the Virtual Earth Interactive SDK and see the Via Virtual Earth site.

Peter Seibel Video

2006-05-22T00:00:00+00:00

Pete Seibel, author of Practical Common Lisp talks about:

How language affects the way you think.

The double dispatch problem and how it goes away with generic functions.

Why the conditions system is better that try catch.

Google Video

Seeing the World Through Rose-Tinted Spectacles

2006-04-19T00:00:00+00:00

You may have noticed that the glasses I wear have a pink tint. Contrary to popular opinion, this isn’t a fashion statement; they stop me getting headaches when spending long periods of time in front of a computer monitor. They also make reading faster and more comfortable. The same principle is apparently used with dyslexic kids.

My optician recommended that I try Screen Tinter from Thomson Software Solutions. The software makes it easy to tweak window colour backgrounds. Setting Red: 255 Green 230 and Blue 255 does it for me!

It’s free as in beer, but only works on the Windows platform.

World Jamboree 1929

2006-04-16T00:00:00+00:00

This certificate was awarded to my Grandfather in 1929.

To A E Rowe Esquire who worked unselfishley and wholeheartedly behind the scenes at the Boy Scouts’ World Jamboree 1929 and thereby contributed so effectively towards its success, I offer the grateful THANKS of the whole Scout brotherhood. Baden Powell of Gilwell. Chief Scout.

Ordinary Telegrapher AER Rowe

2006-04-16T00:00:00+00:00

When my Grandfather was posted to Skegness for training in 1941, he wrote to his Dad at Romford. Here is the text of the letter with some of the images from the Butlins letter card. I have tried to preserve grammatical errors and use of language; I think these are an indication of the pressure and uncertainty he would have been experiencing. For information, the 29th was a Saturday.

Uncle W is Uncle Will White married to Elizabeth who I think was Grandad Rowe’s sister. The families were very close and used to stay with each other quite often I believe. They were spiritualists and took Dad to their meetings sometimes! [Janet Palmer]

O/TEL AER ROWE
JX 310177
TOP DIVISION
CLASS 137
HMS Royal Arthur
Skegness Lincs

29 November 1941

Dear Dad and all at Romford,

I expect you will be anxiously awaiting a line from me as you knew, of course, I was here last Tuesday. I hope you wonâ€™t think me thoughtless in having delayed so long writing but I expect Dad will remember that the rookie has quite a lot to do + learn in his first few days + in addition, of course, I was summoned somewhat hurriedly + I’ve to put my affairs in order as best I can from here.

These views may give you some idea but only faint + not so colourful of the place I stay in. We live in chalets + have a good few comforts old soldiers would think but it’s nothing like home.

I seem to do nothing but go on parade “at the double” all the time. My sailor suit is very tight as yet but twill stretch I’m told and it would be quite in order with a few more pockets. I haven’t anywhere to put pipe, bacca + matches. I seem to get on fairly well so far – haven’t been into trouble yet but have been interviewed by ???? Officer as have two other university men here with me in my class.

I expect I may hear about the RAF but I’m afraid there’s little hope for the Air ?? will not ask for my release the Admiralty may release me if I’m not to far on in training â€“ They are quite fine chaps in my class and I shall enjoy myself here I have no doubt tho’, of course I was disappointed at having to stay. I share a chalet with two others one an ex Palestine Police Constable the other a railway clerk. Weâ€™ve one other teacher here + also a man from Westcliffe. In addition two small boys of 12th West Ham who are here as P.T. instructors (They were of cub age when I last saw them) have both stopped me and spoken to me and we’ve had quite a chat – I think I’ve also sometime to interview the naval schoolmaster here. Well that’s all for now my love to you all + remember I think about you quite a lot. Remember me to Auntie Liz + Uncle W when you see them also Hilda when you write – I have to do so much kit arranging + boot cleaning etc + running to parade I’ve little time for letter writing. I had Vaccination and inoculation yesterday + my arm has been very stiff – Had shore leave today + went into Skeg. We have an Ensa concert tonight.

Love to you all your affectionate son and brother, Arthur.

P.S. Expect to be here about five weeks.

P.S. II If Ada has time could and would she knit me a pair of gloves and perhaps a scarf. I’ll square her for the wool. Please. And would you mention this to Maud. It has been cold here.

Better LSharp Params

2006-03-12T00:00:00+00:00

Kelly Murphy has been busy again. We now have &optional and &key parameters as well as an updated Win Forms demo in subversion.

Leppie has been working on an L Sharp to C Sharp compiler / translator .

Leading Edge L Sharp

2006-03-04T00:00:00+00:00

I have been inundated with L Sharp related email over the last few weeks â€“ thatâ€™s great news, except that I havenâ€™t been able to keep on top of it all, so here is a summary of the latest L Sharp progress.

The master source code repository for L Sharp is now available on the Sourceforge Subversion server at

https://svn.sourceforge.net /svnroot/lsharp

This should allow anybody to get the latest source and try out features which are, as yet unreleased. Information about accessing the Subversion repository can be found at http://sourceforge.net/docs/E09

Apart from several bug fixes and improvements, Iâ€™m excited to announce two major new features â€“ event handling and web development. These are still experimental, but early adopters can grab the latest code and give it a try.

Event Handling

You can now properly add LSharp event handlers to controls on Windows Forms applications using handle-event or defevent. Here is a code example:

(reference "System.Windows.Forms") (using "System.Windows.Forms")


(= f (new Form))
(defevent Click f (sender args)

	(writeline console “You clicked x {0} y {1}” (x args) (y args)))

(spawn ’(run application f))

Web Application Development

We now have an HttpHandler which allows L Sharp to be used within ASP.NET applications.

Create a new ASP.NET 2 web application. Create a bin subdirectory and copy in LSharp.dll and LSharp.Web.dll. Modify your web.config file to include


<httpHandlers>

  <add verb=“*” path=“*.lsp” type=“LSharp.Web.LSharpHttpHandler,LSharp.Web”/>

</httpHandlers>

Then write your L Sharp Page as a text file with a .lsp extension. The contents of the file will be passed to the L Sharp evaluator when the page is requested, so it must be a valid L Sharp program. Here is a simple example:
;;; Hello World (write *response“Hello World”)

As you can see, response is bound to the HttpResponse object. request, session and application are available similarly. In addition, context is bound to the HttpContext. Take a look at the MSDN documentation for System.Web.HttpContext, the object model is fairly self explanatory. Also see the example web site included with the L Sharp source distribution.

A Big Thanks

The thing that most excites me about L Sharp at the moment is the momentum that we seem to be gaining. There is a small but growing number of users, and perhaps more importantly, a growing number of contributors.

L Sharp isnâ€™t my day job, so I canâ€™t always respond quickly to email, but I want to thank everybody who has contacted me. I value all the feedback, comments, patches and words of encouragement â€“ Thanks!

Script Your Application with L Sharp

2006-02-17T00:00:00+00:00

My article Script Your Application with L Sharp has just appeared on the CSharpHelp site.

Protocol Design for Very Low Bandwidth Systems.

2006-02-10T00:00:00+00:00

I’ve helped design a couple of protocols for telemetry systems over the last couple of years and I want to summarise some learning points for future reference.

The general advice these days is “don’t design a new protocol”. With so many standards to choose from and with XML supposedly being the common data format, it’s hard to see why you’d want to create a new communications protocol. The trouble is that we’re talking very low bandwidth here (as low as 10 bytes per minute), so every bit has to earn its place.

Reserve a few bits as a message type. This gives you future extensibility; new terminals can add new features whilst the receiver retains backwards compatibility with older models.

It’s surprising how few bits you need to represent apparently complex information. You only need 43 bits for a full GPS position for example.

Consider using checksums or Cyclic Redundancy Checks. CRCs are useful to ensure data integrity. By seeding the CRC bytes with an agreed bit pattern you can achieve a kind of crude digital signature for non repudiation of messages. ISO/IEC 13239 (formerly ISO/IEC 3309) contains a popular CRC algorithm.

Provide a sequence number count. Even if you only have three bits, you can number successive messages from 0 through 7, rolling back to 0 instead of 8. Itâ€™s then easy to detect missed packets or packets delivered out of sequence.

Telemetry systems often report on a periodic basis whilst allowing “polling” to query status at any given time. It’s a good idea to distinguish periodic reports from poll responses to aid diagnosis if a terminal is misbehaving.

Implement diagnostics facilities. Build a mobile terminated message which causes the transponder to reply with a diagnostic packet. Use individual bits to represent the current state of various sub systems. Make 0 mean faulty and 1 mean good so that you assume a fault unless the system actively reports otherwise. Transponders are often deployed to far flung or inaccessible places, so you need to be able to debug them from the comfort of your office.

Consider implementing a reset message to allow remote reboot of systems.

You can implement surprisingly good text messages using six bit ASCII.

Many communications systems are asymmetric both in terms of bandwidth, quality of service and cost. Sometimes itâ€™s cheaper to send an important message three or four times to more or less guarantee delivery rather than implement handshaking acknowledgements.

Consider packet layout. Sometimes aligning bits to byte boundaries can make parser implementations easier, especially in low level computer languages. If certain notions such as priority apply to several different packet types, make sure the same bit positions are used. That way you don’t have to parse the whole packet.

If you have a GPS then you have access to an accurate clock. If you encode just the seconds past the minute within your message, then the receiver can calculate transit time (if it too has an accurate clock) for the cost of just 5 bits.

When considering adding generic features, read End-to-End Arguments in System Design Saltzer et al.

Question security. Is your communications system an open or closed channel? Could it be hacked?

L Sharp 1.2.1 News

2006-01-29T00:00:00+00:00

LSharp 1.2.1 has been released , see www.lsharp.org* . It includes various fixes and improvements, many as a direct result of Hans-Peter Wickern’s input – thanks!

LSharp on Mono is still not perfect, but many of the sample programs now run! I’d welcome more feedback or help from Mono programmers.

There’s a new IDE called Xacc that’s great for developing L Sharp programs. It even bundles the L Sharp runtime – defintely one to watch. See http://blogs.wdevs.com/leppie/ for more information. Thanks Leppie – Keep up the good work!

LSharp 1.2.1 changes include:

Fix to TopLoop so that it prints results correctly under Mono
Fixed problem with reader so that it sees lparen as a terminating character and thus correctly accepts expressions like (setq a(list 1 2))
Fixed Assoc so that it returns the pair not the cadr of the pair
Fixed Assoc so that the key can be a value type or reference
Primitive functions and special forms such as prl etc are no longer case sensitive
Added copy-list and append
Updates and corrections to documentation

Unfortunately Sourceforge seems to be suffering this afternoon (Sunday) and www.lsharp.org is intermittent or unavailable. Could this be due to the a high demand for LSharp 1.2.1 I wonder ;-)

Practical Common Lisp - Free Download

2006-01-22T00:00:00+00:00

Peter Seibel’s excellent book “Practical Common Lisp” is now available as a free download from the publisher, Apress.

L Sharp Version 1.2 Released

2005-12-29T00:00:00+00:00

Thank goodness for the Christmas Holidays – at last, a chance to get some work done!

This new version of L Sharp incorporates several bug fixes and minor modifications based on feedback received throughout 2005.

I suppose the most visible change is the move to Visual Studio 2005 and Microsoft .NET 2.0 (although the source code will still compile to .NET 1.1 if you wish).

Here is a partial list of changes:

Moved to Visual Studio 2005 and .NET 2.0
Added some built in macros – defun, defmacro and listp
Changes to the evaluation model to deal with runtime types – symbols now evaluate to themselves, not their types You must now use (typeof string) to get string type object
Added assoc
read now allows you to specify an optional eof value
(is type expression) now checks supertypes as well as actual types
Added binary operators (and & , exclusive-or ^ , or | ).
Removed the prototype IDE / GUI from source release – no longer works with .NET2 and using emacs is better
Updates and corrections to documentation

Seasons Greetings to all Lispers.

Special Delivery

2005-12-24T00:00:00+00:00

Suzanne’s daughter Rebecca gave birth to a baby boy, Edward Jordan Barker at 23:00 on 22 December 2005.

He weighed in at 9 lb 8 oz.

They’re home for Christmas.

Popularity

2005-12-21T00:00:00+00:00

In the pub today I mentioned that Java was the most popular programming language and how it had overtaken Cobol. See Paul Graham’s essay Great Hackers .

The Tiobe index makes for interesting reading, but you need to be careful about drawing too many conclusions because of it’s data points.

I also checked the stats on web server technology at Netcraft.
Apache has about 70% of the market and Microsoft 21%.

IT Conversations

2005-12-19T00:00:00+00:00

IT Conversations provides audio programs and interviews with some of the leading people in the IT industry.

It’s great material for your iPod and helps to pass the time while decorating.

As you might expect, I particularly recommend the Paul Graham stuff.

WordPress

2005-12-18T00:00:00+00:00

I’ve decided to consolidate my home site and weblog using WordPress .

WordPress in open source and runs on LAMP. I found it easy to install and get started. I hope that the change will prompt me to make more frequent updates.

It might take me a while to restore all my content, but I hope to have this done by the new year.

ECOOP2005

2005-07-27T00:00:00+00:00

I’ve just returned from the 2nd European LISP and Scheme Workshop, part of the 19th European Conference on Object-Oriented Programming (ECOOP2005).

I presented a paper entitled Introducing L Sharp, a Modern Lisp Dialect for the .NET Platform in which I talked about L Sharp and showed some demonstrations. In general, I think it was well received. I certainly got some useful feedback which was my prime objective.

Richard Gabriel suggested that I consider the idea of different name, different thing; that I need to be able to say that L Sharp is a something language. He pointed out how, when Java started, it didn’t compete with C++ and how Python, Ruby etc are positioned as scripting languages so that they don’t compete with Java. L Sharp needs to find a something to differentiate it – maybe it’s a configuration language for Vista for example.

I already knew from ILC2005 that there were people for whom L Sharp will always be too much of a compromise; they’d loose too much if they moved away from Common Lisp. One thing that I hadn’t considered until today is the view that Lisp doesn’t need to become more mainstream and that popularity is not necessarily a good goal – as long as it survives and there is enough support that’s okay. Edi Weitz seemed to enjoy the niche. I respect that point of view, but I ’m interested in how I can bridge the gap between how I program professionally and how I program in my spare time. As I said at the conference, delivering in Lisp is hard and getting harder because of this requirement for managed code.

Anyway, my half hour slot turned into nearly one hour with debate and questions and all the feedback and advice was welcome.

Brent Benson talked about building web applications in LISP – On the Internet nobody knows you’re a Lambda. I found the talk encouraging, especially as I’m hoping to embark upon a Common Lisp based Web application soon. I intend to take a look at BKNR which is built in LISP.

Richard Gabriel’s key note talk covered some similar ground to his address at ILC2005, but it’s still interesting to me how he can stand back and look at software development from new perspectives. Most people seem to be concerned with making incremental improvements, but he seems to be suggesting that we’ve hit a local maxima and that we need to back track. I need to think about autopoietic software some more.

Richard was kind enough to sign my copy of his book ‘Patterns of Software: Tales from the Software Community’.

International Lisp Conference 2005

2005-06-26T00:00:00+00:00

I’ve just returned from the International Lisp Conference 2005 at Stanford University.

I’m pleased to report that the Lisp Community is alive and well and still full of colourful characters and interesting people.

I’m used to computing conferences being high budget affairs with lots of stands (TechEd, Devweek), but Lisp only has two real vendors, Franz and LispWorks and whilst both companies were present they were fairly low profile; this conference was about Lisp developers sharing ideas, not having marketing messages drilled into them – very refreshing!

Alex Peake showed how his company Comac is using generative programming to get big productivity gains when developing .NET applications. The .NET community seems to be content with employing armies of programmers to churn out tedious GUI and database access code. These patterns can be identified and abstracted to good effect. Alex was kind enough to mention LSharp. He is also a technical director of a software house and we had some interesting discussions about disruptive technology, recruitment, Language Workbench etc. I thank him for his encouragement of my L Sharp endeavours.

Richard P Gabriel‘s talk on Conscientious Software was excellent and very inspiring. We need to start thinking of software that can take care of itself and ’not just sit their like a toaster’. Think Autopoietic not Allopoietic. It was a reminder that we are still in the infancy of Software Development.

Will Clinger’s talk concluded with ’We’re the Common Larceny team, boy do we have a Scheme for you’. These guys have a Scheme implementation running on the CLR with reflective invocation and even a Tiny CLOS. In many ways they are working on the same problem as LSharp, just that they are coming at it from an Academic perspective. I think that L Sharp has a cleaner syntax that is more accessible by .NET people, but Common Larceny is further ahead in terms of implementation.

Harold Carr’s talk on LLava highlighted the language design challenges faced when developing new dialects of Lisp. His principle of ‘maximum leverage of Java – only add what is missing or cannot be done in Java’ makes for an interesting syntax which he is still grappling with.

Talks about Kawa and CLforJava underlined the level of interest there is in having Lisp dialects running on mainstream virtual machines.

Franz presented their forthcoming AllegroCache product. It’s a persistent object database that’s really well integrated into CLOS and has impressive performance and scalability. There will be a SQL interface, but they showed querying it with their embedded Prolog. I’d like to get my hands on this for some work at Active Web Solutions. I’m now the proud owner of a (got-lisp-p) T Shirt.

There are many cool Lisp applications out there and we saw demos of the following to name a few; Bio Lingua, Honda crash test analysis, particle accelerator management and NASA autonomous helicopter.

When I first worked for BT’s Research and Development in the early nineties, there were great expectations for AI, expert systems and natural language processing. It seems that some of the goals have been quietly achieved, with the Langutils demo showing a high degree of competence in natural language understanding and common sense reasoning.

I met Patrick Dussad, Chief Architect for the Common Language Runtime at Microsoft. He had downloaded LSharp and liked it. He gave me plenty of good feedback including the need to be able to define classes and methods; too much consing; the need for a compiler, CodeDom etc. We discussed the fact that you can’t currently garbage collect dynamically generated code; this is fixed in a forthcoming CLR and the workaround is to unload the AppDomain. He said that I could email with questions and that he would try to get answers for me.

I asked him about the CLR implementation and he confirmed that the Garbage Collector had been prototyped in Lisp before being put through a Lisp to C translator.

In his talk, Patrick called for a new, modern lisp implementation on the .NET platform and shared his ideas about how he would go about this task if he had the time. There was quite a lot of negativity and hostility from the audience which is a real shame. Somebody said that the Lisp Community was waiting for the mainstream to find them and that there was no need to change; I think this is a head-in-the-sand view. Patrick said that if the Lisp Community didn’t fill the dynamic language void on the .NET platform, somebody else would. I think that if we could provide a kind of ‘cut down’ Lisp on .NET, that would be very useful in itself and would probably be a signpost to help more people discover Common Lisp. I think any community has to adapt or die.

Here are some other observations about California life:

The Internet is ubiquitous here (more so than the UK) – hotel rooms with free broadband, Wireless just about everywhere campus-wide.

I’m a part time Apple Mac user. I’d read Paul Graham’s piece on Return of the Mac saying that all the best hackers are now on Mac, but I was amazed at just how widespread Apple was in the whole of the Bay Area. Wherever I went it seemed that Apple was there – either on the advertising boards or there’d be somebody using a Mac at a coffee shop. Macs and IPods are fashion accessories as well as functional devices. In the Stamford Computer Shop the whole focus was Apple; yes you could buy a PC, but PC’s are just not appealing any more. I also came across three Apple stores – one in Palo Alto, one in Stanford Shopping Centre and one in San Francisco. All were doing a roaring trade. All were appealing to the man in the street rather than techies per se. I particularly like the idea of being able to book an appointment at the Genius Bar – a kind of clinic where you can get advice from and apple expert.

We concluded with a couple of days in ‘The City’ (San Francisco) to do the usual tourist stuff and that was a great end to a fantastic trip.

My Grandfather

2005-02-09T00:00:00+00:00

My Aunt Janet, a teacher, asked my grandfather to record some World War II Memories in 1994 so that she could use the material in her class history lesson.

I recently converted the audio cassette recording to MP3 in an effort to preserve it for future generations and make it more widely available.

The recording describes my maternal grandfather’s experience as a Geography teacher in 1939, the announcement of Hitler invading Poland, evacuation from West Ham and an air raid.

You can download it here [ 7Mb MP3 file].

Sadly there are no further installments. It would have been interesting to hear of his experiences taking Morse code in The Navy.

Arthur Edward Richard Rowe, c1942.