Amir Chaudhry

Unikernels and Modularity at GlueCon 2016

Amir Chaudhry — 2016-05-25T00:01:00+00:00

Above are the slides from my talk at GlueCon today. Following the overview, I spend a little more time describing the benefits of having an ecosystem of libraries we can use for unikernels. The demo I gave this time was a little more involved and showed how we can build unikernels using your normal development approaches but then re-target the same application to different backends (i.e. for ARM and the Internet of Things).

Below are some details of how you can step through the demo for yourselves and you can find more information about unikernels over at unikernel.org.

Prerequisites

The following instructions use Docker for Mac, which is currently in beta, but should work with any Docker installation. In order to deploy to a Cubieboard2, you need to have a board set up and running Xen. You can follow the relevant instructions for details.

The 2048 game I used here was originally put together as a tutorial for teaching OCaml and using js_of_ocaml. If you’re interested, you can find more instructions and follow the tutorial yourself on over on the original repo.

Building the 2048 game for the Unix backend

I used Docker for Mac, together with pre-prepared Docker images to speed things up. Bear in mind that things change across beta releases so if the following doesn’t work, please tweet me.

The first thing to show is how you can do development by targeting unix and using the appropriate system libraries. The Docker image already has a clone of the repo so you can use that. To follow my steps from the demo:

# On your terminal
$ docker run -it -p 80:8080 amirmc/gluecon2016-unix bash

# Inside the container
$ eval `opam config env`
$ cd gluecon2016
$ mirage configure --unix --net socket
$ make
$ sudo ./mir-www

You’ll see that the tool runs some checks and tries to install any necessary components for the target — in this case the unix-related libraries.

Once you’ve completed the above steps, you can point your browser at localhost to see the site. Congratulations, you should see the unix-based version of the 2048 game! Try not to get too distracted by playing. The next step is to use the same code base to build an ARM-based unikernel that will run on the Cubieboard2.

Building 2048 for the ARM backend

You will again use Docker but this time with the ARM images, based on resin/armv7hf-debian (NB: This is not a cross-compilation step). You can build the Xen image in the container using Docker but in order to deploy, you’ll need the Cubieboard2 already set up. This time, you can clone the demo repo on your machine and mount it into the container. The steps from this part of the demo were:

# Clone the repo and mount it in the ARM container
$ git clone https://github.com/amirmc/gluecon16.git
$ cd gluecon16
$ docker run -it -v `pwd`:/src amirmc/gluecon2016-arm bash

# Inside the container
$ eval `opam config env`
$ cd /src
$ mirage configure --xen --dhcp=true --network=0 --no-opam
$ make

The --no-opam flag is used because the container already has the necessary packages and you can skip that check. The --network=0 is to override a default in the config.ml file.

You should now have a file on your Mac called mir-www.xen. This is the binary you need to copy over to the Cubieboard2. You’ll notice an additional file, www.xl, which contains some configuration and will also need to be copied over (you will need to update the path in that file). Once these files are on the Cubieboard2, then it’s a simple case of sudo xl create -c www.xl. The IP address of the unikernel will be displayed and you can navigate to <IP address>:8080 from your browser. You’re now playing the 2048 game from a unikernel being served from the Cubieboard2! During this re-targeting, none of the application code had to be changed, the appropriate system libraries were pulled in at the right time.

Although the above may seem quite involved, it does demonstrate how we can use the existing tools to to develop and deploy applications that span both the cloud and IoT. Over time, the tooling for unikernels will improve and mature so that all of the above will become much easier!

Other resources

I mentioned Containers as a Service and you can find out more about Docker Products.
The Docker Desktops beta, which uses a number of the MirageOS libraries.
The MirageOS website, https://mirage.io.
- The mirage-skeleton repo, which has a number of examples

To get involved in the development work, please do join the unikernel forum, the MirageOS devel list, and try out some of the examples for yourselves!

Thanks to Mindy for reviewing an earlier draft.

Notes on my Craft Conf Talk

Amir Chaudhry — 2016-04-28T03:00:00+00:00

These are the slides from my talk at Craft Conf today. If you’re reading this at the conference now, please do follow the link at the end of my talk to rate it and give me feedback!

Some of the specific items I mention are below, with links to more detail. You can find more information and resources about unikernels over at unikernel.org.

Build the MirageOS website with Docker for Mac

I quickly showed people how to build and serve the MirageOS website from a container on my Mac. I used the Docker for Mac beta (beta 8) for this, together with a prepared Docker image. Bear in mind that things change across beta releases so if the following doesn’t work, please tweet me.

To follow my steps in the demo:

# On your terminal
$ docker run -it -p 80:80 amirmc/demo-craft bash

# Inside the container
$ cd mirage-www/src
$ mirage configure --unix --net socket
$ make
$ sudo ./mir-www

You should be able to see the site by pointing your browser to localhost! Congrats, you just built MirageOS! You can read the hello world guide on the website to understand more about the configure flags we used.

The above image (amirmc/demo-craft) is one I made just for the conference, so that I didn’t have to wait for additional downloads. To do it ‘properly’, it’s better to use the unikernel/mirage images, since those are maintained. The mirage tool within the container will then pull down the required dependencies based on the mirage configure step.

# Clone the repo on your machine
$ git clone https://github.com/mirage/mirage-www.git
$ cd mirage-www

# Pull and run the maintained image, mount
# the local directory, and set up the ports
$ docker run -it -v `pwd`:/src -p 80:80 unikernel/mirage bash

# Now that you're in the container
$ cd src
$ mirage configure --unix --net socket # installs packages
$ make
$ sudo ./mir-www

Then point your browser at localhost.

Using Docker for Mac makes all of the above much easier, especially if you don’t have (or don’t want) a local OCaml environment set up.

The Bitcoin Piñata

This is a unikernel, built using the TLS stack, which holds the private key to some bitcoin. If you manage to break in then you can simply transfer the bitcoin away. The model itself is a novel way to run a bounty programme, since a successful breach leads directly to the reward — though the main benefit is to stress-test the whole stack in the wild. In the talk, I describe how the unikernel is about 4% the size of its equivalent in a traditional OS and you can read the article for more detail about that (and much more).

Article on the TLS stack, “Not-quite-so-broken TLS”
Some background to the Bitcoin Pinata, “The Bitcoin Piñata!”
The Pinata itself “You have reached the BTC Piñata”
Looking over the attempts “Reviewing the Bitcoin Piñata”

Other resources

I mentioned Containers as a Service and you can find out more about Docker Products.
The Docker Desktops beta, which uses a number of the MirageOS libraries.
The MirageOS website, https://mirage.io.
- The mirage-skeleton repo, which has a number of examples
The Nymote site, http://nymote.org.
- *The Introductory post is a useful place to start.

To get involved in the development work, please do join the unikernel forum, the MirageOS devel list, and try out some of the examples for yourselves!

A logo for unikernels!

Amir Chaudhry — 2016-01-06T18:00:00+00:00

I’ve been working towards a logo for unikernels and it’s time to gather feedback from the unikernel community! There’s been a 99designs contest running for a while — where designers have submitted just over 400 entries — and a short-list for community feedback is now at https://99designs.co.uk/logo-design/vote-99gjyi (link open until at least 15th Jan).

At this stage, the aim is to gather wider feedback to inform the next step. The final design will be selected at a later date — after further iteration with designers. Once a logo is chosen, it will be used on unikernel.org and will help inform the design of the rest of the site (with input from a professional firm).

For more background you can read the design brief and you can even look through all the entries.

Why do unikernels need a logo?

The unikernel approach as a concept has been getting much more attention recently and this will only increase throughout 2016.

Since there are a number of different projects — each approaching the problem from a different perspective — it will become even more important to ease the process of bringing in new users and contributors across all the projects.

The website at unikernel.org is already helping with this but we will also need a simple visual identity to go alongside it. Building this kind of brand will greatly improve the standing of unikernels as a whole and will be just as important as the work going on in each of the implementations. Ultimately, this will help to increase the rate of adoption.

What about projects’ existing logos?

To avoid any confusion, I’ll say upfront that this logo is not intended to replace any project’s existing brand! It’s meant to help us represent unikernels as a whole and each project will make it own choices, just as they do currently. The logo will mainly be used on unikernel.org and projects can make use of it if they wish.

Projects have already benefited from shared code and the same will be true from this shared brand.

What now?

Give feedback on the short-list of designs via https://99designs.co.uk/logo-design/vote-99gjyi!
Discuss this post on the new forum at devel.unikernel.org!

CodeMesh 2015

Amir Chaudhry — 2015-11-03T11:15:00+00:00

These are the slides from my talk today at CodeMesh. This time around I was earlier in the schedule so I get to enjoy the rest of the conference! If you’re reading this at the conference now, please do follow the link in my talk to rate it and give me feedback!

The specific items I reference in the talk are below with links to more information.

Security and the Bitcoin Piñata

This is a bounty where we have locked away some bitcoin in a unikernel that is running our new TLS stack. This was a new model of running a bounty and has proven a great way to stress test the code in the wild.

Some background to the Bitcoin Pinata, “The Bitcoin Piñata!”
The Pinata itself “You have reached the BTC Piñata”
Looking over the results of the attempts “Reviewing the Bitcoin Piñata”

You can follow up with more of the background work on the TLS stack by looking at the paper, “Not-quite-so-broken TLS: lessons in re-engineering a security protocol specification and implementation” and find other users of the libraries via https://nqsb.io.

Automated deployment

I’ve previously written about how we do unikernel deployments for MirageOS. Although the scripts themselves have evolved and become more sophisticated, these are still a good introduction.

Initial post on building a unikernel “From Jekyll to Unikernel”
The deployment steps for one of our repos “Heroku for Unikernels: Part 1 - Automated deployment”
Some thoughts on what the future might look like “Heroku for Unikernels: Part 2 - Self Scaling Systems”

Summoning on demand

The work on summoning unikernels was presented at Usenix this year and you can read the paper, “Jitsu: Just-In-Time Summoning of Unikernels”. The example I showed in the talk can be found at http://www.jitsu.v0.no.

Other resources

The MirageOS website, https://mirage.io
- The mirage-skeleton repo, which has a number of examples
The Rump Kernels site, http://rumpkernel.org
The Nymote site, http://nymote.org
- *The Introductory post is a useful place to start.

To get involved in the development work, please do join the MirageOS devel list and try out some of the examples for yourselves!

Governance of OCaml.org

Amir Chaudhry — 2015-09-18T14:00:00+00:00

For several months, I’ve been working with the maintainers of OCaml.org projects to define and document the governance structure around the domain name. I wrote about this previously and I’m pleased to say that the work for this phase has concluded, with the document now live.

Recurring themes

There were some recurring themes that cropped up during my email discussions with people and I thought it would be useful to present a summary of them, along with my thoughts. Broadly, the discussions revolved around the philosophy of the document, the extent of its scope, and the depth of coverage. This discourse was very important for refining and improving the document.

Ideals and Reality

Some of the comments I received were essentially that the document did not represent how we should be organising ourselves. There was occasionally the sense (to me at least) that the only appropriate form of governance is a fully democratic and representational one.

That would entail things like official committees, ensuring that various communities/organisations were represented, and perhaps establishing some form of electoral processes. Overall, something relatively formal and quite carefully structured. Of course, instituting such an arrangement would necessarily require somewhat involved procedures, documentation, and systems — as well as the volunteer time to manage those processes.

These may be noble aims — and I expect one day we’ll be closer to such ideals — but one of the critical factors for the current approach was that we record how things are right now. In my experience, anything else is purely aspirational and therefore would have little bearing with how things currently function.

To put it another way, the current document must not describe the structure we desire to have, but the organisation we actually have — warts and all. Yes, right now we have a BDFL*, who personally owns the domain and therefore can do as he pleases with it. Irrespective of this, the community has been able to come together, coordinate themselves, and build very useful things around the domain name. This has happened independently of any formal community processes and, in my view, has largely been driven by people supporting each other’s works and generally trying to ‘do the right thing’.

Another aspect to point out is that is that such documents and procedures are not necessary for success. This is obvious when you consider how far the OCaml community has come in such a relatively short space of time. Given this, one might argue why we need any kind of written governance at all.

To answer that, I would say that once things grow beyond a certain scale, I believe it helps to gather the implicit behaviours and document them clearly. This allows us to be more systematic in our approach and also enables newcomers to understand how things work and become involved more quickly. In addition, having a clear record of how things operate in the present is an invaluable tool in helping to clarify what exactly we should work on changing for the future.

Extent of scope

It’s a little confusing to consider that ‘OCaml.org’ is simultaneously a collection of websites, infrastructural components, and projects. Disambiguating these from the wider OCaml community was important, and relatively straightforward, but there were a few questions about the relationship between the domain name and the projects that use it.

Although the governance covers the OCaml.org domain name, it necessarily has an impact on the projects which make use of it. This matters because anything under the OCaml.org domain will, understandably, be taken as authoritative by users at large. In a way, OCaml.org becomes the sum of the projects under it, hence it’s necessary to have some lightweight stipulations about what is expected of those projects.

Projects themselves are free to organise as they wish (BDFL/Democracy/etc) but there are certain guiding principles for OCaml.org that those projects are expected to be compatible with (e.g. openness, community-related, comms, etc). These stipulations are already met by the current projects, so codifying them is intended to clarify expectations for new projects.

Depth of coverage

Another of the recurring points was how the current document didn’t capture every eventuality. Although I could have attempted this, the end result would have been a lengthy document, full of legalese, that I expect very few people would ever read. The document would also have needed to cover eventualities that have not occurred (yet) and/or may be very unlikely to occur.

Of course, this is not a legal document. No-one can be compelled to comply with it and there are very few sanctions for anyone who chooses not to comply. However, for those who’ve agreed to it, acceptance signals a clear intent to take part in a social contract with the others involved in work around the domain name.

Overall, I opted for a lightweight approach that would cover how we typically deal with issues and result in a more readable document. Areas that are ‘unchartered’ for us should be dealt with as they have been so far — through discussion and action — and can subsequently be incorporated when we have a better understanding of the issues and solutions.

A solid starting position

The current version of the governance document is now live and it is very much intended to be a living document, representing where we are now. As the community continues to grow and evolve, we should revisit this to ensure it is accurate and is meeting our needs.

I look forward to seeing where the community takes it!

In case you’re interested, the set of links below covers the journey from beginning to end of this process.

Background — “Towards a governance framework for OCaml.org”
Discussion phase — “Adopting a Governance framework…”
Tracking issue — ocaml/ocaml.org#700
Ratification — “Governance document is now ratified…”
Governance doc — “Governance of the OCaml.org domain”

* Yeah, I made sure to add Xavier to the BDFL list before publishing this. :)

Thanks to Ashish, Philippe and Anil for comments on an earlier draft.

Unikernels at PolyConf!

Amir Chaudhry — 2015-07-04T13:00:00+00:00

Updated: 14 July (see below)

Above are my slides from a talk at PolyConf this year. I was originally going to talk about the MISO tool stack and personal clouds (i.e. how we’ll build towards Nymote) but after some informal conversations with other speakers and attendees, I thought it would be way more useful to focus the talk on unikernels themselves — specifically, the ‘M’ in MISO. As a result, I ended up completely rewriting all my slides! Since I pushed this post just before my talk, I hope that I’m able to stick to the 30min time slot (I’ll find out very soon).

In the slides I mention a number of things we’ve done with MirageOS so I thought it would be useful to list them here. If you’re reading this at the conference now, please do give me feedback at the end of my talk!

Thomas’ Hello world and REST service, “My First Unikernel”
Magnus on “A unikernel experiment: A VM for every URL”
Mindy on “I Am Unikernel (and So Can You!)”
The mirage-skeleton repo, which has a number of examples
My previous posts (referred to in the talk)

To get involved in the development work, please do join the MirageOS devel list and try out some of the examples for yourselves!

Update — 14 July

The video of the talk is now available and it’s embedded below. Overall, the talk seemed to go well and there was enough time for questions.

At the end of the talk, I asked people to give me feedback and shared a URL, where I had a very short form. I had 21 responses with a rating of 4.52/5.00. I’m quite pleased with this and the feedback was also useful. In a nutshell, the audience seemed to really appreciate the walkthrough (which encourages me to make some screencasts). There was one comment that I didn’t do enough justice to the security benefits. Specifically, I could have drawn more reference to the OCaml TLS work, which prevents bugs like heartbleed. Security is definitely one of the key benefits of MirageOS unikernels (see here), so I’ll do more to emphasise that next time.

Here’s the video and I should mention that the slides seem to be a few seconds ahead. You’ll notice that I’ve left the feedback link live, too. If you’d like to tell me what you think of the talk, please do so! There are some additional comments at the end of this post.

Finally, here are few things I should clarify:

Security is one of the critical benefits, which is why we need new systems for personal clouds (rather than legacy stacks).
We still get to use all the existing tools for storage (e.g. EBS), it doesn’t have to be Irmin.
The Introducing Irmin post is the one I was trying to point an audience member at.
When I mention the DNS server, I said it was 200MB when I actually meant 200KB. More info in the MirageOS ASPLOS paper.
I referred to the “HAT Project” and you should also check out the “Databox paper”.
A summary of other unikernel approaches is also available.

Towards Heroku for Unikernels: Part 2 - Self Scaling Systems

Amir Chaudhry — 2015-04-03T15:30:00+00:00

In the previous post I described the continuous end-to-end system that we’ve set up for some of the MirageOS projects — automatically going from a git push all the way to live deployment, with everything under version-control.

Everything I described previously already exists and you can set up the workflow for yourself, the same way many others have done with the Travis CI scripts for testing/build. However, there are a range of exciting possibilities to consider if we’re willing to extrapolate just a little from the tools we have right now. The rest of this post explores these ideas and considers how we might extend our system.

Previously, we had finished the backbone of the workflow and I discussed a few ideas about how we should flesh it out — namely more testing and some form of logging/reporting. There’s substantially more we could do when we consider how lean and nimble unikernels are, especially if we speculate about the systems we could create as our toolstack matures. A couple of things immediately come to mind.

The first is the ability to boot a unikernel only when it is required, which opens up the possibility of highly-elastic infrastructure. The second is the ease with which we can push, pull or otherwise distribute unikernels throughout a system, allowing new forms of deployment to both cloud and embedded systems. We’ll consider these in turn and see where they take us, comparing with the current ‘mirage-decks’ deployment I described in Part 1.

Demand-driven clouds

The way cloud services are currently provisioned means that you may have services operating and consuming resources (CPU, memory, etc), even when there is no demand for them. It would be significantly more efficient if we could just activate a service when required and then shut it down again when the demand has passed. In our case, this would mean that when a unikernel is ‘deployed to production’, it doesn’t actually have to be live — it merely needs to be ready to boot when demand arises. With tools like Jitsu (Just-In-Time Summoning of Unikernels), we can work towards this kind of architecture.

Summon when required

Jitsu allows us to have unikernels sitting in storage then ‘summon’ them into existence. This can occur in response to an incoming request and with no discernible latency for the requester. While unikernels are inactive, they consume only the actual physical storage required and thus do not take up any CPU cycles, nor RAM, etc. This means that more can be achieved with fewer resources and it would significantly improve things like utilization rates of hardware and power efficiency.

In the case of the decks.openmirage.org unikernel that I discussed last time, it would mean that the site would only come online if someone had requested it and would shut down again afterwards.

In fact, we’ve already been working on this kind of system and Jitsu will be presented at NSDI in Oakland, California this May. In the spirit of looking ahead, there’s more we could do.

Hyper-elastic scaling

At the moment, Jitsu lets you set up a system where unikernels will boot in response to incoming requests. This is already pretty cool but we could take this a step further. If we can boot unikernels on demand, then we could use that to build a system which can automate the scale-out of those services to match demand. We could even have that system work across multiple machines, not just one host. So how would all this look in practice for ‘mirage-decks’?

Auto-scaling and dispersing our slide decks

Our previous toolchain automatically boots the new unikernel as soon as it is pulled from the git repo. Using Jitsu, our deployment machine would pull the unikernel but leave it in the repo — it would only be activated when someone requests access to it. Most of the time, it may receive no traffic and therefore would remain ‘turned off’ (let’s ignore webcrawlers for now). When someone requests to see a slide deck, the unikernel would be booted and respond to the request. In time it can be turned off again, thus freeing resources. So far, so good.

Now let’s say that a certain slide deck becomes really popular (e.g. posted to HackerNews or Reddit). Suddenly, there are many incoming requests and we want to be able to serve them all. We can use the one unikernel, on one machine, until it is unable to handle the load efficiently. At this point, the system can create new copies of that unikernel and automatically balance across them. These unikernels don’t need to be on the same host and we should be able to spin them up on different machines.

To stretch this further, we can imagine coordinating the creation of those new unikernels nearer the source of that demand, for example starting off on a European cloud, then spinning up on the East coast US and finally over to the West coast of the US. All this could happen seamlessly and the process can continue until the demand passes or we reach a predefined limit — after all, given that we pay for the machines, we don’t really want to turn a Denial of Service into a Denial of Credit.

After the peak, the system can automatically scale back down to being largely dormant — ready to react when the next wave of interest occurs.

Can we actually do this?

If you think this is somewhat fanciful, that’s perfectly understandable — as I mentioned previously, this post is very much about extrapolating from where the tools are right now. However, unikernels actually make it very easy to run quick experiments which indicate that we could iterate towards what I’ve described.

A recent and somewhat extreme experiment ran a unikernel VM for each URL. Every URL on a small static site was served from its own, self-contained unikernel, complete with it’s own web server (even the ‘rss.png’ icon was served separately). You can read the post to see how this was done and it also led to an interesting discussion on the mailing list (e.g. if you’re only serving a single item, why use a web server at all?). Of course, this was just an experiment but it demonstrates what is possible now and how we can iterate, uncover new problems, and move forward. One such question is how to automatically handle networking during a scale-out, and this is an area were tools like Signpost can be of use.

Overall, the model I’ve described is quite different to the way we currently use the cloud, where the overhead of a classic OS is constantly consuming resources. Although it’s tempting to stick with the same frame of reference we have today we should recognise that the current model is inextricably intertwined with the traditional software stacks themselves. Unikernels allow completely new ways of creating, distributing and managing software and it takes some thought in order to fully exploit their benefits.

For example, having a demand-driven system means we can deliver more services from just the one set of physical hardware — because not all those services would be consuming resources at the same time. There would also be a dramatic impact on the economics, as billing cycles are currently measured in hours, whereas unikernels may only be active for seconds at a time. In addition to these benefits, there are interesting possibilities in how such scale-outs can be coordinated across different devices.

Hybrid deployments

As we move to a world with more connected devices, the software and services we create will have to operate across both the cloud and embedded systems. There have been many names for this kind of distributed system, ranging from ubiquitous computing to dust clouds and the ‘Internet of Things’ but they all share the same idea of running software at the edges of the network (rather than just cloud deployments).

When we consider the toolchain we already have, it’s not much of a stretch to imagine that we could also build and store a unikernel for ARM-based deployments. Those unikernels can be deployed onto embedded devices and currently we target the Cubieboard2.

We could make such a system smarter. Instead of having the edge devices constantly polling for updates, our deployment process could directly push the new unikernels out to them. Since these devices are likely to be behind NATs and firewalls, tools like Signpost could deal with the issue of secure connectivity. In this way, the centralized deployment process remains as a coordination point, whereas most of the workload is dealt with by the devices the unikernels are running on. If a central machine happens to be unavailable for any reason, the edge-devices would continue to function as normal. This kind of arrangement would be ideal for Internet-of-Things style deployments, where it could reduce the burden on centralised infrastructure while still enabling continuous deployment.

In this scenario, we could serve the traffic for ‘mirage-decks’ from a unikernel on a Cubieboard2, which could further minimise the cost of running such infrastructure. It could be configured such that if demand begins to peak, then an automated scale-out can occur from the Cubieboard2 directly out onto the public cloud and/or other Cubieboards. Thus, we can still make use of third-party resources but only when needed and of the kind we desire. Of course, running a highly distributed system leads to other needs.

Remember all the things

When running services at scale it becomes important to track the activity and understand what is taking place in the system. In practice, this means logging the activity of the unikernels, such as when and where they were created and how they perform. This becomes even more complex for a distributed system.

If we also consider the logging needs of a highly-elastic system, then another problem emerges. Although scaling up a system is straightforward to conceptualise, scaling it back down again presents new challenges. Consider all the additional logs and data that have been created during a scale-out — all of that history needs to be merged back together as the system contracts. To do that properly, we need tools designed to manage distributed data structures, with a consistent notion of merges.

Irmin addresses these kinds of needs and it enables a style of programming very similar to the Git workflow, where distributed nodes fork, fetch, merge and push data between each other. Building an end-to-end logging system with Irmin would enable data to be managed and merged across different nodes and keep track of activity, especially in the case of a scale down. The ability to capture such information also means the opportunity to provide analytics to the creators of those unikernels around performance and usage characteristics.

The use of Irmin wouldn’t be limited to logging as the unikernels themselves could use it for managing data in lieu of other file systems. I’ll refrain from extrapolating too far about this particular tool as it’s still under rapid development and we’ll write more as it matures.

On immutable infrastructure

You may have noticed that one of the benefits of the unikernel approach arises because the artefacts themselves are not altered once they’re created. This is in line with the recent resurgence of ideas around ‘immutable infrastructure’. Although there isn’t a precise definition of this, the approach is that machines are treated as replaceable and can be regularly re provisioned with a known state. Various tools help the existing systems to achieve this but in the case of unikernels, everything is already under version control, which makes managing a deployment much easier.

As our approach is already compatible with such ideas, we can take it a step further. Immutable infrastructure essentially means the artefact produced doesn’t matter. It’s disposable because we have the means to easily recreate it. In our current example, we still ship the unikernel around. In order to make this ‘fully immutable’, we’d have to know the state of all the packages and code used when building the unikernel. That would give us a complete manifest of which package versions were pulled in and from which sources. Complete information like this would allow us to recreate any given unikernel in a highly systematic way. If we can achieve this, then it’s the manifest which generates everything else that follows.

In this world-view, the unikernel itself becomes something akin to caching. You use it because you don’t want to rebuild it from source — even though unikernels are quicker to build than a whole OS/App stack. For more security critical applications, you may want to be assured of the code that is pulled in, so you examine the manifest file before rebuilding for yourself. This also allows you to pin to specific versions of libraries so that you can explicitly adjust the dependencies as you wish. So how do we encode the manifest? This is another area where Irmin can help as it can keep track of the state of package history and can recreate the environment that existed for any given build run. That build run can then be recreated elsewhere without having to manually specify package versions.

There’s a lot more to consider here as this kind of approach opens up new avenues to explore. For the time being, we can recognise that the unikernel approach lends itself to the achieving immutable infrastructure.

What happens next?

As I mentioned at the beginning of this post, most of what I’ve described is speculative. I’ve deliberately extrapolated from where the tools are now so as to provoke more thoughts and discussion about how this new model can be used in the wild. Some of the things we’re already working towards but there are many other uses that may surprise us — we won’t know until we get there and experimenting is half the fun.

We’ll keep marching on with more libraries, better tooling and improving quality. What happens with unikernels in the rest of 2015 is largely up to the wider ecosystem.

That means you.

Edit: discuss this post on devel.unikernel.org

Thanks to Thomas Gazagnaire and Richard Mortier for comments on an earlier draft.

Towards Heroku for Unikernels: Part 1 - Automated deployment

Amir Chaudhry — 2015-03-31T14:30:00+00:00

In my Jekyll to Unikernel post, I described an automated workflow that would take your static website, turn it into a MirageOS unikernel, and then store that unikernel in a git repo for later deployment. Although it was written from the perspective of a static website, the process was applicable to any MirageOS project. This post covers how things have progressed since then and the kind of automated, end-to-end deployments that we can achieve with unikernels.

If you’re already familiar with the above-linked post then it should be clear that this will involve writing a few more scripts and ensuring they’re in the right place. The rest of this post will go through a real world example of such an automated system, which we’ve set up for building and deploying the unikernel that serves our slide decks — mirage-decks. Once you’ve gone though this post, you should be able to recreate such a workflow for your own needs. In Part 2 of this series I’ll build on this post and consider what the possibilities could be if we extended the system using some of our other tools — thus arriving at something very much like our own Heroku for Unikernels.

Standardised build scripts

Almost all of our OCaml projects now use Travis CI for build and testing (and deployment). In fact, there are so many libraries now that we recently put together an OCaml Travis Skeleton, which means we don’t have to manually keep the scripts in sync across all our repos — and fewer copy/paste/edits means fewer mistakes.

If you’re familiar with the build scripts from last time, then you can browse the new scripts and you’ll see that they’re broadly similar. In many cases you may well be able to depend on one or other of the scripts directly and for a handful of scenarios, you can fork and patch them to suit you (i.e. for MirageOS unikernels). We can do this because we’ve made it quick to set up an OCaml environment using an Ubuntu PPA. The rest of the work is done by the mirage tool itself so once that’s installed, the build process becomes fairly straightforward. The complexity around secure keys was also covered last time, which allowed us to commit the final unikernel to a deployment repo. That means the remaining step is to automate the deployment itself.

Automated deployment of unikernels

Committing the unikernel to a deployment repo is where the previous post ended and a number of people forged ahead and wrote about their experiences deploying onto AWS and Linode. Many of these deployments (understandably) involve a number of quite manual steps. It would be particularly useful to construct a set of scripts that can be fully automated, such that a git push to a repo will automatically run through the cycle of building, testing, storing and activating a new unikernel. We’ve done exactly this with some of our repos and this post will talk through those scripts.

The deployment options — Xen or *nix

MirageOS unikernels can currently be built for Xen and Unix backends. This is a straightforward step and typically the build matrix is already set up to test that both of them build as expected. For this post, I’ve only considered the Xen backend as that’s our chosen deployment method but it would be equally feasible to deploy the unix-based unikernels onto a *nix machine in much the same way. In this sense, you get to choose whether you want to deploy the unikernels onto a Hypervisor (for isolation and security) or whether running them as unix-processes better suits your needs. The unikernel approach means that both options are open to you, with little more than a command-line flag between them.

In terms of the deployment machines there are several options to consider. The most obvious is to set up a dedicated host, where you have full access to the machine and can install Xen. Another is to have a machine running on EC2 and create scripts to deal with unikernels. You could also build and deploy onto Xen on the Cubieboard2. If you’d rather test out the complete system first, you could set up an appropriate machine in Virtualbox to work with.

For our workflow, we use Xen unikernels which we deploy to a dedicated host. For the sake of brevity, I won’t go into the details of how to set up the machine but you can follow the instructions linked above.

The scripts for decks.openmirage.org

Decks is the source repo that holds many of our slides, which we’ve presented at conferences and events over the years (I admit that I have yet to add mine). The repo compiles to a unikernel that can then serve those slides, as you see at decks.openmirage.org. For maximum fun-factor, we usually run that unikernel from a Cubieboard2 when giving talks.

The toolchain for this unikernel includes build, store and deploy. We’ll recap the first two steps before going through the final one.

Build — In the root of the decks source repo, you’ll notice the .travis.yml file, which fetches the standard build script mentioned earlier. Building the unikernel proceeds according to the options in the build matrix.

language: c
install: wget https://raw.githubusercontent.com/ocaml/ocaml-travisci-skeleton/master/.travis-mirage.sh
script: bash -ex .travis-mirage.sh
env:
  matrix:
  - OCAML_VERSION=4.02 MIRAGE_BACKEND=unix MIRAGE_NET=socket
  - OCAML_VERSION=4.02 MIRAGE_BACKEND=unix MIRAGE_NET=direct
  - OCAML_VERSION=4.02 MIRAGE_BACKEND=xen
    MIRAGE_ADDR="46.43.42.134" MIRAGE_MASK="255.255.255.128" MIRAGE_GWS="46.43.42.129"
    DEPLOY=1
  global:
  - secure: ".... encrypted data ...."
  - secure: ".... encrypted data ...."
  - secure: ".... encrypted data ...."
  ...

In this case, two builds occur for Unix and one for Xen with different parameters being used for each. If you look at the actual travis file, you’ll notice there are 26 lines of encrypted data. This is how we pass the deployment key to Travis CI, so that it has push access to the separate mirage-decks-deployment repo. You can read the section in the previous post to see how we send Travis a private key.

Store — One of the combinations in the build matrix (configured for Xen), is intended for deployment. When that unikernel is completed, an additional part of the script is triggered that pushes it into the deployment repo.

Deployment scripts

After the ‘build’ and ‘store’ steps above, we have a deployment repository with a collection of Xen unikernels. For this stage, we have a new set of scripts that live in this repo alongside those unikernels. Specifically, you’ll notice a folder called scripts that contains four files.

.
├── Makefile
├── README.md
├── scripts
│   ├── crontab
│   ├── deploy.sh
│   ├── install-hooks.sh
│   └── post-merge.hook
...

A quick summary of the setup is that we clone the repo onto our deployment machine and install some hooks there. Then a simple cronjob will perform git pull at regular intervals. If a merge event occurs, then it means the repo has been updated and another script is triggered. That script removes the currently running unikernel and boots the latest version from the repo. It’s fairly straightforward and I’ll explain what each of the files does below.

Makefile - After cloning the repo, run make install. This will trigger install-hooks.sh to set things up appropriately. It’s worth remembering that from this point on, the git repo on the deployment machine will not be identical to the deployment repo on GitHub.

install-hooks.sh — The first two lines ensure that the commands will be run from the root of the git repo. The third line symlinks the post-merge.hook file into the appropriate place within the .git directory. This is the folder where customized git hooks need to be placed in order to work. The final line adds the file scripts/crontab to the deployment machine’s list of cron jobs.

ROOT=$(git rev-parse --show-toplevel)  # obtain path to root of repo
cd $ROOT

# symlink the post-merge.sh file into the .git/hooks folder
ln -sf $ROOT/scripts/post-merge.hook $ROOT/.git/hooks/post-merge
crontab scripts/crontab                # add to list of cron jobs

crontab — This file is a cronjob that sets up the deployment machine to perform a git pull on the deployment repo at regular intervals. Changing the file in the repo will ultimately cause it to be updated on the deployment machine (cf. deploy.sh). At the moment, it’s set to run every 11 minutes.

*/11 * * * * cd $HOME/mirage-decks-deployment && git pull

post-merge.hook — Since we’ve already run the Makefile, this file is symlinked from the appropriate place on the deployment machine’s copy of the repo. When a git pull results in new commits being downloaded and merged, then this script is triggered immediately afterwards. In this case, it just executes the deploy.sh script.

ROOT=$(git rev-parse --show-toplevel)  # obtain path to root of repo
exec $ROOT/scripts/deploy.sh           # execute the deploy script

deploy.sh — This is where the work actually happens and you’ll notice that there really isn’t much to do! I’ve commented in the code below to explain what’s going on.

VM=mir-decks
XM=xm

ROOT=$(git rev-parse --show-toplevel)
cd $ROOT

crontab scripts/crontab     # Update cron scripts

# Identify the latest build in the repo and then use
# the generic Xen config script to construct a
# specific file for this unikernel. Essentially,
# 'sed' just does a find/replace on two elements and
# the result is written to a new file.
#
KERNEL=$ROOT/xen/`cat xen/latest`
sed -e "s,@VM@,$VM,g; s,@KERNEL@,$KERNEL/$VM.xen,g" \
    < $XM.conf.in \
    >| $KERNEL/$XM.conf

# Move into the folder with the latest unikernel.
# Remove any uncompressed Xen images found there
# (since we may be starting a rebuilt unikernel).
# Unzip the compressed unikernel.
#
cd $KERNEL
rm -f $VM.xen
bunzip2 -k $VM.xen.bz2

# Instruct Xen to remove the currently running
# unikernel and then start up the new one we
# just unzipped.
#
sudo $XM destroy $VM || true
sudo $XM create $XM.conf

At this point, we now have a complete system! Of course, this arrangement isn’t perfect and there are number of things we could improve. For example, it depends on a cron job, which means it may take a while before a new unikernel is live. Replacing this with something triggered on a webhook could be an improvement, but it does mean exposing an end-point to the internet. The scripts will also redeploy the current unikernel, even if the only change is to the crontab schedule. Some extra work in the deploy script, using some git tools, might work around this.

Despite these minor issues, we do have a completely end-to-end workflow that takes us all the way from pushing some new changes to deploying a new unikernel! An additional feature is that everything is checked into version control. Right from the scripts to completed artefacts (including a method of transmitting secure keys/data, over public systems).

There is minimal work done outside the code you’ve already seen, though there is obviously some effort involved in setting up the deployment machine. However, as mentioned earlier, you could either use the unix-based unikernels or experiment with Virtualbox VM with Xen just to test out this entire toolchain.

Overall, we’ve only added around 20 lines of code to the initial 50 or so that we use for the Travis CI build. So for less than 100 lines of code, we have a complete end-to-end system that can take a MirageOS project from a git push, all the way through to a live deployment.

Fleshing out the backbone

In our current system, if the unikernel builds appropriately then we just assume it’s ok to deploy to production. Fire and forget! What could possibly go wrong! Of course, this is a somewhat naive approach and for any critical system it would be better to hook in some additional things.

Testing frameworks

One obvious improvement would be to introduce a more thorough testing regimen, which would include running unit tests as part of the build. Indeed, various libraries in the MirageOS project are already moving towards this model (e.g see the notes for links).

It’s even possible to go beyond unit tests and introduce more functional/systems/stress testing on the complete unikernel before permitting deployment. This would help to surface any wider issues as services interact and we could even simulate network conditions — achieving something like ‘staging on steroids’.

Logging and notifications

The scenario we have above also assumes that things work smoothly and nobody needs to know anything. It would be useful to hook in some form of logging and reporting, such that when a new unikernel is deployed a notification can be sent/stored somewhere. In the short term, there are likely existing tools and ways of doing this so it would be a matter of putting them together.

Looking ahead

Overall, with the above model, we can easily set up a system where we go from writing code, to testing it via CI, to deploying it to a staging server for functional tests, and finally pushing it out into live deployment. All of this can be done with a few additional scripts and minimal interaction from the developer. We can achieve this because we don’t have to concern ourselves with large blobs of code, multiple different systems and keeping environments in sync. Once we’ve built the unikernel, the rest almost becomes trivial.

This is close enough for me to declare it as a ‘Heroku for unikernels’ but obviously, there’s much more we can (and should) do with such a system. If we extrapolate just a little from where we are now, there are a range of exciting possibilities to consider in terms of automation, scalability and distributed systems. Especially if we incorporate other aspects of the toolstack we’re working towards.

Part 2 of this series is where I’ll consider these possibilities, which will be more speculative and less constrained. It will cover the kinds of systems we can create once the tools are more mature and will touch on ideas around hyper-elastic clouds, embedded systems and what this means for the concept of immutable infrastructure.

Since we already have the ‘backbone’ of the toolchain in place, it’s easier to see where it can be extended and how.

Edit: The second part of this series is now up - “Self Scaling Systems”

Edit2: discuss this post on devel.unikernel.org

Thanks to Anil Madhavapeddy and Thomas Leonard for comments on an earlier draft and Richard Mortier for his work on the deployment toolchain.

The Bitcoin Piñata!

Amir Chaudhry — 2015-02-10T15:25:00+00:00

Last summer we announced the beta release of a clean-slate implementation of TLS in pure OCaml, alongside a series of blog posts that described the libraries and the thinking behind them. It took two hackers six months — starting on the beach — to get the stack to that point and their demo server is still going strong. Since then, the team has continued working and recently presented at the 31st Chaos Communication Congress.

The latest example goes quite a bit further than a server that just displays the handshake. This time, the team have constructed a Xen unikernel that’s holding a private key to a bitcoin address and are asking people to try and break in. Hence, they’ve called it the Bitcoin Piñata!*

What the Bitcoin Piñata does

The Piñata unikernel will transmit its private bitcoin key if you can successfully set up a TLS connection but it’s rigged so that it will only create that connection if you can present the certificate it’s expecting to see — which has been signed appropriately. Of course, you’re not being given the secret key with which to do that signing and that means there should be no way for anyone to form a TLS connection with the Piñata. In order to get the private key to the bitcoin address, you’ll have to smash your way in.

Helpfully (perhaps), things are set up so that you can make the Piñata talk to itself, allowing you to eavesdrop on a successful connection and see the encrypted traffic. In addition, all the code and libraries are open-source so you can look through any of the codebase. There isn’t anything that anyone will have to reverse engineer, which should make this a little more enjoyable.

This contest is set to run until mid-March or whenever the coins are taken. If someone does manage to get in, please do let us know how!

The Rubber-hose approach

Of course there are many other ways to get at the private key and as many people like to comment, the human element is sometimes the weakest link — after all, a safe is only as secure as the person with the combination.

In this case, there is obviously a secret key or certificate somewhere that could be presented so it may be tempting to go hunting for that. Perhaps phishing attempts on the authors may yield a way forward, or maybe just straight-forward Rubber-hose cryptanalysis! Sure, these options might provide a result^† but this is meant to be fun. The authors haven’t specified any rules but please be nice and focus on the technical things around the Piñata^‡. Don’t be this guy.

What’s the point of this contest?

Even though the Bitcoin Piñata is clearly a contest, nobody is deluding themselves into thinking that if the coins are still there in March, that somehow the stack can be declared ‘undefeated’ — while pleasing, that result wouldn’t necessarily prove anything. Contests have their place but as Bruce Schneier already pointed out, they are not useful mechanisms to judge security.

However, it does give us the chance to engage in some shameless self-promotion and try to draw vast amounts of attention to the work. That, and the chance to stress-test the stack in the wild. Ultimately, we want to use this code in production but must take a lot of care to get there and want to be sure that it can bear up. This is just one more way of learning what happens when putting something ‘real’ out there.

If the Bitcoins do end up being taken, then there’s definitely something valuable that the team can learn from that. Regardless of the Piñata, if we have more people exploring the TLS codebase or trying it out for themselves, it will undoubtedly be A Good Thing.

Responsible sidenote

For clarity and to avoid any doubt, please be aware that the TLS codebase is missing external code audits and is not yet intended for use in any security critical applications. All development is done in the open, including the tracking of security-related issues, so please do consider auditing the code, testing it in your services and reporting issues.

* If you've never come across a piñata before, hopefully the gif in the post gives you an idea. If not, the wiki page will surely help, where I learned that the origin may be Chinese rather than Spanish!

^† Of course, I'm not suggesting that anyone would actually go this far. I'm simply acknowledging that there is a human factor and asking that we put it aside.

^‡ Edit to add: After seeing Andrea's tweet I should point out that any part of MirageOS, including the networking stack, OCaml runtime etc is a legitimate vector. It's why there's a manifest of the libraries that have been used to build the Piñata!

Edit: discuss this post on devel.unikernel.org

Unikernel demo at FOSDEM

Amir Chaudhry — 2015-02-06T17:50:00+00:00

Last weekend was spent at one of the world’s biggest open source conferences, FOSDEM. You can check out last year’s review to get an idea of the scale of the event. Since there’s no registration process, it’s difficult to estimate how many people attend but given how many rooms there are, and how full they are, it’s easily several thousand. I was impressed last year at how smoothly things went and the same was true this year.

The main reason to attend this time was to run a demo of MirageOS from an ARM board — one of the main advances since the previous conference. I looked over all the things we’d achieved since last year and put together a demo that showcases some of the capabilities as well as being fun.

2048 from a Unikernel on an ARM board

The demo was to serve the 2048 game from a Unikernel running on a Cubieboard2 with its own access point. When people join the wifi network, they get served a static page and can begin playing the game immediately.

The components I needed for the demo were:

Code for the 2048 game — I was able to lift code from a tutorial last year, which Daniel, Leo, Jeremy and Thomas all contributed to. It was first run at CUFP 2014 then adapted and presented at Functional Conf in India (see the IOCaml notebook). Attendees wrote the code in OCaml, which was then compiled into pure JavaScript (via js_of_ocaml). The result can be run completely in the browser and only involves serving two files.
Code for making a static website — Since the game is completely self-contained (one html file and one js file). I only need to convert a static website into a unikernel. That’s trivial and many people have done it before.
A Cubieboard with a wifi access point — There are pre-built images on the MirageOS website, which make part of this easy. However, getting the wifi access point up involves a few more steps.

The first two pieces should be straightforward and indeed, I had a working unikernel serving the 2048 game within minutes (unix version on my laptop). The additional factors around the ARM deployment is where things were a little more involved. Although this was technically straightforward to set up, it still took a while to get all the pieces together. A more detailed description of the steps is in my fosdemo repository and in essence, it revolves around configuring the wifi access point and setting up a bridge (thanks to Mindy, Magnus and David for getting this working).

Once this was all up and running, it was a simple matter to configure the board to boot the unikernel on startup, so that no manual intervention would be required to set things up at the booth.

Running the demo

I gave the demo at the Xen booth and it went very well. There was a small crowd throughout my time at the booth I’m convinced that the draw of a board with glowing LEDs should not be underestimated. Many people we’re happy to connect to the access point and download the game to their browser but there were two main things I learnt.

Firstly, demos involving games will work if people actually know the game. This is obvious, but I’d assumed that most people had already played 2048 — especially the crowd I’d expect to meet at FOSDEM. It turned out that around a third of people had no idea what to do when the game loaded onto their browser. They stared blankly at it and then blankly at me. Of course, it was trivial to get them started and they were soon absorbed by it — but it still felt like some of the ‘cool-factor’ had been lost.

The second thing was that I tried to explain too much to people in much too short a time. This particular demo involved Xen unikernels, js_of_ocaml and a Cubieboard2 with a wifi access point. There’s a surprisingly large amount of technology there, which is difficult explain to a complete newcomer within one or two minutes. When it was obvious someone hadn’t heard of unikernels, I focused on the approach of library operating systems and the benefits that Mirage brings. If a visitor was already familiar with the concept of unikernels, I could describe the rest of the demo in more detail.

Everything else did go well and next time I’d like to have a demo like this running with Jitsu. That way, I could configure it so that a unikernel would spin up, serve the static page and then spin down again. If we can figure out the timing, then providing stats in the page about the lifetime of that unikernel would also be great, but that’s for another time.

Sidenote: The beginnings of a ‘personal cloud’

One of the things we’re keen to work towards is the idea of personal clouds. It’s not a stretch to imagine that a Cubieboard2, running the appropriate software, could act as one particular node in a network of your own devices. In this instance it’s just hosting a fun and simple game but more complex applications are also possible.

Huge range of sessions and talks

Of course, there was lots more going on than just my demo and I had a great time attending the talks. Some in particular that stood out to me were those in the open source design room, which was a new addition this year. It was great to learn that there are design people out there who would like to contribute to open source (get in touch, if that’s you!). I also had a chance to meet (and thank!) Mike McQuaid in his Homebrew talk. FOSDEM is one of those great events where you can meet in person all those folks you’ve only interacted with online.

Overall, it was a great trip and I thoroughly recommend it if you’ve never been before!

Brewing MISO to serve Nymote

Amir Chaudhry — 2015-01-20T14:00:00+00:00

The mission of Nymote is to enable the creation of resilient decentralised systems that incorporate privacy from the ground up, so that users retain control of their networks and data. To achieve this, we reconsider all the old assumptions about how software is created in light of the problems of the modern, networked environment. Problems that will become even more pronounced as more devices and sensors find their way into our lives.

We want to make it simple for anyone to be able to run a piece of the cloud for their own purposes and the first three applications Nymote targets are Mail, Contacts and Calendars, but to get there, we first have to create solid foundations.

Defining the bedrock

In order to create applications that work for the user, we first have to create a robust and reliable software stack that takes care of fundamental problems for us. In other words, to be able to assemble the applications we desire, we must first construct the correct building blocks.

We’ve taken a clean-slate approach so that we can build long-lasting solutions with all the benefits of hindsight but none of the baggage. As mentioned in earlier posts, there are three main components of the stack, which are: Mirage (OS for the Cloud/IoT), Irmin (distributed datastore) and Signpost (identity and connectivity) - all built using the OCaml programming language.

Using the MISO stack to build Nymote

As you’ve already noticed, there’s a useful acronym for the above tools — MISO. Each of the projects mentioned is a serious undertaking in its own right and each is likely to be impactful as a stand-alone concept. However, when used together we have the opportunity to create applications and services with high levels of security, scalability and stability, which are not easy to achieve using other means.

In other words, MISO is the toolstack that we’re using to build Nymote — Nymote is the decentralised system that works for its users.

Each of the projects is at a different phase but they have all have made great strides over the last year.

Mirage

Mirage — a library operating system that constructs unikernels — is the most mature part of the stack. I previously wrote about the Mirage 1.0 release and only six months later we had an impressive 2.0 release, with continuing advances throughout the year. We achieved major milestones such as the ability to deploy unikernels to ARM-based devices, as well as a clean-slate implementation of the transport layer security (TLS) protocol.

In addition to the development efforts, there have also been many presentations to audiences, ranging from small groups of startups all the way to prestigious keynotes with 1000+ attendees. Ever since we’ve had ARM support, the talks themselves have been delivered from unikernels running on Cubieboards and you can see the growing collection of slides at decks.openmirage.org.

All of these activities have led to a tremendous increase in public awareness of unikernels and the value they can bring to developing robust, modern software as well as the promise of immutable infrastructure. As more people look to get involved and contribute to the codebase, we’ve also begun curating a set of Pioneer Projects, which are suitable for a range of skill-levels.

You can find much more information on all the activities of 2014 in the comprehensive Mirage review post. As it’s the most mature component of the MISO stack, anyone interested in the development of code towards Nymote should join the Mirage mailing list.

Source code - Mirage org on GitHub

Irmin

Irmin — a library to persist and synchronize distributed data structures — made significant progress last year. It’s based on the principles of Git, the distributed version control system, and allows developers to choose the appropriate combination of consistency, availability and partition tolerance for their needs.

Early last year Irmin was released as an alpha with the ability to speak ‘fluent Git’ and by the summer, it was supporting user-defined merge operations and fast in-memory views. A couple of summer projects improved the merge strategies and synchronisation strategies, while an external project — Xenstore — used Irmin to add fault-tolerance.

More recent work has involved a big clean-up in the user-facing API (with nice developer documentation) and a cleaner high-level REST API. Upcoming work includes proper documentation of the REST API, which means Irmin can more easily be used in non-OCaml projects, and full integration with Mirage projects.

Irmin is already being used to create a version controlled IMAP server and a version controlled distributed log system. It’s no surprise that the first major release is coming very soon!

Source code - Irmin on GitHub

Signpost

Signpost will be a collection of libraries that aims to provide identity and connectivity between devices. Forming efficient connections between end-points is becoming ever more important as the number of devices we own increases. These devices need to be able to recognise and reach each-other, regardless of their location on the network or the obstacles in between.

This is very much a nascent project and it involves a lot of work on underlying libraries to ensure that security aspects are properly considered. As such, we must take great care in how we implement things and be clear about any trade-offs we make. Our thoughts are beginning to converge on a design we think will work and that we would entrust with our own data, but we’re treating this as a case of ‘Here Be Dragons’. This is a critical piece of the stack and we’ll share what we learn as we chart our way towards it.

Even though we’re at the design stage of Signpost, we did substantial work last year to create the libraries we might use for implementation. A particularly exciting one is Jitsu — which stands for Just In Time Summoning of Unikernels. This is a DNS server that spawns unikernels in response to DNS requests and boots them in real-time with no perceptible lag to the end user. In other words, it makes much more efficient use of resources and significantly reduces latency of services for end-users — services are only run when they need to be, in the places they need to be.

There’s also been lots of efforts on other libraries that will help us iterate towards a complete solution. Initially, we will use pre-existing implementations but in time we can take what we’ve learned and create more robust alternatives. Some of the libraries are listed below (but note the friendly disclaimers!).

Source code

OCaml

OCaml is a mature, powerful and highly pragmatic language. It’s proven ideal for creating robust systems applications and many others also recognise this. We’re using it to create all the tools you’ve read about so far and we’re also helping to improve the ecosystem around it.

One of the major things we’ve been involved with is the coordination of the OCaml Platform, which combines the OCaml compiler with a coherent set of tools and workflows to be more productive in the language and speed up development time. We presented the first major release of these efforts at OCaml 2014 and you can read the abstract or watch the video.

There’s more to come, as we continue to improve the tooling and also support the community in other ways.

Early steps towards applications

Building blocks are important but we also need to push towards working applications. There are different approaches we’ve taken to this, which include building prototypes, wireframing use-cases and implementing features with other toolstacks. Some of this work is also part of a larger EU funded project* and below are brief summaries of the things we’ve done so far. We’ll expand on them as we do more over time.

Mail - As mentioned above, a prototype IMAP server exists (IMAPlet) which uses Irmin to store data. This is already able to connect to a client to serve mail. The important feature is that it’s an IMAP server which is version controlled in the backend and can expose a REST API from the mailstore quite easily.

Contacts - We first made wireframe mockups of the features we might like in a contacts app (to follow in later post) and then built a draft implementation. To get here, code was first written in OCaml and then put through the js_of_ocaml compiler. This is valuable as it takes us closer to a point where we can build networks using our address books and have the system take care of sharing details in a privacy-conscious manner and with minimal maintenance. The summary post has more detail.

Calendar - This use-case was approached in a completely different way as part of a hackathon last year. A rough but functional prototype was built over one weekend, with a team formed at the event. It was centralised but it tested the idea that a service which integrates intimately with your life (to the point of being very invasive) can provide disproportionate benefits. The experience report describes the weekend and our app — Clarity — won first place. This was great validation that the features are desirable so we need to work towards a decentralised, privacy-conscious version.

Time to get involved!

The coming year represents the best time to be working on the MISO stack and using it to make Nymote a reality. All source code is publicly available and the projects are varied enough that there is something for everyone. Browse through issues, consider the projects or simply write online and share with us the things you’d like to see. This promises to be an exciting year!

Sign up to the Nymote mailing list to keep up to date!

* The research leading to these results has received funding from the European Union's Seventh Framework Programme FP7/2007-2013 under the UCN project, grant agreement no 611001.

Unikernels for everyone!

Amir Chaudhry — 2015-01-19T13:00:00+00:00

Many people have now set up unikernels for blogs, documenting their experiences for others to follow. Even more important is that people are going beyond static sites to build unikernels that provide more complicated services and solve real-world problems.

To help newcomers get started, there are now even more posts that that use different tools and target different deployment methods. Below are summaries of some of the posts I found interesting and that will make it easier for you try out different ways of creating and deploying your unikernels.

Unikernel blogs with MirageOS

Mindy picked up where the first set of instructions finished and described her work to get an Octopress blog running on Amazon EC2. As one of the first people outside the core team to work on this, she had a lot of interesting experiences — which included getting into the Mirage networking stack to debug an issue and submit a bugfix! More recently, she also wrote a couple of excellent posts on why she uses a unikernel for her blog. These posts cover the security concerns (and responsibility) of running networked services on today’s Internet and the importance of owning your content — both ideas are at the heart of the work behind Nymote and are well worth reading.

Mindy’s posts
- Overview - “I Am Unikernel (and So Can You!)”
- First in her Mirage series - “It’s a Mirage! (or, How to Shave a Yak.)”
- “Attack Surface: Why I Unikernel, Part 1”
- “My Content Is Mine: Why I Unikernel, Part 2”

Ian took a different path to AWS deployment by using Vagrant and Test Kitchen to to get his static site together and build his unikernel, and then Packer to create the images for deployment to EC2. All succinctly explained with code available on GitHub for others to try out!

Ian on “Mirage with Jekyll on Amazon EC2”

Toby wanted to put together a blog that was a little more complicated than a traditional static site, with specific features like subdomains based on tags and the ability to set future dates for posts. He also pulled in some other libraries so he can use Mustache for sever-side rendering, where his blog posts and metadata are stored as JSON and rendered on request.

Toby on “Building a Blog with MirageOS”

Chris saw others working to get unikernel blogs on EC2 and decide he’d try getting his up and running on Linode instead. He is the first person to deploy his unikernel to Linode and he provided a great walkthough with helpful screenshots, as well as brief notes about the handful of differences compared with EC2. Chris also wrote about the issue he had with clean urls (i.e serving /about/index.html when a user visits /about/) — he describes the things he tried out until he was finally able to fix it.

Chris’ posts
- Setting up a unikernel - “About Not Found”
- Deploying to Linode - “Hello Linode”

Phil focused on getting unikernels running on a cubieboards, which are ARM based development boards — similar to the Raspberry Pi. He starts by taking Mirage’s pre-built Cubieboard images — which makes it easy to get Xen and an OCaml environment set up on the board — and getting this installed on the Cubieboard. He also noted the issues he came across along with the simple tweaks he made to fix them and finally serves a Mirage hello world page.

Phil on “Some Notes on Building and Running Mirage Unikernels on Cubieboard2”

More than just static sites

Static sites have become the new ‘hello world’ app. They’re simple to manage, low-risk and provide lots of opportunities to experience something new. These aspects make them ideal for discovering the benefits (and trade offs) of the unikernel approach and I look forward to seeing what variations people come up with — For instance, there aren’t any public instructions for deploying to Rackspace so it would be great to read about someone’s experiences there. However, there are also many other applications that also fit the above criteria of simplicity, low risk and plentiful learning opportunities.

Thomas Leonard decided to create a unikernel for a simple REST service for queuing package uploads for 0install. His post takes you from the very beginning, with a simple hello world program running on Xen, all the way through to creating his REST service. Along the way there a lots of code snippets and explanations of the libraries being used and what they’re doing. This is a great use-case for unikernels and there are a lot of interesting things to take from this post, for example the ease with which Thomas was able to find and fix bugs using regular tools. There’s also lots of information on performance testing and optimising of the unikernel, which he covers in a follow-up post, and he even built tools to visualise the traces.

Thomas’ posts
- Hello world and REST service - “My First Unikernel”
- Profiling and optimisations - “Optimising the Unikernel”
- Tool to visualise traces - “Visualising an Asynchronous Monad”

Of course, there’s much more activity out there than described in this post as people continually propose ideas on the Mirage mailing list — both for things they would like to try out and issues they came up against. In my last post, I pointed out that the workflow is applicable to any type of unikernel and as Thomas showed, with bit of effort it’s already possible to create useful, real-world services using the many libraries that already exist. There’s also a lot of scaffolding in the mirage-skeleton repo that you can build on which makes it even easier to get involved. If you want to dive deeper into the libraries and perhaps learn OCaml, there are lots of resources online and projects to get involved with too.

Now is a great time to try building a unikernel for yourself and as you can see from the posts above, shared experiences help other people progress further and branch out into new areas. When you’ve had a chance to try something out please do share your experiences online!

Edit: discuss this post on devel.unikernel.org

This post also appears on the Nymote blog.

Towards a governance framework for OCaml.org

Amir Chaudhry — 2015-01-08T18:15:00+00:00

The projects around the OCaml.org domain name are becoming more established and it’s time to think about how they’re organised. 2014 saw a lot of activity, which built on the successes from 2013. Some of the main things that stand out to me are:

More volunteers contributing to the public website with translations, bug fixes and content updates, as well as many new visitors — for example, the new page on teaching OCaml received over 5k visits alone. The increasing contributions are a result of the earlier work on re-engineering the site and there are many ways to get involved so please do contribute!

The relentless improvements and growth of OPAM, both in terms of the repository — with over 1000 additional packages and several new repo maintainers — and also improved workflows (e.g the new pin functionality). The OPAM site and package list also moved to the ocaml.org domain, becoming the substrate for the OCaml Platform efforts. This began with the work towards OPAM 1.2 and there is much more to come (including closer integration in terms of styling). Join the Platform list to keep up to date.

Much more activity on the mailing lists in general and user groups requesting to have lists made (e.g the teaching list). If anyone has a need for a new list, just ask on the infrastructure list!

There is other work besides those I’ve mentioned and I think by any measure, all the projects have been quite successful. As the community continues to develop, it’s important to clarify how things currently work to improve the level of transparency and make it easier for newcomers to get involved.

Factors for a governance framework

For the last couple of months, I’ve been looking over how larger projects manage themselves and the governance documents that are available. My aim has been to put such a document together for the OCaml.org domain without introducing burdensome processes. There are number of things that stood out to me during this process, which have guided the approach I’m taking.

My considerations for an OCaml.org governance document:

A governance document is not necessary for success but it’s valuable to demonstrate a commitment to a stable decision-making process. There are many projects that progress perfectly well without any documented processes and indeed the work around OCaml.org so far is a good example of this (as well as OCaml itself). However, for projects to achieve a scale greater than the initial teams, it’s a significant benefit to encode in writing how things work (NB: please note that I didn’t define the type of decision-making process - merely that it’s a stable one).
It must clarify its scope so that there is no confusion about what the document covers. In the case of OCaml.org, it needs to be clear that the governance covers the domain itself, rather than referring to the website.
It should document the reality, rather than represent an aspirational goal or what people believe a governance structure should look like. It’s very tempting to think of an idealised structure without recognising that behaviours and norms have already been established. Sometimes this will be vague and poorly defined but that might simply indicate areas that the community hasn’t encountered yet (e.g it’s uncommon for any new project to seriously think about dispute resolution processes until they have to). In this sense, the initial version of a governance document should simply be a written description of how things currently stand, rather than a means to adjust that behaviour.
It should be simple and self-contained, so that anyone can understand the intent quickly without recourse to other documents. It may be tempting to consider every edge-case or try to resolve every likely ambiguity but this just leads to large, legal documents. This approach may well be necessary once projects have reached a certain scale but to implement it sooner would be a case of premature optimisation — not to mention that very few people would read and remember such a document.
It’s a living document. If the community decides that it would prefer a new arrangement, then the document conveniently provides a stable starting point from which to iterate. Indeed, it should adapt along with the project that it governs.

With the above points in mind, I’ve been putting together a draft governance framework to cover how the OCaml.org domain name is managed. It’s been a quiet work-in-progress for some time and I’ll be getting in touch with maintainers of specific projects soon. Once I’ve had a round of reviews, I’ll be sharing it more widely and posting it here!

Describing the MISO stack at Entrepreneur First

Amir Chaudhry — 2014-07-25T11:30:00+00:00

I’m speaking to the Entrepreneur First cohort this morning about the future of resilient, distributed systems and what I’m working on to get us there. Firstly, I’m describing the kinds of solutions we have today, the great things they offer developers as well as the issues they create. This leads into the new toolstack we’re creating, called the MISO stack, and the benefits and trade-offs.

I’m spending more time talking about Mirage OS – the ‘M’ in the MISO stack – because the workflow we’ve developed here underpins how we build, deploy and maintain such applications at scale. As an example of how things can work, I point at my earlier post on how to go from jekyll to unikernel. This uses TravisCI to do all the hard work and all the relevant artefacts, including the final VM, can be version-controlled through Git. I actually deployed this post while the audience was watching, so that I could point at the build logs.

One of the use cases for our toolstack is to make it possible for individuals to create and maintain their own piece of the cloud, a project called Nymote, which will also make it possible to run the Internet of my Things – which itself is related to another things I’m working on, such the Hub of All Things - HAT and the User Centric Networking projects.

This is an exciting summer for all the tools we’re putting together, since we’ve recently announced Mirage OS v2.0, which now works on ARM, are going full steam ahead with Irmin and working hard on improvements to the OCaml ecosystem. It’s a great time to explore these projects, learn a new language and build awesome stuff.

Some of the research leading to these results has received funding from the European Union's Seventh Framework Programme FP7/2007-2013 under the UCN project, grant agreement no 611001.

Winning Seedhack 5.0

Amir Chaudhry — 2014-07-23T12:00:00+00:00

[This story was also a guest post on the Seedcamp blog.]

A couple of weeks ago, I took part in Seedhack 5.0, on the theme of life-logging. My team were overall winners with Clarity, our calendar assistant app. This post captures my experiences of what happened over the weekend, the process of how we built the app and the main things I learned. You’ll find out what Clarity is at the end – just like we did.

Meeting people and pitches on Friday

The weekend began with some information on the APIs available to us, which was followed by pizza and mingling with everyone. I spoke to a few people about what they were working on and the technologies they were used to. It was good to find a mixture of experience and I was specifically looking for folks with an interest in functional programming – that’s how I first met Vlad over Twitter.

After pizza, those people with ideas, even if not fully formed, were invited to share them with the room. I came in to Seedhack with specific thoughts on the kind of things I wanted to work on so I spoke about one of those.

Pitching Personal Clouds

I described the problem of silos, poor interoperability and how all the life-logging data should really be owned by the user. That would allow third parties to request access and provide way more value to users, while maintaining privacy and security. Building a centralised service makes a lot of sense in the first instance but what’s more disruptive than eschewing the current model of yet-another-silo and putting the user in control? If that sounds familiar, it’s because I’m trying to solve these problems already.

I’m working on a open source toolstack for building distributed systems that I call the MISO stack, which is analogous to the LAMP stack of old but is based on Mirage OS. With this stack, I’m putting together a system to help people create and run their own little piece of the cloud – Nymote. The introductory post and my post on ‘The Internet of my Things’ has more detail on why I’m working on this.

For systems like this to be viable, we must be willing to trust them with the core applications of Email, Contacts and Calendar. Without advanced and robust options for running these, it’s unlikely that anyone (including me) would want to switch away from the current providers. Of these three applications, I decided to talk about the contact management solution, since I happen to have wireframes and thought it might be simpler to implement something over the weekend.

There was quite a bit of interest in the overall concept but what really piqued my curiosity was that someone else presented some thoughts around Calendars and analytics. After a brief chat, we decided to join forces and tackle the problems of Calendar management. The team had a great mix of experience from product to design and several of them had worked together before.

The Clarity Team

Amir - Product - Worked in several startups, product & programme management experience, currently a Post Doc at Cambridge University Computer Science dept.

River - Product - Programme Manager at Dotforge Accelerator and lead organiser of StartupBus UK 2014.

Mani - Developer - Freelance web dev (CMS and APIs), winner of multiple hackathons, currently studying at Sheffield University.

Vlad - Developer - Started programming long ago and attended many competitions and hackathons. Currently studying Computer Science at the University of Southampton.

João - Developer - PhD in Theoretical Physics, Python enthusiast and moving into data science, currently doing data analysis at Potential.

Jeremy - Designer - Freelance UI/UX Designer, hackathon enthusiast, currently studying medicine at Sheffield University.

Thick fog and ambiguity on Friday evening

We had all decided to work together and we knew it would be on the problem of calendar management and analytics. We were fired up but it quickly became obvious that was all we knew.

The next four to five hours were spent discussing the rough shape of what we were going to build, what specific problem we thought we were solving and whether there were enough people with such a problem to care.

We had a look at each other’s calendars and talked about how we each use them and the things we like and dislike about them. For example, I have around nine calendars and I curate them carefully, adding contextual information and sometimes even correcting old events to reflect what happened. We even bounced around the idea of the contact management app several times as well as a few other ideas that came up during the discussions.

These conversations took a while and it seemed like we were going around in circles. Despite this, it didn’t feel particularly frustrating. I realised that the same sticking point was coming up repeatedly because we were forcing ourselves to imagine a prototypical customer and the problems they might have. This was never going to work since we wouldn’t have time to go and find such people and do basic customer development. Far better to constrain the problem to something we experience so that we can look to ourselves for initial customer feedback. Once we did this, things seemed to go a little faster and taking breaks for food helped us keep our energy up.

There were a few occasions where I looked around the room and saw other teams with their heads down, headphones in, and bashing away at keyboards – we hadn’t even figured out what we were doing yet. Despite this, it was a great exercise because it allowed all of us to get a feel for what aspects each of us cared about most and it helped us form some kind of shared language for the product.

The only outcome from this first evening was an outline but it was an important one. It distilled what we what we were going to work on and the components of it. We did this so we’d have a clear starting point the next morning and could get going quickly. Here’s a paraphrased version of what we sent ourselves.

Smart Calendar App.

We are collecting data from mobile and from desktop. 
- Mobile includes:
  - Call logs
  - Location (if we can)
  - Messages
  - What application is being used and when
- Desktop includes:
  - What application is active and timestamps of it
  - Location?
  - Git Logs ... ?
  - Taking logs of their existing calendars.

Working out what people are doing.
- Extrapolating info from active application (e.g browser page)

Present info back via calendar UI
- Need to turn all this information into webcal events

Useful info we want
- Time spent travelling (how much?)
- Time in meetings 
- Time on phone
- Who the meetings/calls were with
- Relevant docs/emails these are linked with
- Use labels

Then it was time for some late night snacks.

With one challenge out of the way, the next one was finding somewhere to sleep for a few hours (Campus was closing from 01:30). I had nothing planned but luckily for me, a couple of team members had booked a hotel room for the night. The minor complication was that we had to first find the hotel and then somehow get six people into a room meant for two – without the night manager kicking us out. That’s a whole other story, but it suffices to say that James Bond has nothing to worry about.

Rays of light and clearing haze on Saturday

After some card games and a few hours of sleep, we headed back to Campus and during this walk, we came up with ‘Clarity’ as the name of the application.

Once we arrived, development began. River volunteered his digital assets to the cause (i.e. his whole Google life). Mani worked on the Android app, Vlad on a Chrome extension with Joao pulling in the Google Apps data as well as combining it with data from the various platform apps. Jeremy worked on the front-end of the site, while River and I began wireframing the UI and user flow through the site.

Once the development was well underway, I realised how superfluous ‘the business guys’ can be. It would have been easy to simply sit there and let everyone get on with it but there were other things River and I did while developers were writing code.

Wireframing - We spent time thinking about what a user would actually see and engage with once they visited the Clarity site. We made a lot of sketches on paper and this was helpful because communication with the team was smoother with something to guide the discussion. It also helped to inform the design work and gave River and I something to show to potential users.

Talk to people - aka early customer development. We already knew that we were our own customers but it was useful to talk to other people for two reasons. Firstly, to get an idea of how they use their calendars and whether they have similar problems to us and secondly, to see what thoughts we prompt when we describe our solution (or show our wireframes). This led to useful information on how we should refine the product and and position ourselves against perceived competition.

Refine the product - Going through the wireframing and talking to people helped us come up with several new ideas for how to display the data back to users. Some of these seemed great at the time, but after showing some paper sketches to other people, we realised customers didn’t care about certain things, so we discarded them. Even though Jeremy had already done the work of putting together the UI for them (sorry, Jeremy!).

Examine the competition - After we described what we were working on. A few people mentioned potential competitors and asked how we were different. Initially, we didn’t know much about these companies but it was something we could explore while development was underway and consider our positioning.

Remind people to regroup - Every few hours, we would make sure everyone caught up with each other. We would check that things were going well, share what we’d learned from talking to people and discuss any technical problems and possible workarounds – including changing the scope of the product. The discussions we’d had on Friday meant that we spent less time debating when these questions came up during the weekend.

Work on the pitch - River and I began working on the pitch from just after Saturday lunchtime and kept building on it until Sunday afternoon. This, combined with showing our sketches to people, made it much easier to think about the story we wanted to tell the audience. In turn, that made it easier to think about the product development that had to be completed by Sunday. Especially in terms of a kick-ass demo.

Bright sunshine and achieving Clarity on Sunday afternoon

Development carried on through the night and we took a break to watch some sports via River’s laptop – at this point Clarity was actually logging this and other events. The next morning, we reiterated what we needed to get done for the demo and I was pretty ruthless about practising the pitch. River and I practised endlessly while everyone else made sure the the technical pieces were working smoothly. We had a lot of moving parts and making sure they were glued together seamlessly was important. At this point, we knew what Clarity was and how to tell its story.

Introducing Clarity

We all have calendars and we put a lot of time and effort into managing them but get very little back. A simple glance at your calendar for the past month will show you a sea of information with no idea where your time was actually spent. We believe your calendar should be working harder for you. Your calendar should give you clarity.

Over the course of the weekend, we built tools that can go through your calendar and understand the events you’re involved in and tie them back to the relevant emails, documents and people. With a suite of software that spans across your GDrive, Chrome and Android, we’re able to combine your calendars with rich, contextual information so you can really understand what your time is being spent on.

We built this system and plugged it into River’s digital life. If we take a look at River’s summary for the last month, we see that he’s spent around 32 hours in meetings in London last month, despite living in Sheffield. We can also see that he’s spent an hour on the phone with someone called Lee, but that all of them were short calls. The next person also totalled an hour on the phone but only across 3 calls. Already, River has learned something about the the people he interacts with most and how. We can also drill down further and see all this activity presented in a calendar view, except this now represents where his time did go, rather than where he thought it went. For example, he’s most active via text message between 4pm and 5pm during the week, and we can see that he spent a few hours watching sports last night.

Clarity can do much more than provide an accurate retrospective view of your time. Since it interacts with all the important components of your life, like your phone and laptop, it can even perform helpful actions for you. For example, say I have a meeting set up with River but I want to reschedule it. I simply send a text to him as I normally would, suggesting that we move it to another day. Clarity can pick up that message and is smart enough to understand its intent, find the relevant calendar event and reschedule it automatically. River doesn’t have to lift a finger and his diary is always up to date. It’s easy to imagine a future where we might never have to add or edit events ourselves.

Clarity is a smart calendar assistant that understands the context around you, provides you with insight into your life and helps you seamlessly organise your future. You can find out more at http://clarityapp.me

Judging and announcements

After the pitches, we met a lot of people who had the same problems as we did with calendar management. Several offered to be beta testers. After waiting for the judges, prizes were announced and Clarity was declared the overall winner of Seedhack 5.0!

Things I learned

Looking back, there were a lot of things we did which I think helped us get to the winning slot, so I thought I’d summarise them here.

Think first - We spent time up front to define what we were going to work on. I thought this step was crucial as it meant we all understood the shape of the problem but also the areas that each of us were interested in. That helped later in the weekend as we could refer back to things we discussed on Friday.

Move fast - Once we did figure out what we were doing, then it was a matter of building the software to gather and crunch the data. A lot of this was done in parallel as Jeremy worked on the front-end UI while Mani, Joao and Vlad took care of the data aggregation, analytics and platform products. Don’t be afraid to throw away ideas if you find that they don’t work for people and remember it’s a hackathon (i.e gruesome hacks are the norm).

Remember the demo - At some point you’re going to be forced to stand up and talk about what you’ve done. We started thinking about this from Saturday lunchtime and sketched out the slides and the elements of the app we wanted to show. This helped inform the UI and technology that we were building and the pitch never felt like it was rushed.

Practice, practice, practice - we were told we’d have 3 minutes to pitch/demo and maybe a few additional questions. I made sure River and I practised repeatedly to get our time down to 3mins and ensure we getting across everything we wanted to. The important thing with such a short amount of time, is that we were forced to cut things out as well as ensure we emphasised the main points. It was a ruthless exercise in saying ‘no’. During the actual pitches, we realised everyone was taking longer (without repercussions), so I added an extra 30 seconds to cover the potential market sizes and business models.

Leave artefacts - After the pitches, we knew that we had to take the site down. It was built at high speed and in a way that ended up exposing someone’s data to the internet at large (thanks, River!). It might have been somewhat easier to build something using faked data that we could happily share the URL to and leave online. On the other hand, our demo would not have been half as compelling if we weren’t running it real-time on live data.

Looking to the future

This is a product we all want to use and the team is interested in taking this forward. There are a lot of things to think about and many things we would build differently so we’re discussing the next steps. For example, there are likely ways to empower the end-users to control their data and give them more flexibility, even though the work at the hackathon was already quite impressive. Given how well Seedhack went, you might even see us at Seedcamp Week later in the year. If you think this is something you’d like to work on with us, do get in touch!

To wrap things up, here’s a victory selfie!

Writing Planet in pure OCaml

Amir Chaudhry — 2014-04-29T09:30:00+00:00

I’ve been learning OCaml for some time now but not really had a problem that I wanted to solve. As such, my progress has been rather slow and sporadic and I only make time for exercises when I’m travelling. In order to focus my learning, I have to identify and tackle something specific. That’s usually the best way to advance and I recently found something I can work on.

As I’ve been trying to write more blog posts, I want to be able to keep as much content on my own site as possible and syndicate my posts out to other sites I run. Put simply, I want to be able to take multiple feeds from different sources and merge them into one feed, which will be served from some other site. In addition, I also want to render that feed as HTML on a webpage. All of this has to remain within the OCaml toolchain so it can be used as part of Mirage (i.e. I can use it when building unikernels).

What I’m describing might sound familiar and there’s a well-known tool that does this called Planet. It’s a ‘river of news’ feed reader, which aggregates feeds and can display posts on webpages and you can find the original Planet and it’s successor Venus, both written in Python. However, Venus seems to be unmaintained as there are a number of unresolved issues and pull requests, which have been languishing for quite some time with no discussion. There does appear to be a more active Ruby implementation called Pluto, with recent commits and no reported issues.

Benefits of a Planet in pure OCaml

Although I could use the one of the above options, it would be much more useful to keep everything within the OCaml ecosystem. This way I can make the best use of the unikernel approach with Mirage (i.e lean, single-purpose appliances). Obviously, the existing options don’t lend themselves to this approach and there are known bugs as a lot has changed on the web since Planet Venus (e.g the adoption of HTML5). Having said that, I can learn a lot from the existing implementations and I’m glad I’m not embarking into completely uncharted territory.

In addition, the OCaml version doesn’t need to (and shouldn’t) be written as one monolithic library. Instead, pulling together a collection of smaller, reusable libraries that present clear interfaces to each other would make things much more maintainable. This would bring substantially greater benefits to everyone and OPAM can manage the dependencies.

Breaking down the problem

The first cut is somewhat straightforward as we have a piece that deals with the consumption and manipulation of feeds and another that takes the result and emits HTML. This is also how the original Planet is put together, with a library called feedparser and another for templating pages.

For the feed-parsing aspect, I can break it down further by considering Atom and RSS feeds separately and then even further by thinking about how to (1) consume such feeds and (2) output them. Then there is the HTML component, where it may be necessary to consider existing representations of HTML. These are not new ideas and since I’m claiming that individual pieces might be useful then it’s worth finding out which ones are already available.

Existing components

The easiest way to find existing libraries is via the OPAM package list. Some quick searches for RSS, XML, HTML and net bring up a lot of packages. The most relevant of these seem to be xmlm, ocamlrss, cow and maybe xmldiff. I noticed that nothing appears, when searching for Atom, but I do know that cow has an Atom module for creating feeds. In terms of turning feeds into pages and HTML, I’m aware of rss2html used on the OCaml website and parts of ocamlnet that may be relevant (e.g nethtml and netstring) as well as cow. There is likely to be other code I’m missing but this is useful as a first pass.

Overall, a number of components are already out there but it’s not obvious if they’re compatible (e.g html) and there are still gaps (e.g atom). Since I also want to minimise dependencies, I’ll try and use whatever works but may ultimately have to roll my own. Either way, I can learn from what already exists. Perhaps I’m being overconfident but if I can break things down sensibly and keep the scope constrained then this should be an achievable project.

The first (baby) steps - an Atom parser

As this is an exercise for me to learn OCaml by solving a problem, I need to break it down into bite-size pieces and take each one at a time. Practically speaking, this means limiting the scope to be as narrow as possible while still producing a useful result for me. That last part is important as I have specific needs and it’s likely that the first thing I make won’t be particularly interesting for many others.

For my specific use-case, I’m only interested in dealing with Atom feeds as that’s what I use on my site and others I’m involved with. Initial feedback is that creating an Atom parser will be the bulk of the work and I should start by defining the types. To keep this manageable, I’m only going to deal with my own feeds instead of attempting a fully compliant parser (in other words, I’ll only consider the subset of RFC4287 that’s relevant to me). Once I can parse, merge and write such feeds I should be able to iterate from there.

To make my requirements more concrete:

Only consider my own Atom feeds for now
Initially, be able to parse and emit just one Atom feed
Then be able to merge 2+ feeds, specifically:
- Use tag-based feeds from my personal site as starting points
- Be able to de-dupe content
No database or storage (construct it afresh every time)
Minimise library dependencies

Timeframes and workflow

I’ve honestly no idea how long this might take and I’m treating it as a side-project. I know there are many people out there who could produce a working version of everything in a week or two but I’m not one of them (yet). There are also a lot of ancillary things I need to learn on the way, like packaging, improving my knowledge of Git and dealing with build systems. If I had to put a vague time frame on this, I’d be thinking in months rather than weeks. It might even be the case that others start work on parts of this and ship things sooner but that’s great as I’ll probably be able to use whatever they create and move further along the chain.

In terms of workflow, everything will be done in the open, warts and all, and I expect to make embarrassing mistakes as I go. You can follow along on my freshly created OCaml Atom repo, and I’ll be using the issue tracker as the main way of dealing with bugs and features. Let the fun begin.

Acknowledgements: Thanks to Daniel, Ashish, Christophe, Philippe and Thomas for discussions on an earlier draft of this post and providing feedback on my approach.

IoT - Brave New World or Simply evolved M2M?

Amir Chaudhry — 2014-03-13T21:00:00+00:00

I was at an event run by Cambridge Wireless today on the Internet of Things. These are my notes. Although I’ve named people, I’m paraphrasing a lot so best not to treat anything you read here as a direct quote.

There were a few audience polls conducted using little keypads they handed out. There were only 30-ish keypads so it’s not necessarily indicative of how the whole room felt. I took some hasty pics of a few of the slides (mainly the poll results) so apologies for the poor quality.

“The Enterprise and the Internet of Things - Evolution or Revolution?”

John Hicklin, Principal Consultant | Commercial Enterprise Markets, CGI UK

Nowadays, you can walk into the C-suite and start a discussion about the IoT. They may not know what they’re talking about but someone’s told them they need to be thinking about it. But what is IoT?

One of the problems in industry you go to your Director and you explain this new tech and all the awesome things you can do with it and he asks the dreaded question, “So what?” Example of the Connected Toothbrush. With a connected toothbrush you can automatically have a remote relationship with your dentist. But he already drives a BMW and it’s not clear how he benefits from that. He wants you in the office where, every time you open your mouth, he sees new business development opportunities. For the moment, he neither needs nor wants a remote relationship with you. Leads to the question about who’s going to sign up for the business case.

Number of key areas that need to be addressed (paraphrased):

Crack open the Data: We are awash with data but don’t know how to use it. Problem with clients is that they really buy into the vision but they have a lot of legacy systems (green screens!) which already run things.
Identifying value(?): Volume is not a sign of success. Need to know where the value is, which means understanding it. e.g is Enron. Thousands of documents, accountants all over the place with tons of data. It took a small team of a few researchers doing a case study that discovered no tax had ever been paid. They got to the right data and asked the right questions.
Connectivity(?): M2M has grown in a silo’d manner where a device takes data and sends it to one place. This needs to open up more and give access to other areas of business.
Business operations: People matter too. Process changes, systems that allow real-time decisions, empowering individuals.

Overall, IoT is an enabler for a number of wider trends. IoT can improve operational efficiency and enhance customer experience. Normally these are in conflict with each-other. However, if we just get into a tech-fest, then we might just end up with a large number of failures as we’ve seen with ‘Big Data’

Poll - Which (categories of) business-related infrastructure categories are most urgently needed for building the Internet of Things.

“Overcoming the Internet of Things Connectivity Challenge”

David Dunn, Software Engineer, Electric Imp

Electric Imp aims to help people in Product Development in the IoT space, namely with the connectivity problem. The name ‘IMP’ came from ‘Internet Messaging Processor’. Name is also from Terry Pratchett who’s stories had devices that contained an imp which did the work :)

Building connected devices is difficult. Have to be secure, easy to set up. There are many choices for connectivity, problems with managing the code on the devices. Electric Imp picked wifi and decided to do it well. This satisfies a lot of markets but not all.

Side story of the hacked fridge sending spam emails.

Things run through Electric Imp system and they manage the security. It’s security as a service. Electric Imp manages deployment of your code too so developers can just push code and it will end up on all their devices.

Simplicity comes from one-step setup. It’s optical and uses a smartphone and flashes the screen, which sends the wifi SSID and credentials to the device (via a sensor). Customer code runs within a VM on top of with custom OS (impOS) taking care of I/O (TLS-encrypted wifi connection to service).

Difficult to debug devices in the field so cloud based- service helps with that. They do embedded stuff on the device and cloud stuff in the cloud. There’s another VM in the cloud that’s paired with the VM on the device. Heavy lifting can happens elsewhere so no need to parse JSON on the device itself.

Q - How do you deal with identity of units?
A - Imps have unique serials and the devices they’re plugged into are also unique (missed how).

“Real-life experiences from rolling out M2M networks and how we can learn from these in the shift to IoT”

Jon Lewis, Chief Innovation Officer, Plextek Consulting

Plextek was working in this field when it was called ‘telemetry’. One of the earliest things they worked on was ‘Lojack’ (car tracking). Things then shifted to ‘smart cities’ and now people talk about ‘IoT’ and want consumer devices with consumer volumes.

2014 has got to be year of IoT. However, still some way from business cases as it’s too expensive to simply track your cat (approx £100 set up fee and then around £30 pcm).

Deployment often costs more than then the sensor devices. c.f street lighting systems. Choice of radio system is critical. Projects in UK/Europe are about incremental cost savings with pay-back in less than 5yrs. In industrialising countries, there is new build which is easier then retrofit. Urbanisation creates strain and demand. Local issues relate to the Spectrum issues and local regulations (e.g mandate to use govt encryption streams). Data privacy is also concern as we don’t know who holds the monkey. That’s made even more difficult when you go abroad. Local politics also matters.

Summary:

Big opportunity in B2C: but need consolidation in B2B markets to get necessary economies of scale
Big B2B opportunity in developing countries: but is tough and needs local partners
Application developers need common platforms: to lower R&D costs and enable access to global markets
Whole value chain needs to work: building an ecosystem that connects application developers to markets is key to achieving scale

“The role of LTE in M2M”

Georg Steimel, Head of M2M Solutions, Huawei

About Huawei

Three businesses: Carrier network, Consumer business, Enterprise business
Revenues of $39Bn, No2. Telecom solution provider, 150k+ employees worldwide

Georg is in Consumer branch for Europe. Huawei wants to be top M2M module vendor in the coming years. They’re operator and system integrator agnostic.

Tech evolution brings new opportunities. Mobile data traffic needs, Network upgrading etc. Ave user expected to generated 1G data traffic per month (currently 63MB today) and there are predicted to be 50Bn+ devices. Examples of users are highly varied including monitoring of pregnant cows.

Challenges for mass rollout are a scattered market requiring complex solutions. Alignment of all parties in one adequate business model is quite tricky, esp when it comes to customer ownership. Should consider the value chain.

Complex business model:

From sales to service
Many stakeholders
- Who pays, how much, to whom, for what service, for what benefits?
Individual benefit must exceed cost
- Realistic pricing
- Appropriate effort
- Total benefit, direct vs indirect TCO
- Sustainability

Amount of data generated is also going to be huge. Smart meters in German households could send over 4TB per day, which is a huge strain for mobile networks.

LTE advantage for M2M is obviously bandwidth but that’s only relevant to a few verticals (signage, routers, and in-car entertainment). Low latency is more important for industrial alarms and medical devices.

Poll - What business models?

Break down of poll results.

“Using networked visualisation for better decision making”

Fredrik Sjostedt, Vice President | Corporate Marketing, Barco Ltd

Barco does a lot of visualising information. Displays for radiographers, stuff in cinema screens, screens in flight simulators and a lot more.

Need to take in data from sensors, display relevant information and allow operators to make SMART decisions. e.g number of CCTV cameras in London - who’s looking at them? Real ‘Big Data’ step is at the operator level.

Important to use standards, rather than proprietary systems.

Q - In transport visualisation do you have low enough latency to get real-time info?
A- Yes, e.g on a route in Brazil near Sao Palo it’s closed dues to fog. Real-time info important for understanding the whole route.

“Regulatory backdrop on IP and standards issues for the M2M eco-system”

Justin Hill, Co-Head of the Patent Prosecution Group and Purvi Parekh, Co-Head of International Telecoms, Olswang

Purvi on the Regulatory environment
No tailor made regulation v high regulatory focus. Regulatory focus from leading bodies and many of them said that existing regulation can be amended to facilitate M2M/IoT. Made more complicated by lack of globally agreed definition of IoT (which make it difficult for lawyers). There’s generally promotion of Open Standards. EU does believe that this is v important. People should read the RAND report (Nov 2013), commissioned by the EU to examine if there should be standards.

Justin on IP issues
With any standardisation process, you’re hoping to increase adoption with the trade-off of Right of IPR owners. FRAND (fair, reasonable, and non-discriminatory terms) are also important but that’s also debatable. When considering how to protect IPR, it’s worth comparing whole-system claims with protecting at different layers of the tech/value stack and build a portfolio.

“The impact of autonomous systems on surface transport”

Paul Copping, Corporate Development Director, TRL

TRL is the Transport Research Laboratory. Did some work to consider the impact of autonomous transport on the overall value chain. Some of things we’re trying now with autonomous driving have been successfully done in the past by other means (e.g car tracking a lane with magnets).

“Connected Products - Designing for Scale”

Pilgrim Beart, Founder Director, 1248-io

What is IoT? Humans don’t scale. Population is predicted to tail off at around 9Bn. Connected devices are growing and the rate of change is exponential (and increasing). In the 90s we used to be our own sysadmin with maybe one device. Now we have several and some cloud services. In the future there will be many more and they’ll have to manage themselves. There are different problem you face as you go from 1 to 10 to 100 to 10000 to 1MM+ deployed devices. Issues ranging from device updates al the way to regulation and externalities that affect your business.

In 10 years time, when all is said and done, we won’t be calling it the Internet of Things. It’ll just be called the Internet.

Q from an application developers perspective wht would you different earlier in your development if you were to go back and start again
A - AlertMe started a bit too early so there were a lot things were not off-the-shelf. Had to build. Right now can see a lot people reinventing the wheel but shouldn’t need to do this.

“A new standard to connect the Internet of Everything”

Antony Rix, Senior Consultant, TTP

One of the things TTP works on are wireless standards.

The size of the grey blobs are in an indication of market size.

Users will expect things to cost pretty much the same as they did before. Cost will be critical and as we buy things we generally still expect them to last for years.

TTP Matrix is a proprietary standard specifically for use by low-cost wireless connected devices. It’s optimised for efficient use of radio spectrum, long range, low cost and power efficient. Use-case example (DisplayData) are the pricing labels on supermarket. 50k products running through only 3 gateways (other standards would find this difficult). Low power consumption means a battery can last 10 years.

Poll - Which (categories of) technical / architecture building blocks are most urgently needed for building the Internet of Things.

Debate

Interesting discussion of Humans role in IoT. For home consumer devices people may not want to have to make any decisions (completely automated) but corporates my want all information in order to integrate and decide.

How quickly will this happen? It’s difficult to say when you’re in the middle of it. We’ll know it’s happened after the event. It takes about a decade for things to get things in to mature in the market. On the other hand, it’s happening.

Poll - Big and small data - Which statement do you agree with most/least
I only got a pic of the latter but it’s the inverse of the former (unsurprisingly).

From Jekyll site to Unikernel in fifty lines of code.

Amir Chaudhry — 2014-03-10T18:30:00+00:00

Mirage has reached a point where it’s possible to easily set up end-to-end toolchains to build unikernels! My first use-case is to be able to generate a unikernel which can serve my personal static site but to do it with as much automation as possible. It turns out this is possible with less than 50 lines of code.

I use Jekyll and GitHub Pages at the moment so I wanted a workflow that’s as easy to use, though I’m happy to spend some time up front to set up and configure things. The tools for achieving what I want are in good shape so this post takes the example of a Jekyll site (i.e this one) and goes through the steps to produce a unikernel on Travis CI (a continuous integration service) which can later be deployed. Many of these instructions already exist in various forms but they’re collated here to aid this use-case.

I will take you, dear reader, through the process and when we’re finished, the workflow will be as follows:

You’ll write your posts on your local machine as normal
A push to GitHub will trigger a unikernel build for each commit
The Xen unikernel will be pushed to a repo for deployment

To achieve this, we’ll first check that we can build a unikernel VM locally, then we’ll set up a continuous integration service to automatically build them for us and finally we’ll adapt the CI service to also deploy the built VM. Although the amount of code required is small, each of these steps is covered below in some detail. For simplicity, I’ll assume you already have OCaml and Opam installed – if not, you can find out how via the Real Word OCaml install instructions.

Building locally

To ensure that the build actually works, you should run things locally at least once before pushing to Travis. It’s worth noting that the mirage-skeleton repo contains a lot of useful, public domain examples and helpfully, the specific code we need is in mirage-skeleton/static_website. Copy both the config.ml and dispatch.ml files from that folder into a new _mirage folder in your jekyll repository.

Edit config.ml so that the two mentions of ./htdocs are replaced with ../_site. This is the only change you’ll need to make and you should now be able to build the unikernel with the unix backend. Make sure you have the mirage package installed by running $ opam install mirage and then run:

(edit: If you already have mirage, remember to opam update to make sure you’ve got the latest packages.)

$ cd _mirage
$ mirage configure --unix
$ make depend         # needed as of mirage 1.2 onward
$ mirage build
$ cd ..

That’s all it takes! In a few minutes there will be a unikernel built on your system (symlinked as _mirage/mir-www). If there are any errors, make sure that Opam is up to date and that you have the latest version of the static_website files from mirage-skeleton.

Serving the site locally

If you’d like to see this site locally, you can do so from within the _mirage folder by running unikernel you just built. There’s more information about the details of this on the Mirage docs site but the quick instructions are:

$ cd _mirage
$ sudo mirage run

# in another terminal window
$ sudo ifconfig tap0 10.0.0.1 255.255.255.0

You can now point your browser at http://10.0.0.2/ and see your site! Once you’re finished browsing, $ mirage clean will clear up all the generated files.

Since the build is working locally, we can set up a continuous integration system to perform the builds for us.

Setting up Travis CI

</img>

We’ll be using the Travis CI service, which is free for open-source projects (so this assumes you’re using a public repo). The benefit of using Travis is that you can build a unikernel without needing a local OCaml environment, but it’s always quicker to debug things locally.

Log in to Travis using your GitHub ID which will then trigger a scan of your repositories. When this is complete, go to your Travis accounts page and find the repo you’ll be building the unikernel from. Switch it ‘on’ and Travis will automatically set your GitHub post-commit hook and token for you. That’s all you need to do on the website.

When you next make a push to your repository, GitHub will inform Travis, which will then look for a YAML file in the root of the repo called .travis.yml. That file describes what Travis should do and what the build matrix is. Since OCaml is not one of the supported languages, we’ll be writing our build script manually (this is actually easier than it sounds). First, let’s set up the YAML file and then we’ll examine the build script.

The Travis YAML file - .travis.yml

The Travis CI environment is based on Ubuntu 12.04, with a number of things pre-installed (e.g Git, networking tools etc). Travis doesn’t support OCaml (yet) so we’ll use the c environment to get the packages we need, specifically, the OCaml compiler, Opam and Mirage. Once those are set up, our build should run pretty much the same as it did locally.

For now, let’s keep things simple and only focus on the latest releases (OCaml 4.01.0 and Opam 1.1.1), which means our build matrix is very simple. The build instructions will be in the file _mirage/travis.sh, which we will move to and trigger from the .travis.yml file. This means our YAML file should look like:

language: c
before_script: cd _mirage
script: bash -ex travis.sh
env:
  matrix:
  - MIRAGE_BACKEND=xen DEPLOY=0
  - MIRAGE_BACKEND=unix

The matrix enables us to have parallel builds for different environments and this one is very simple as it’s only building two unikernels. One worker will build for the Xen backend and another worker will build for the Unix backend. The _mirage/travis.sh script will clarify what each of these environments translates to. We’ll come back to the DEPLOY flag later on (it’s not necessary yet). Now that this file is set up, we can work on the build script itself.

The build script - travis.sh

To save time, we’ll be using an Ubuntu PPA to quickly get pre-packaged versions of the OCaml compiler and Opam, so the first thing to do is define which PPAs each line of the build matrix corresponds to. Since we’re keeping things simple, we only need one PPA that has the most recent releases of OCaml and Opam.

#!/usr/bin/env bash
ppa=avsm/ocaml41+opam11
echo "yes" | sudo add-apt-repository ppa:$ppa
sudo apt-get update -qq
sudo apt-get install -qq ocaml ocaml-native-compilers camlp4-extra opam

[NB: There are many other PPAs for different combinations of OCaml/Opam which are useful for testing]. Once the appropriate PPAs have been set up it’s time to initialise Opam and install Mirage.

export OPAMYES=1
opam init
opam install mirage
eval `opam config env`

We set OPAMYES=1 to get non-interactive use of Opam (it defaults to ‘yes’ for any user input) and if we want full build logs, we could also set OPAMVERBOSE=1 (I haven’t in this example). The rest should be straight-forward and you’ll end up with an Ubuntu machine with OCaml, Opam and the Mirage package installed. It’s now trivial to do the next step of actually building the unikernel!

mirage configure --$MIRAGE_BACKEND
mirage build

You can see how we’ve used the environment variable from the Travis file and this is where our two parallel builds begin to diverge. When you’ve saved this file, you’ll need to change permissions to make it executable by doing $ chmod +x _mirage/travis.sh.

That’s all you need to build the unikernel on Travis! You should now commit both the YAML file and the build script to the repo and push the changes to GitHub. Travis should automatically start your first build and you can watch the console output online to check that both the Xen and Unix backends complete properly. If you notice any errors, you should go back over your build script and fix it before the next step.

Deploying your unikernel

</img>

When Travis has finished its builds it will simply destroy the worker and all its contents, including the unikernels we just built. This is perfectly fine for testing but if we want to also deploy a unikernel, we need to get it out of the Travis worker after it’s built. In this case, we want to extract the Xen-based unikernel so that we can later start it on a Xen-based machine (e.g Amazon, Rackspace or - in our case - a machine on Bytemark).

Since the unikernel VMs are small (only tens of MB), our method for exporting will be to commit the Xen unikernel into a repository on GitHub. It can be retrieved and started later on and keeping the VMs in version control gives us very effective snapshots (we can roll back the site without having to rebuild). This is something that would be much more challenging if we were using the ‘standard’ web toolstack.

The deployment step is a little more complex as we have to send the Travis worker a private SSH key, which will give it push access to a GitHub repository. Of course, we don’t want to expose that key by simply adding it to the Travis file so we have to encrypt it somehow.

Sending Travis a private SSH key

Travis supports encrypted environment variables. Each repository has its own public key and the Travis gem uses this public key to encrypt data, which you then add to your .travis.yml file for decryption by the worker. This is meant for sending things like private API tokens and other small amounts of data. Trying to encrypt an SSH key isn’t going to work as it’s too large. Instead we’ll use travis-senv, which encodes, encrypts and chunks up the key into smaller pieces and then reassembles those pieces on the Travis worker. We still use the Travis gem to encrypt the pieces to add them to the .travis.yml file.

While you could give Travis a key that accesses your whole GitHub account, my preference is to create a new deploy key, which will only be used for deployment to one repository.

# make a key pair on your local machine
$ cd ~/.ssh/
$ ssh-keygen -t dsa -C "travis.deploy" -f travis-deploy_dsa
$ cd -

Note that this is a 1024 bit key so if you decide to use a 2048 bit key, then be aware that Travis sometimes has issues. Now that we have a key, we can encrypt it and add it to the Travis file.

# on your local machine

# install the necessary components
$ gem install travis
$ opam install travis-senv

# chunk the key, add to yml file and rm the intermediate
$ travis-senv encrypt ~/.ssh/travis-deploy_dsa _travis_env
$ cat _travis_env | travis encrypt -ps --add
$ rm _travis_env

travis-senv encrypts and chunks the key locally on your machine, placing its output in a file you decide (_travis_env). We then take that output file and pipe it to the travis ruby gem, asking it to encrypt the input, treating each line as separate and to be appended (-ps) and then actually adding that to the Travis file (--add). You can run $ travis encrypt -h to understand these options. Once you’ve run the above commands, .travis.yml will look as follows.

language: c
before_script: cd _mirage
script: bash -ex travis.sh
env:
  matrix:
  - MIRAGE_BACKEND=xen DEPLOY=0
  - MIRAGE_BACKEND=unix
  global:
  - secure: ".... encrypted data ...."
  - secure: ".... encrypted data ...."
  - secure: ".... encrypted data ...."
  ...

The number of secure variables added depends on the type and size of the key you had to chunk, so it could vary from 8 up to 29. We’ll commit these additions later on, alongside additions to the build script.

At this point, we also need to make a repository on GitHub and add the public deploy key so that Travis can push to it. Once you’ve created your repo and added a README, follow GitHub’s instructions on adding deploy keys and paste in the public key (i.e. the content of travis-deploy_dsa.pub).

Now that we can securely pass a private SSH key to the worker and have a repo that the worker can push to, we need to make additions to the build script.

Committing the unikernel to a repository

Since we can set DEPLOY=1 in the YAML file we only need to make additions to the build script. Specifically, we want to assure that: only the Xen backend is deployed; only pushes to the repo result in deployments, not pull requests (we do still want builds for pull requests).

In the build script (_mirage/travis.sh), which is being run by the worker, we’ll have to reconstruct the SSH key and configure Git. In addition, Travis gives us a set of useful environment variables so we’ll use the latest commit hash ($TRAVIS_COMMIT) to name the the VM (which also helps us trace which commit it was built from).

It’s easier to consider this section of code at once so I’ve explained the details in the comments. This section is what you need to add at the end of your existing build script (i.e straight after mirage build).

# Only deploy if the following conditions are met.
if [ "$MIRAGE_BACKEND" = "xen" \
            -a "$DEPLOY" = "1" \
            -a "$TRAVIS_PULL_REQUEST" = "false" ]; then

    # The Travis worker will already have access to the chunks
    # passed in via the yaml file. Now we need to reconstruct 
    # the GitHub SSH key from those and set up the config file.
    opam install travis-senv
    mkdir -p ~/.ssh
    travis-senv decrypt > ~/.ssh/id_dsa # This doesn't expose it
    chmod 600 ~/.ssh/id_dsa             # Owner can read and write
    echo "Host some_user github.com"   >> ~/.ssh/config
    echo "  Hostname github.com"          >> ~/.ssh/config
    echo "  StrictHostKeyChecking no"     >> ~/.ssh/config
    echo "  CheckHostIP no"               >> ~/.ssh/config
    echo "  UserKnownHostsFile=/dev/null" >> ~/.ssh/config

    # Configure the worker's git details
    # otherwise git actions will fail.
    git config --global user.email "user@example.com"
    git config --global user.name "Travis Build Bot"

    # Do the actual work for deployment.
    # Clone the deployment repo. Notice the user,
    # which is the same as in the ~/.ssh/config file.
    git clone git@some_user:amirmc/www-test-deploy
    cd www-test-deploy

    # Make a folder named for the commit. 
    # If we're rebuiling a VM from a previous
    # commit, then we need to clear the old one.
    # Then copy in both the config file and VM.
    rm -rf $TRAVIS_COMMIT
    mkdir -p $TRAVIS_COMMIT
    cp ../mir-www.xen ../config.ml $TRAVIS_COMMIT

    # Compress the VM and add a text file to note
    # the commit of the most recently built VM.
    bzip2 -9 $TRAVIS_COMMIT/mir-www.xen
    git pull --rebase
    echo $TRAVIS_COMMIT > latest    # update ref to most recent

    # Add, commit and push the changes!
    git add $TRAVIS_COMMIT latest
    git commit -m "adding $TRAVIS_COMMIT built for $MIRAGE_BACKEND"
    git push origin master
    # Go out and enjoy the Sun!
fi

At this point you should commit the changes to ./travis.yml (don’t forget the deploy flag) and _mirage/travis.sh and push the changes to GitHub. Everything else will take place automatically and in a few minutes you will have a unikernel ready to deploy on top of Xen!

You can see both the complete YAML file and build script in use on my test repo, as well as the build logs for that repo and the deploy repo with a VM.

[Pro-tip: If you add *[skip ci] anywhere in your commit message, Travis will skip the build for that commit. This is very useful if you’re making minor changes, like updating a README.]*

Finishing up

Since I’m still using Jekyll for my website, I made a short script in my jekyll repository (_deploy-unikernel.sh) that builds the site, commits the contents of _site and pushes to GitHub. I simply run this after I’ve committed a new blog post and the rest takes care of itself.

#!/usr/bin/env bash
jekyll build
git add _site
git commit -m 'update _site'
git push origin master

Congratulations! You now have an end-to-end workflow that will produce a unikernel VM from your Jekyll-based site and push it to a repo. If you strip out all the comments, you’ll see that we’ve written less than 50 lines of code! Admittedly, I’m not counting the 80 or so lines that came for free in the *.ml files but that’s still pretty impressive.

Of course, we still need a machine to take that VM and run it but that’s a topic for another post. For the time-being, I’m still using GitHub Pages but once the VM is hosted somewhere, I will:

Turn off GitHub Pages and serve from the VM – but still using Jekyll in the workflow.
Replace Jekyll with OCaml-based static-site generation.

Although all the tools already exist to switch now, I’m taking my time so that I can easily maintain the code I end up using.

Expanding the script for testing

You may have noticed that the examples here are not very flexible or extensible but that was a deliberate choice to keep them readable. It’s possible to do much more with the build matrix and script, as you can see from the Travis files on my website repo, which were based on those of the Mirage site and Mort’s site. Specifically, you can note the use of more environment variables and case statements to decide which PPAs to grab. Once you’ve got your builds working, it’s worth improving your scripts to make them more maintainable and cover the test cases you feel are important.

Not just for static sites (surprise!)

You might have noticed that in very few places in the toolchain above have I mentioned anything specific to static sites per se. The workflow is simply (1) do some stuff locally, (2) push to a continuous integration service which then (3) builds and deploys a Xen-based unikernel. Apart from the convenient folder structure, the specific work to treat this as a static site lives in the *.ml files, which I’ve skipped over for this post.

As such, the GitHub+Travis workflow we’ve developed here is quite general and will apply to almost any unikernels that we may want to construct. I encourage you to explore the examples in the mirage-skeleton repo and keep your build script maintainable. We’ll be using it again the next time we build unikernel devices.

Acknowledgements: There were lots of things I read over while writing this post but there were a few particularly useful things that you should look up. Anil’s posts on Testing with Travis and Travis for secure deployments are quite succinct (and were themselves prompted by Mike Lin’s Travis post several months earlier). Looking over Mort’s build script and that of mirage-www helped me figure out the deployment steps as well as improve my own script. Special thanks also to Daniel, Leo and Anil for commenting on an earlier draft of this post.

Scientific evidence for health supplements

Amir Chaudhry — 2014-01-30T00:00:00+00:00

This is an update to an earlier version of this infographic, which I came across two years ago. It was interesting at the time since it did a good job at displaying the myriad supplements in a way that indicated both their likely efficacy, according to research papers, and how popular the search term was. For example, the previous image showed ‘strong’ evidence for licorice root as an aid for coughs, but it had a much smaller search volume than the equivalently effective fish-oil for blood pressure. I also really liked the concept of the ‘Worth it Line’.

The graphic has been updated a couple of times now, with the most recent change a few weeks ago. There are quite a few differences as garlic didn’t feature so prominently on the previous image and it looks like a number of things may have been removed (e.g licorice root).

Vitamin D, as beneficial to general health, is still way up there. I decided to try Vitamin D3 supplements over the winter, under the theory that I’m not getting enough sun so a supplement may be useful. I was quite surprised at how much better I felt after a day or so of taking it, to the point where I tried looking up work on Vitamin D and mood*. Scanning the research seemed to indicate that there may be links between Vitamin D, serotonin levels and depression but the extent isn’t clear and conclusions were mixed. Of course, that mostly supports what the infographic is saying and it puts it above the ‘Worth it Line’ (for depression specifically but not general ‘mood’ per se).

I encourage you to go visit the site as there’s an interactive version, where you can highlight a bubble to get more info and be transported to the main study it references. If you like, you can also dig into the spreadsheet itself.

via informationisbeautiful.net

* I'm well aware that I may be experiencing nothing more than a placebo effect :)

Switching from Bootstrap to Zurb Foundation

Amir Chaudhry — 2013-11-26T21:05:00+00:00

I’ve just updated my site’s HTML/CSS and moved from Twitter Bootstrap to Zurb Foundation. This post captures my subjective notes on the migration.

My use of Bootstrap

When I originally set this site up, I didn’t know what frameworks existed or anything more than the basics of dealing with HTML (and barely any CSS). I came across Twitter Bootstrap and immediately decided it would Solve All My Problems. It really did. Since then, I’ve gone through one ‘upgrade’ with Bootstrap (from 1.x to 2.x), after which I dutifully ignored all the fixes and improvements (note that Bootstrap was up to v2.3.2 while I was still using v2.0.2).

For the most part, this was fine with me but for a while now, I’ve been meaning to make this site ‘responsive’ (read: not look like crap from a mobile). Bootstrap v3 purports to be mobile-first so upgrading would likely give me what I’m after but v3 is not backwards compatible, meaning I’d have to rewrite parts of the HTML. Since this step was unavoidable, it led me to have another look at front-end frameworks, just to see if I was missing anything. This was especially relevant since we’d just released the new OCaml.org website, itself built with Bootstrap v2.3.1 (we’d done the design/templating work long before v3 was released). It would be useful to know what else is out there for any future work.

Around this time I discovered Zurb Foundation and also the numerous comparisons between them (note: Foundation seems to come out ahead in most of those). A few days ago, the folks at Zurb released version 5, so I decided that now is the time to kick the tires. For the last few days, I’ve been playing with the framework and in the end I decided to migrate my site over completely.

Swapping out one framework for another

Over time, I’ve become moderately experienced with HTML/CSS and I can usually wrangle things to look the way I want, but my solutions aren’t necessarily elegant. I was initially concerned that I’d already munged things so much that changing anything would be a pain. When I first put the styles for this site together, I had to spend quite a bit of time overwriting Bootstrap’s defaults so I was prepared for the same when using Foundation. Turns out that I was fine. I currently use Jekyll (and Jekyll Bootstrap) so I only had three template files and a couple of HTML pages to edit and because I’d kept most of my custom CSS in a separate file, it was literally a case of swapping out one framework for another and bug-fixing from there onwards. There’s definitely a lesson here in using automation as much as possible.

Customising the styles was another area of concern but I was pleasantly surprised to find I needed less customisation than with Bootstrap. This is likely because I didn’t have to override as many defaults (and probably because I’ve learned more about CSS since then). The one thing I seemed to be missing was a way to deal with code sections, so I just took what Bootstrap had and copied it in. At some point I should revisit this.

It did take me a while to get my head around Foundation’s grid but it was worth it in the end. The idea is that you should design for small screens first and then adjust things for larger screens as necessary. There are several different default sizes which inherit their properties from the size below, unless you explicitly override them. I initially screwed this up by explicitly defining the grid using the small-# classes, which obviously looks ridiculous on small screens. I fixed it by swapping out small-# for medium-# everywhere in the HTML, after which everything looked reasonable. Items flowed sensibly into a default column for the small screens and looked acceptable for larger screens and perfectly fine on desktops. I could do more styling of the mobile view but I’d already achieved most of what I was after.

Fixing image galleries and embedded content

The only additional thing I used from Bootstrap was the Carousel. I’d written some custom helper scripts that would take some images and thumbnails from a specified folder and produce clickable thumbnails with a slider underneath. Foundation provides Orbit, so I had to spend time rewriting my script to produce the necessary HTML. This actually resulted in cleaner HTML and one of the features I wanted (the ability to link to a specific image) was available by default in Orbit. At this point I also tried to make the output look better for the case where JavaScript is disabled (in essence, each image is just displayed as a list). Below is an example of an image gallery, taken from a previous post, when I joined the computer lab.

Foundation also provides a component called Flex Video, which allows the browser to scale videos to the appropriate size. This fix was as simple as going back through old posts and wrapping anything that was <iframe> in a <div class="flex-video">. It really was that simple and all the Vimeo and YouTube items scaled perfectly. Here’s an example of a video from an earlier post, where I gave a walkthrough of the ocaml.org site. Try changing the width of your browser window to see it scale.

Video demo

Framework differences

Another of the main difference between the two frameworks is that Bootstrap uses LESS to manage its CSS whereas Foundation uses SASS. Frankly, I’ve no experience with either of them so it makes little difference to me. It’s worth bearing in mind for anyone who’s workflow does involve pre-processing. Also, Bootstrap is available under the Apache 2 License, while Foundation is released under the MIT license.

Summary

Overall, the transition was pretty painless and most of the time was spent getting familiar with the grid, hunting for docs/examples and trying to make the image gallery work the way I wanted. I do think Bootstrap’s docs are better but Foundation’s aren’t bad.

Although this isn’t meant to be a comparison, I much prefer Foundation to Bootstrap. If you’re not sure which to use then I think the secret is in the names of the frameworks.

Bootstrap (for me) was a great way to ‘bootstrap’ a site quickly with lots of acceptable defaults – it was quick to get started but took some work to alter.
Foundation seems to provide a great ‘foundation’ on which to create more customised sites – it’s more flexible but needs more upfront thought.

That’s pretty much how I’d recommend them to people now.

Announcing the new OCaml.org

Amir Chaudhry — 2013-11-20T23:00:00+00:00

As some of you may have noticed, the new OCaml.org site is now live!

The DNS may still be propagating so if http://ocaml.org hasn’t updated for you then try http://166.78.252.20. This post is in two parts: the first is the announcement and the second is a call for content.

New OCaml.org website design!

The new site represents a major milestone in the continuing growth of the OCaml ecosystem. It’s the culmination of a lot of volunteer work over the last several months and I’d specifically like to thank Christophe, Ashish and Philippe for their dedication (the commit logs speak volumes).

We began this journey just over 8 months ago with paper, pencils and a lot of ideas. This led to a comprehensive set of wireframes and walk-throughs of the site, which then developed into a collection of Photoshop mockups. In turn, these formed the basis for the html templates and style sheets, which we’ve adapted to fit our needs across the site.

Alongside the design process, we also considered the kind of structure and workflow we aspired to, both as maintainers and contributors. This led us to develop completely new tools for Markdown and templating in OCaml, which are now available in OPAM for the benefit all.

Working on all these things in parallel definitely had it challenges (which I’ll write about separately) but the result has been worth the effort.

The journey is ongoing and we still have many more improvements we hope to make. The site you see today primarily improves upon the design, structure and workflows but in time, we also intend to incorporate more information on packages and documentation. With the new tooling, moving the website forward will become much easier and I hope that more members of the community become involved in the generation and curation of content. This brings me to the second part of this post.

Call for content

We have lots of great content on the website but there are parts that could do with a refresh and gaps that could be filled. As a community driven site, we need ongoing contributions to ensure that the site best reflects its members.

For example, if you do commercial work on OCaml then maybe you’d like to add yourself to the support page? Perhaps there are tutorials you can help to complete, like 99 problems? If you’re not sure where to begin, there are already a number of content issues you could contribute to.

Although we’ve gone through a bug-hunt already, feedback on the site is still very welcome. You can either create an issue on the tracker (preferred), or email the infrastructure list.

It’s fantastic how far we’ve come and I look forward to the next phase!

Migration plan for the OCaml.org redesign

Amir Chaudhry — 2013-11-06T11:00:00+00:00

We’re close to releasing the new design of ocaml.org but need help from the OCaml community to identify and fix bugs before we switch next week.

Ashish, Christophe, Philippe and I have been discussing how we should go about this and below is the plan for migration. If anyone would like to discuss any of this, then the infrastructure list is the best place to do so.

We’ve made a new branch on the main ocaml.org repository with the redesign. This branch is a fork of the master and we’ve simply cleaned up and replayed our git commits there.
We’ve built a live version of the new site, which is visible at http://preview.ocaml.org - this is rebuilt every few minutes from the branch mentioned above.
Over the course of one week, we ask the community to review the new site and report any bugs or problems on the issue tracker. We triage those bugs to identify any blockers and work on those first. This is the phase we’ll be in from today.
After one week (7 days), and after blocking bugs have been fixed, we merge the redesign branch into the master branch. This would effectively present the new site to the world.

During the above, we would not be able to accept any new pull requests on the master branch but would be happy to accept them on the new, redesign branch. Hence, restricting the time frame to one week.

Please note that the above is only intended to merge the design and toolchain for the new site. Specifically, we’ve created new landing pages, have new style sheets and have restructured the site’s contents as well as made some new libraries (OMD and MPP). The new toolchain means people can write files in markdown, which makes contributing content a lot easier.

Since the files are on GitHub, people don’t even need to clone the site locally to make simple edits (or even add new pages). Just click the ‘Edit this page’ link in the footer to be taken to the right file in the repository and GitHub’s editing and pull request features will allow you to make changes and submit updates, all from within your browser (see the GitHub Article for details).

There is still work to be done on adding new features but the above changes are already a great improvement to the site and are ready to be reviewed by the OCaml community and merged.

Review of the OCaml FPDays tutorial

Amir Chaudhry — 2013-10-28T12:30:00+00:00

Last Thursday a bunch of us from the OCaml Labs team gave an OCaml tutorial at the FPDays conference (an event for people interested in Functional Programming). Jeremy and I led the session with Leo, David and Philippe helping everyone progress and dealing with questions.

It turned out to be by far the most popular session at the conference with over 20 people all wanting to get to grips with OCaml! An excellent turnout and a great indicator of the interest that’s out there, especially when you offer a hands-on session to people. This shouldn’t be a surprise as we’ve had good attendance for the general OCaml meetups I’ve run and also the compiler hacking sessions, which Jeremy and Leo have been building up (do sign up if you’re interested in either of those!). We had a nice surprise for attendees, which were uncorrected proof copies of Real World OCaml and luckily, we had just enough to go around.

For the tutorial itself, Jeremy put together a nice sequence of exercises and a skeleton repo (with helpful comments in the code) so that people could dive in quickly. The event was set up to be really informal and the rough plan was as following:

Installation/Intro - We checked that people had been able to follow the installation instructions, which we’d sent them in advance. We also handed out copies of the book and made sure folks were comfortable with OPAM.
Hello world - A light intro to get people familiar with the OCaml syntax and installing packages with OPAM. This would also help people to get familiar with the toolchain, workflow and compilation.
Monty Hall browser game - Using js_of_ocaml, we wanted people to create and run the Monty Hall problem in their browser. This would give people a taste of some real world interaction by having to deal with the DOM and interfaces. If folks did well, they could add code to keep logs of the game results.
Client-server game - The previous game was all in the browser (so could be examined by players) so here the task was to split it into a client and server, ensuring the two stay in sync. This would demonstrate the re-usability of the OCaml code already written and give people a feel for client server interactions. If people wanted to do more, they could use ctypes and get better random numbers.

We did manage to stick to the overall scheme as above and we think this is a great base from which to improve future tutorials. It has the really nice benefit of having visual, interactive elements and the ability to run things both in the browser as well as on the server is a great way to show the versatility of OCaml. js_of_ocaml is quite a mature tool and so it’s no surprise that it’s also used by companies such as Facebook (see the recent CUFP talk by Julien Verlaguet - skip to 19:00).

We learned a lot from running this session so we’ve captured the good, the bad and the ugly below. This is useful for anyone who’d like to run an OCaml tutorial in the future and also for us to be aware of the next time we do this. I’ve incorporated the feedback from the attendees as well as our own thoughts.

Things we learnt

The Good

Most people really did follow the install instructions beforehand. This made things so much easier on the day as we didn’t have to worry about compile times and people getting bored. A few people had even got in touch with me the night before to sort out installation problems.
Many folks from OCaml Labs also came over to help people, which meant no-one was waiting longer than around 10 seconds before getting help.
We had a good plan of the things we wanted to cover but we were happy to be flexible and made it clear the aim was to get right into it. Several folks told us that they really appreciated this loose (as opposed to rigid) structure.
We didn’t spend any time lecturing the room but instead got people right into the code. Having enough of a skeleton to get something interesting working was a big plus in this regard. People did progress from the early examples to the later ones fairly well.
We had a VM with the correct set up that we could log people into if they were having trouble locally. Two people made use of this.
Of course, It was great to have early proofs of the book and these were well-received.

The Bad

In our excitement to get right into the exercises, we didn’t really give an overview of OCaml and its benefits. A few minutes at the beginning would be enough and it’s important so that people can leave with a few sound-bites.
Not everyone received my email about installation, and certainly not the late arrivals. This meant some pain getting things downloaded and running especially due to the wifi (see ‘Ugly’ below).
A few of the people who had installed, didn’t complete the instructions fully but didn’t realise this until the morning of the session. There was a good suggestion about having some kind of test to run that would check everything, so you’d know if there was something missing.
We really should have had a cut-off where we told people to use VMs instead of fixing installation issues and 10-15 minutes would have been enough. This would have been especially useful for the late-comers.
We didn’t really keep a record of the problems folks were having so we can’t now go back and fix underlying issues. To be fair, this would have been a little awkward to do ad-hoc but in hindsight, it’s a good thing to plan for.

The Ugly

The only ugly part was the wifi. It turned out that the room itself was a bit of a dead-spot and that wasn’t helped by 30ish devices trying to connect to one access point! Having everyone grab packages at the same time in the morning probably didn’t help. It was especially tricky as all our mitigation plans seemed to revolve around at least having local connectivity. In any case, this problem only lasted for the morning session and was a little better by the afternoon. I’d definitely recommend a backup plan in the case of complete wifi failure next time! One such plan that Leo got started on was to put the repository and other information onto a flash drive that could be shared with people. We didn’t need this in the end but it’ll be useful to have something like this prepared for next time. If anyone fancies donating a bunch of flash drives, I’ll happily receive them!

Overall, it was a great session and everyone left happy, having completed most of the tutorial (and with a book!). A few even continued at home afterwards and got in touch to let us know that they got everything working. It was a great session and thanks to Mark, Jacqui and the rest of the FPDays crew for a great conference!

(Thanks to Jeremy, Leo, David and Philippe for contributions to this post)

FP Days OCaml Session

Amir Chaudhry — 2013-10-22T21:00:00+00:00

On Thursday, along with Jeremy and Leo, I’ll be running an OCaml Hands-on Session at the FPDays conference. Below are some prep instructions for attendees.

Preparation for the session

If you’re starting from scratch, installation can take some time so it’s best to get as much done in advance as possible. You’ll need OPAM (the package manager), OCaml 4.01 (available through OPAM) and a few libraries before Thursday. If you have any issues, please contact Amir.

OPAM: Follow the instructions for your platform at http://opam.ocaml.org/doc/Quick_Install.html. OPAM requires OCaml so hopefully the relevant dependencies will kick in and you’ll get OCaml too (most likely version 3.12). You can get a cup of coffee while you wait. After installation, run opam init to initialise OPAM.
OCaml 4.01: We actually need the latest version of OCaml but OPAM makes this easy. Just run the following (and get more coffee):

$ opam update
$ opam switch 4.01.0
$ eval `opam config env`

Libraries: For the workshop you will need to check that you have the following installed: libffi, pcre and pkg-config. This will depend on your platform so on a Mac with homebrew I would do brew install libffi pcre pkg-config or on Debian or Ubuntu apt-get install libffi-dev. After this, two OCaml packages it’s worth installing in advance are core and js_of_ocaml so simply run:

$ opam install core js_of_ocaml

OPAM will take care of the dependencies and the rest we can get on the day!

The real problem of humanity

Amir Chaudhry — 2013-10-01T18:45:00+00:00

I heard this today. It made me stop and think so I decided to look it up.

“The real problem of humanity is the following: we have paleolithic emotions; medieval institutions; and god-like technology. And it is terrifically dangerous, and it is now approaching a point of crisis overall.”

By Edward O Wilson and taken from Harvard Magazine, 2009

Feedback requested on the OCaml.org redesign

Amir Chaudhry — 2013-09-24T14:00:00+00:00

There is a work-in-progress site at ocaml-redesign.github.io, where we’ve been developing both the tools and design for the new ocaml.org pages. This allows us to test our tools and fix issues before we consider merging changes upstream.

There is a more detailed post coming about all the design work to date and the workflow we’re using, but in the meantime, feedback on the following areas would be most welcome. Please leave feedback in the form of issues on the ocaml.org sandbox repo. You can also raise points on the infrastructure mailing list.

OCaml Logo - There was some feedback on the last iteration of the logo, especially regarding the font, so there are now several options to consider. Please look at the images on the ocaml.org GitHub wiki and then leave your feedback on issue #16 on the sandbox repo.
Site design - Please do give feedback on the design and any glitches you notice. Text on each of the new landing pages is still an initial draft so comments and improvements there are also welcome (specifically: Home Page, Learn, Documentation, Platform, Community). There are already a few known issues, so do add your comments to those threads first.

I'm working on The Internet of my Things

Amir Chaudhry — 2013-09-02T00:00:00+00:00

For a long time I’ve been interested what the future holds in terms of how we will interact with each other and our environments. Back in the day, these areas used to be called ubiquitous computing and home automation but have since been subsumed by the (only slightly) catchier moniker of Internet of Things (IoT).

A couple of my ideas involved controlling devices at home via SMS messages (this was way before smart phones and ‘apps’ came along). For example, being able to set my VCR to record a show while I’m out* or pre-heat the oven when I’m heading home with a frozen pizza^†. Of course, if I were to implement these now I’d take a completely different approach. With all the advances we’ve made, the Internet of Things has huge potential to change our lives the way the Internet itself did.

However, if we take the last decade of the Internet and Web as an example of how the IoT may play out it’s actually not a pretty picture. We’re in the age of the Internet behemoths, with huge centralised data silos containing lots of our personal data. We trust such services to secure our data but repeated breaches undermine that trust. Sure, there is some choice among providers but it’s fast becoming a case of “choose your dictator”. Our information is trapped, interoperating is becoming difficult and to top it off, these giants are busy fighting legal battles with each other instead of out-competing. Users just get caught in the cross-fire.

If we extrapolate this to the Internet of Things then, ultimately, it’ll become a winner-take-all-my-data arrangement, with lock-ins to restrict me from migrating. Remember, we’re talking about connecting more devices to the Internet and giving almost ubiquitous access to our daily lives and habits. Trading more of our personal lives, from within our homes, with increasingly faceless companies and for not very much in return, isn’t particularly appealing.

If the IoT plays out this way, then we will effectively increase the Privacy Tax that we’ve already had to pay for our social networking tools. In order to benefit from the world of connected devices, we have to adjust (again) the notion of ownership. I want this to be the Internet of my Things, so how do we make that happen?

To really achieve that, we need to have software infrastructure that incorporates Privacy by Design, so that an individual can regain the ability to control their data and decide for themselves whom to share it with. These infrastructure tools must also work in a distributed fashion, so there’s no requirement to ally yourself to any particular centralised service. Of course, the tools themselves also have to be open-source, so that developers can be reassured that the ground they’re building their products on won’t be taken away from them.

In the last year and a half, I’ve been exploring these ideas at the Cambridge Computer Laboratory. A lot of cutting edge work is being done on the problems of Identity and connectivity as well as the operating systems for the future. Infrastructure and tools are being shaped that together form a new toolstack to make the creation and management of decentralised networks easier^‡. There’s a larger picture emerging from this work and it’s time to start talking more about it and describing the pieces. We’re calling this open-source toolstack Nymote and there’s a new site where you can read more about it and keep up with news and releases. The introductory post there describes more about the background and what we’re working on.

I’ll be writing about this work both here and on the Nymote Blog and I’d love to hear from others who are interested in this area – please do get in touch with me or leave a comment. If you’d like to sign up for updates by email, you can join the Nymote list here.

* Sky+ came out with this feature a few months after I started discussing it with people.

^† Yes, people told me this was insane and I'd burn my house down. Luckily for my housemates, I never actually got around to implementing anything.

^‡ This is somewhat ironic as decentralisation was one of the founding tenets of the Internet. We're just helping people get back to that.

Wireframe demos for OCaml.org

Amir Chaudhry — 2013-03-14T00:00:00+00:00

Making mockups

Over the last few months, I’ve been working on various aspects of the OCaml.org design project. This covers things like the design, information architecture and how to incorporate new functionality. One of the methods for thinking through these was to put together a bunch of wireframes using Balsamiq and use these to express (and generate) ideas as well as get feedback quickly.

If you haven’t used wireframes before, think of them as a slightly more advanced form of sketching things out on a whiteboard. The best aspect is that it’s far quicker, easier and cheaper to iterate using wireframes than on an actual website. As you’ll see below, you can also convey a lot of information about how a site might work by showing people a clickable demo.

I want to make this work public and I thought the best way would be to show you some screencasts of how the upcoming OCaml.org website might work and also make the demo available to all of you. The three videos below cover three aspects of the site and I’d encourage you to go through them in order (about 16 minutes in total). Apologies if my screen isn’t particularly clear in the videos but you can visit the demo for yourself and see things in more detail (link and info on feedback at the end of this post).

Video walkthroughs

For those who’d like to watch the videos back-to-back and scaled to fit your browser window, you can view the Vimeo album in ‘couchmode’. Otherwise, individual videos are embedded below (total time 16m17s).

Video Part 1 - Overview - http://player.vimeo.com/video/61768157

Video Part 2 - Documentation - http://player.vimeo.com/video/61768235

Video Part 3 - Continuous Integration - http://player.vimeo.com/video/61768273

Public wireframe demo

A demo you can interact with can be found at OCaml.org wireframe demo and image files for each page are available on the github ocaml.org wiki. Please bear in mind the following:

Not everything that looks like it might be clickable actually is (and vice versa). There’ll be a toggle on the bottom right of the browser window that will highlight what can be clicked.
There are parts of the site which are ‘work in progress’ and are marked as such.
The designs you see aren’t necessarily final. Your feedback will help shape our decisions and the best way to provide it is via the infrastructure mailing list.

OCaml - Installation and hello world

Amir Chaudhry — 2012-10-04T00:00:00+00:00

This post is part of a series where I'm trying to teach myself OCaml.
You might want to start at the beginning.

It’s been a few days into my OCaml experience so this is a write-up of what I’ve come across so far. I’ve spent more time reading background rather than getting stuck in so I’ve copied in some of the links that I’ve found interesting/useful at the end.

Installation

There are number of ways you can get OCaml on your machine. The most obvious would be to get the source via the release page, but you could also use something called GODI, which apparently bundles a bunch of other stuff alongside the language.

I don’t really want to install from source ‘by hand’ and I definitely don’t need all the stuff that comes with GODI. I happen to use Homebrew on my machine, so I checked to see if I can install that way. Turns out I can, so that’s what I ended up doing.

$ brew install objective-caml

The current version of OCaml is 4.00.0 and you can check the version you have installed by typing ocaml -version in your terminal.

notes

Be aware that Homebrew has it’s own installation process and depends on Ruby. To get Ruby, someone recommended that I install via Ruby Version Manager. To be able to do the above, you’ll need to have the OSX developer tools installed, which means having Xcode.

Hello World!

The first thing to do is get a hello world programme working. Since OCaml is a compiled language, that means writing the necessary source code into a file, compiling it and then executing it. In this case I only need one line that prints ‘hello world’ to the screen and I’m taking it from INRIA’s site (link).

print_string "Hello, world!\n";;

Take the one line above and save it in a file called hello.ml. Now we need to compile that file using the OCaml compiler. At the command prompt, type the following:

$ ocamlc -o hello hello.ml
$ ./hello

Since I don’t really understand the first line I should break it down. ocamlc is the command to invoke the compiler, the option -o hello means you want to name the output executable to be (in this case) ‘hello’ and the final argument is the source code file. It’s useful to look at the man page for ocamlc to see what other options are available. The second line executes the programme, which prints hello world to the screen with a line-break.

I also notice that I now have two other files in addition to the source, hello.cmi and hello.cmo. According to the man page, these are the ‘compiled interface’ and ‘compiled object code file’ respectively. I have no idea what that means but removing the files doesn’t affect the executable.

OCaml ‘toplevel’

Even though OCaml is a compiled language, there’s something called ‘toplevel’ that allows interactive use (more on toplevel). To enter this mode, you simply have to type ocaml at the prompt so let’s try running the above hello world program using toplevel.

$ ocaml
        OCaml version 4.00.0

# print_string "Hello, world!\n";;
Hello, world!
- : unit = ()
# #quit;;
$

The $ prompt is the command line and the # prompt is where toplevel is awaiting a new line of input. The input can span multiple lines and is terminated by ;; (as in the source code).

To exit toplevel, type #quit;;. It took me three attempts to get that right. I haven’t really played around with toplevel much and I think I’m likely to stick with source code and compiling until I’m a little more comfortable with the syntax.

That’s pretty much where I am for the moment and everything so far has been straightforward. Obviously, I should push myself a bit harder :)

Resources

This is the material I found and have been looking over for the last couple of days. Useful as background but I’d say I’m in danger of (semi-productive) procrastination if I’m not careful.

OCaml Manual (from INRIA)
OCaml examples (also INRIA)
OCaml Intro (wikibooks)
Minsky on ML (Jane Street)
OCaml pros (Steve Yegge)
OCaml cons (Sam Steingold)
Think OCaml (PDF book)

Thirty Days of OCaml

Amir Chaudhry — 2012-10-01T00:00:00+00:00

I’ve set myself a challenge that over the next thirty days, I’ll teach myself some functional programming using OCaml. This will be my first experience of FP in addition to learning a new language so I expect it’ll be quite challenging.

As I go along, I’ll try and write regular posts – hopefully one a day – describing my experiences and frustrations as well as the questions that occur to me. I suspect a number of the posts might be summaries to help me organise my understanding, especially where things seem unexpected. For the times where I can formulate well-posed questions, I’ll put them up in places like Stack Overflow as well as describing them here. Either way, keeping some form of diary will be useful for me and maybe also for other folks who’ve considered trying out FP and OCaml in particular.

Obviously, I’m not completely new to programming but I would say I have any deep experience (especially compared to the folks I tend to hang out with). Just to give you a better picture, my coding experience so far has involved:

Some C/C++ with a little OOP programming for a High Energy Physics simulation (during my undergrad - a long time ago so I’ve forgotten everything)
A bunch of Matlab and Bash scripting for analysis of neuroimaging data (during my PhD)
A bunch of VBA within MS Excel for analysing behavioural data (also during my PhD – I really wish someone had pointed me to Python back then)
Part of Learn Python the Hard Way (I keep dipping in and out of this)

I tried to dump most of my old scripts – at least those I could find – into a github repo, but I still haven’t collated the Matlab bits. You can judge my coding skills for yourself.

By the end of the 30 days, I’m hoping to have the ability to pick up and read other people’s OCaml code and write basic stuff for myself. I don’t actually have a particular project in mind but I’m not going to let that be an excuse for not getting started.

I’ll post again tomorrow with the mundane parts of how to get things installed and running on a Mac*. Onward!

*Admittedly, I already did this some time ago but I want this series of posts to start right from the beginning.

I'm in San Francisco (and later NYC)

Amir Chaudhry — 2012-08-28T00:00:00+00:00

Actually, I’ve been here for a while now and feeling (mostly) settled in. I’m in the US for another couple of months, most of which will be in SF, with a short trip to Colorado and New York thrown in.

So far, I’ve met a bunch of folks from ‘back home’, people that they know, and a handful of new ones. While I’m out here, I’m hoping to meet a lot more people who might be interested in the work we’re doing in the Lab. Specifically things related to Software Defined Networking, Identity and Functional Programming – basically anything that overlaps with projects like Signpost and Mirage (if you happen to know anyone, bonus points for putting me in touch).

I’ve also managed to get out of town, courtesy of friends-with-a-car, and saw the Monterey Aquarium (some pics below). Not been hiking yet but will hopefully make it out of town again at some point.

Overall, SF is a very interesting place and I can see the draw for people to come here. Although a lot of the companies are further south, there’s much more ‘stuff’ happening here. I’m looking forward to the next couple of months.

The Illusion of Choice

Amir Chaudhry — 2012-04-26T00:00:00+00:00

Click on the image for a bigger version.

Not sure where this came from but it’s interesting (e.g I had no idea that Nestle did pet food nor that they had a stake in L’Oreal).

Pics from TEDx Granta

Amir Chaudhry — 2012-04-13T00:00:00+00:00

About a month ago I was at TEDx Granta. The theme was Alive & Kicking and it was a good event with some really interesting speakers, including:

Eben Upton, on the Raspberry Pi Foundation (pic 2)
Merlin Crossingham, on Wallace & Gromit (pics 5&6)
Tim Minshall, on how engineers make the future (pic 9)

Videos will be on the website at some point for those who want to catchup.

There’s also another TED event taking place in June. TEDx Oxbridge, organised by students from Oxford and Cambridge, will be on the theme of Unconventional Wisdom. Not sure who the speakers are yet.

More likely to die Dancing than Skydiving

Amir Chaudhry — 2012-04-12T10:15:00+00:00

From the infographic below.

“Odds of dying while skydiving in the US: 1 in 101,083”

“Dance parties: 1 in 100,000 chance of dying”

Wonder what the BCPA would make of this, since they recklessly combine both Skydiving and parties on a regular basis (how extreme).

Seriously though, this nicely depicts that many things have worse odds than skydiving (although comparisons like this should be taken with a pinch of salt). However, base jumping (which is not the same thing as skydiving) seems to be the most dangerous by far (1 in 60).

via BMD

Joining the Computer Lab Systems Group

Amir Chaudhry — 2012-04-11T00:00:00+00:00

This was the post I meant to write all the way back in January.

I’ve joined the Systems Research Group in the Cambridge Computer Laboratory. Much of the work here has a strong practical angle and people like to build useful things. Several of the profs here have been involved in creating some prominent companies (one recent example is XenSource, which powers much of the public cloud - bought by Citrix).

My work in the lab is related to Personal Containers, which aims to give users access and control over their lifelong digital footprint. It’s part of a wider research program funded by the Horizon Digital Economy Research Institute, which tackles topics ranging from energy efficiency to “Death 2.0” (that might sound odd but what does happens to your online identity/ies when you die? What should happen? How?).

One of the first things I worked on was a paper with several other folks in the lab, which is currently being presented at a conference. It’s about infrastructure to infer the carbon footprint of employees’ daily commute by analysing high resolution geolocation data from their smartphones. Obviously, this must be done in a privacy-sensitive manner, which is what the paper discusses. Once it’s published in the conference proceedings I’ll add a link to it here.

I’ve also been writing a bit on the Personal Conatainer’s blog about privacy and personal data. The posts there will be reposted here too, over the next few weeks.

Other than that, here are some random pics from the lab. The second pic will make more sense if you’ve seen the xkcd comic it’s based on.

I moved my site to GitHub Pages

Amir Chaudhry — 2012-04-07T00:00:00+00:00

Yet again, it’s been several months since my last post (it’s almost a habit now). This isn’t intended to be another catchup post, though.

The reason for this post is that I’ve moved my site (amirchaudhry.com) to GitHub Pages.

Since I’m using my own domain, this is unlikely to affect anyone but I’ve copied this post into both the new GitHub site and old Posterous site just in case. If you’re reading this via my Posterous site (presumably through the share/follow features there), then it’ll be the last post to go up there.

There are still a bunch of things that are bit broken in the new place.

No tweet/sharing buttons*
Tags don’t sort properly
Search doesn’t work well
RSS/Atom feed is probably a mess
Syntax highlighting isn’t working**

Moving from Posterous to GitHub Pages

The process of moving to GitHub was a useful experience. Here are the steps I ended up taking before I pointed the domain to the new site. Overall, I’m really glad that my posts aren’t tied up in someone else’s platform anymore.

Getting set up with GitHub Pages

I used Jekyll Bootstrap and the default bootstrap theme. It was ridiculously quick and easy to get a basic site up and start playing around. The first thing I did was try to re-create structure of my old site and then worry about how it looked. I’d never dealt with CSS before so this was always going to take a while. For example, it took me ages to figure out why items kept shifting between pages. Turns out I needed the following line in my style file:

html { overflow-y: scroll; }

Although dealing with the CSS took a while, Bootstrap made it really easy to copy/paste/edit the bits I wanted to change. The nav section was a pain to sort out. It took a few weeks to get to this point but you’re looking at a bootstrap site, which looks nothing like bootstrap (win!). It’s likely that I’ll keep tweaking things but I’m not expecting to do anything major.

Migrating the posts

There wasn’t an easy way to migrate directly from Posterous to GitHub. There are scripts available but either through bad-luck or ineptitude, I couldn’t get them to work. A guy called Chad recently created Export My Posts, which pulled out all my content in html format (phew!). I really didn’t want to deal with learning an API or parsing an xml file, so I’m glad he put this together. It works extremely well.

Next step was to run all the html posts through pandoc to convert them to markdown. I wrote a fairly trivial shell script to do this.

#!/bin/bash
echo -e "Files to convert to md? "
read -a allfiles
#args=("$@")
for file in ${allfiles[*]}; do
    printf "   %s\n" $file
    pandoc -o ./convert/$file.md $file
done

Then came the slow process of checking each file, removing the left-over html, adding the appropriate headers, dealing with image galleries etc. Part of this could have been automated but I don’t have that many posts so I went through them manually.

Once all that was done, I simply had to switch the domain over and everything was fine.

Lessons learned

I need to keep a local copy of content. Of course, this is now solved but even before-hand I would write and save things locally before sending them up. The problem was that I’d always have to tweak something or edit a typo and those changes happened on the site (never on my local copy).

Having existing things I can dig into and copy from (i.e. bootstrap) really speeds up my learning. I don’t think I could have gotten this far this quickly without such resources. If you look at the code itself it’s probably a mess, but it’s my mess.

Scripting things will help. Quickly creating image galleries is the next thing I need to figure out. The process is obvious to me now (since I did parts of it by hand) but I have to automate it using ImageMagick.

Overall, I’m quite pleased with myself. I’ve gone from knowing nothing about CSS, static sites, and bootstrap to being vaguely (in)competent. Rock on!

*Though I'll probably use something like AddThis. ... Edit: Ok, so at the time of writing, I didn't have the buttons but by the time of publishing I'd figured it out :)

**Yup, managed to fix that too :)

Long overdue catchup post

Amir Chaudhry — 2012-01-25T00:00:00+00:00

aka ‘highlights of the last few months’

It’s been quite a while since I posted anything and it’s time I did. The last few months have been pretty busy so I thought I’d write some notes about the main items in one post and then we can pretend that I’m current again. I’m well aware that the sheer size of this post means it’s unlikely to be read fully by most people. There are some pics so perhaps that might be enough to hold your interest :)

Later this week, I’ll write a separate post about the things I’m working on now and I’ll try to keep more up to date from then onwards.

All of the following covers October to December.

Silicon Valley comes to the UK

SVc2UK was a 5/6 day conference with events taking place in Cambridge and London. The aim is to introduce students from universities to high-profile folks from the Valley and inspire the students to consider startups as a career path (either by founding or joining them). I was involved in this conference back in 2009 and decided to join the the new team for the 2011 events. This year the conference was preceded by a several ‘Appathons’ (read: hackathons), which took place at 6 Universities, including Cambridge. You can find out more about the Appathon and the conference via svc2uk.com but here are some short notes and some pics.

– Cambridge Appathon –

The hackathons were 2-day events aimed at students and the theme was open government data. I ran the Cambridge event on behalf of Cambridge University Entrepreneurs and we had about 80 people taking part over the course of the weekend. Representatives from Google, Facebook, Apple and the Technology Strategy Board were present to advise the student teams and folks from the local tech community also got involved. At the end of the weekend, we awarded Amazon Kindles to three winning teams (as decided by audience vote) and everyone was encouraged to enter the national competition too. One of the Cambridge teams also won the national prize (a trip to SV). Yeay!

Given the sheer amount of brain-ache involved, I could (should?) write a separate post on how to run a hackathon. Maybe I’ll get around to it at some point but in the meantime, here’s the team that actually pulled this together: Matko, Alistair and Andy, Ben, Ivan, Simona, Saar, Roger. I’m also especially grateful to the sponsors/supporters of the event: Red Gate Software, Cocoa Controls, Nick (at Google), SVc2UK and the Cambridge Computer Laboratory (who hosted us for a weekend). Below are some pictures of the event itself and the national awards ceremony at Downing Street.

– SVc2C Conference –

This was the main part of the conference and the guests from the Valley were in Cambridge for two days. In that time, they took part in a pretty large number of events and hopefully met a lot of interesting students. If you’re interested in the list of guests, you can check them out via the link on the conference website (svc2uk.com).

You can find videos of this year’s sessions on the Vimeo account.

Stanford online courses

I did all three of the inaugural classes. Machine Learning, Artificial Intelligence and Databases … and I enjoyed them in that order. The following is a short summary/review of each.

ML-Class was fantastic since you had to write actual scripts and I really looked forward to both the videos and exercises every week. At times, the exercises felt a bit contrived (one line of code in a file) but there were occasions where I had to really think properly to get that one line. Once you got the hang of matrix multiplication the exercises got easier. The scores don’t really mean anything since you could repeat the questions until you got a perfect score. At some point, I’ll go back over the iTunes U content from the profs 2008 class. It has more theory/maths and covers reinforcement learning, which the applied course didn’t. You can find my ‘programming’ exercises on GitHub.

AI-Class was a very different course and it didn’t seem as ‘slick’ as the ML class but I learnt way more maths, specifically about probability. Since I was terrible at probability at school, I really wanted to get a better handle on it and this class really made me work. I have pages and pages of calculations for some of the in-class questions and I hope I can retain what I’ve learnt. At times there was only a cursory overview of a topic before moving on but I should have expected that from an “Intro to…” course. This was the only class with both graded homeworks and exams. Overall, I was in the ‘top 25%’ according the statement I got (overall mark was 95%).

DB-Class was the one I least understood but the one I originally thought I was going to stick with. Within the first few lectures, I knew I was going to find it tough. Relational Algebra made no sense to me and the videos seemed to cover each example once and move on to something else. Since I didn’t have enough time to find additional resources, I ended up barely following along and basically flunked the course. There was a mid-term and final exam and the homeworks could be repeated until you got a perfect score (but I ended up not bothering). No idea what score I got but you needed >50% to get a statement and I didn’t get one :)

In general, I was very impressed by the dedication and effort the profs and the teams working with them put into the courses. Although the profs were the folks centre stage, there were a group of largely unsung heroes who kept things running, put together exercises and enabled everything to happen. This method of learning is really going to take off and if people can figure out how to deal with proper certification, then it might even challenge the notion of going to University. There have been a slew of new courses available and I think I’ll try two more if I can fit them in (i.e. Probabilistic Graphical Models and Model Thinking).

Learning to code

I’ve also been trying to keep up with learning Python via LPtHW (see GitHub repo for progress). I’d describe myself as chugging along and I’m glad that I’m not finding anything too difficult.

The trickiest part is finding ways to actually fit some coding into things I need done. Since I finished analysing my PhD data, I’ve not had the need to write scripts or manipulate data in any substantive way and I don’t fancy re-writing any of my old stuff (NB I also tried to collate all of that and added it to GitHub too). I’m sure I can find something worth doing and I suspect it might involve Django sooner than I thought.

I also signed up to Code Academy’s ‘Code Year’ but I’ve not opened any of their emails yet (and I suspect I won’t have time).

Events and occasional startup stuff

As always there have been a bunch of events and startup-related items in the last few months. Normally I wouldn’t bother mentioning these but I’m in a list-y mood.

Seedcamp Demo Day (Sep)
CharityHack (Oct - not that I did much)
Silicon Roundabout Social Club (Oct)
Product Camp (Nov)
Connections talks (Nov & Dec - Cyberwarfare and Energy)
Eric Ries - Lean Startup (Jan - UK Book Launch)
HackerNews London Meetups
Visit to Olympic park (ok, so this was waaay back in May and should have had it’s own post but I never got around to it)

Pics of Visit to Olympic Park (in May)

… and finally

I managed to get this done in October. Took a while.

Infographic of the day - What success really looks like

Amir Chaudhry — 2011-09-20T00:00:00+00:00

via Ian Bicking via Kim Sherrell via Jason Keath

Snake oil? Scientific evidence for health supplements

Amir Chaudhry — 2011-07-28T00:00:00+00:00

via informationisbeautiful.net

Great graphic. I’m surprised to see Vitamins A & E so far down the list though.

Comment from original post: Jul 28, 2011 - benmordue said... A possible reason that the "tangible health benefits" of Vitamins A & E is low is that a "healthy diet" (mentioned at the top of the graphic) already includes enough of those nutrients, and consuming extra vitamins above the required level does not confer extra health benefits. Also: hello! How do? ;-)

Comment from original post: Jul 28, 2011 - benmordue said ... Aargh! Meant to say, "of Vitamin A & E *supplements*

My thoughts on the Seedsummit Term Sheet

Amir Chaudhry — 2011-07-11T14:02:00+00:00

A few days ago, Seedsummit released a Term Sheet intended to be (or become) standard across Europe for early stage investment.

It’s been several days since the announcement and so far I haven’t seen much in-depth commentary on it (other than acknowledging its existence and how difficult it must have been to achieve consensus). That might be because the announcement seemed to come as a surprise, so perhaps folks haven’t had a chance to digest it yet. Or maybe nobody cares?

One post from the Guardian offered some criticism, but beyond that it’s all been a bit quiet. Since no-one else has posted any thoughts, I thought I’d go through the document and offer my opinions. I’ve put the notes in a more comprehensive post and summarised the main points here (mostly because going through a legal document is a little bit dreary).

Main comments on Seedsummit term sheet

There are a few of things that founders should be careful of. First is the liquidation preference, where founders should be getting the non-participating option (Option 1). Second is vesting, since founders probably want to adjust the values described. Third is how the option pool is set up (before/after the funding). Finally, the anti-dilution provisions don’t seem to provide any useful information so founders should check this carefully if they’re presented with it. For more detail on any of these points, see the relevant section in the detailed review.

The overall impression I got was that these are documents for dealing with VCs (edit: It seems an additional note was added to the download page to this effect). Unsurprising since it was put together by 21 VC firms from across Europe. Surprising because I’m guessing no Angel investors were involved, despite the term sheet being advertised for ‘early stage’ financing. Since Seedsummit is trying to build a network of european investors (and credit to them for that), I did find it a little odd that there didn’t seem to be any Angels listed as helping to create this document. Perhaps it’s because the Angels aren’t investing on a pan-european basis? I’d love to see a Series A term sheet for comparison and my gut feel is that the Seedsummit document has a lot more in common with Series A than with Angel docs. This does beg the question, what does ‘seed’ actually mean?

What does seed mean?

In 2009, I set up and ran the first iteration of Springboard (a seed accelerator). We were helping early-stage teams to get a product out the door and to start getting feedback from users (customer development). When teams came out of the programme, I considered them ‘seed-stage’, i.e likely to be ready for seed investment, wherever that may come from. Ideally, the team would have a version of the product that was being used by early adopters. The team would be adapting that product based on feedback, although they would likely have little/no revenue. That basically sums up what I picture in my head when I think of ‘seed-stage’.

Over the last week I’ve met several VCs from different firms and asked them directly what seed means to them. For each of them, seed-stage investing involves companies that have a product, customers and revenues. This was pretty much the case for all the VCs I spoke to. I wish I’d probed for more detail as to what kind of revenue/customer numbers they expect but never mind.

In any case, despite using the same language, there’s obviously a gap between the above viewpoints. I’m not suggesting either of them is right or wrong but I think it’s important to point out such differences, especially in the light of the new ‘standard’ term sheet.

What does this term sheet mean for founders?

In brief, founders should do their homework. In going through this document, I spent quite a while researching and there’s actually a lot of information out there about what the terms mean and the impact they might have. As we now have a European VC term sheet in the open it’ll make comparisons with US VCs a lot easier. Any major disparities will quickly be picked on by the popular startups who might go elsewhere to raise money. Indeed, the real test will be whether or not a ‘hot’ new startup uses these terms and as they stand, I doubt that will be the case. Founders will still need lawyers and VCs will still be willing to negotiate terms.

We’re not quite at the point-and-click stage for early stage financing but with more and more information being put into the open, it does become easier for startups to make better informed decisions.

If you haven’t already, you can go check my review of the Seedsummit terms now.

Did you find this useful? Upvote and comment on HN

Photo credit: Affärsbilder by Dan Eriksson on Flickr

Review of the Seedsummit Term Sheet

Amir Chaudhry — 2011-07-11T14:00:00+00:00

Please read the summary post (above) before reading this one.

Summary: Last week, Seedsummit released a Term Sheet intended to be (or become) standard across Europe for early stage investment. To achieve this, 21 European VC firms convened and agreed on a ‘standard’ set of terms. The aim of producing this document is to make “… seed funding easier to access, better to understand, and fair for all parties”. If you want to, you can download the general term sheet for yourself. I’ll go through this document below.

Things to note

Before I get into the document itself, there are a few things I should point out:

The most important is that I Am Not A Lawyer. Anyone dealing with legal docs should obviously get legal advice. However, since this term sheet is meant to be “reader-friendly” (as SeedSummit says), I’d hope that someone like myself should be able to understand and interpret it to some degree. If not, one of the benefits of this document diminishes since I still need a lawyer to translate it for me.
I’m only reviewing the general term sheet, not the EIS specific sheet.
I’ve read typical angel investment docs since I’ve led one angel investment before (for someone else). Hence my point of view is likely to be influenced by that.
I’ve never seen a term sheet from a VC so I can’t make any specific comparisons between the new SeedSummit sheet or a prior VC term sheet (though I’d like to).
Obviously, everything you read is here is just my opinion. Whether you agree or disagree, I just want to see some discussion.

About Seedsummit

I realised I know very little about Seedsummit itself so I looked around their website and also for other links via Google.

In a few words, SeedSummit is Europe’s answer to AngelList. More specifically, it’s SeedCamp’s answer to AngelList (it’s run by the same people rather than independently). It seems there are about 120-ish investors (assuming all profiles are visible) and the investment range is from 10k to 500k EUR (which is the kind of range I consider to be ‘seed’). Randomly clicking through the profiles, I see that the majority of them say “I am investing: My own money”. At first, I took that to mean there were more Angels on the platform but looking closer I noticed that some of those profiles are of Partners in VC funds. Although these folks have ‘skin in the game’ I expect they only invest via their funds so it would have been more appropriate to say “Money from a fund”. In any case, this made it more tricky to guess at the ratio between Angels and VCs on Seedsummit, so I gave up trying. If anyone else knows the answer, please let me know.

Comments on the Seedsummit Term Sheet

I’ll go through this term by term and note my comments so you can follow along with the document. I’ll skip the stuff that doesn’t warrant much discussion (e.g Company, Founders, etc). Since this document was based on the Series Seed documents from the US, I’ll also make a quick comparison between them.

Structure of financing

The pre-money valuation includes the employee share option pool, which effectively dilutes the founders a little more than they would otherwise have expected. There’s a pretty good post by Fred Wilson on valuations and option pools and I’d recommend people read that for more information.

Type of security

This basically says that the new shares have preferential rights (i.e they are not common stock).

Liquidation preferences

There are two options here and both state “… one times the original purchase price …”, in other words it’s the same amount of money the investor put in. Had this been anything other than ‘one’ it would be cause for concern. The main difference between the two options is whether the investor is non-participating or fully-participating which are quite different.

Non-participating means that the investor can do one of two things when there’s a ‘liquidity event’ (e.g acquisition). They can either make sure they get their money back (i.e one times the purchase price), with any left over being divided amongst the other shareholders. Or they can choose to get an amount in proportion to the shares they hold. They will take whichever returns the most money. Obviously, both the founder and investor want the sale to be as large as possible so both will be aiming for the latter case. In the case where the exit isn’t quite what the investor imagined, then they’ll try and get their money back. It seems there is an example of this in practice when Fred wrote about Slide.

Fully-participating means the investor gets to do both of the above. They will get their money out first and then will also take a share of any left over money (sometimes referred to as ‘double-dipping’). This is definitely the less founder-friendly option and I expect founders would not accept it (assuming they have enough bargaining power to push back). It does seem a little odd to me that this is listed as an option in an early stage term sheet, especially since terms usually get carried forward into subsequent funding rounds. For an example of how liquidation preferences can affect the way money gets distributed, you can read VentureBeat on liquidation preferences (scroll down to the example).

Anti-dilution provisions

This term, which is indicated as optional, dictates what happens if a subsequent round of funding occurs at a lower valuation than the current one. Except it doesn’t. Those deceptive square brackets could hold a range of scary sounding words and in some cases you might even find equations in there. Since there’s no suggested text here, I don’t have much to say. Feel free to google and you’ll find some variations on what could end up in here.

Important Decisions

A list of things you can’t do without the investor’s consent. The options here basically whether the list is written in the terms or referred to in an appendix. From the suggested list, there isn’t anything I haven’t seen before.

One thing that might be missing is the proportion of shares that must be maintained by these investors to maintain this right. As it stands, the investor could hold any amount of shares and still maintain the right (e.g they could hold less than 5% and still block a sale of the company even if the other 95+% agreed to it).

Pre-emption, Right of first-refusal and Co-sale

I’ve seen most of this before but there are a couple of things I don’t understand. Firstly, for pre-emption, “The Investors my assign this right to another member of their fund group”. I have no idea what this means since I can’t figure what ‘fund group’ is being referred to. Secondly, for the Right of first-refusal, I’m not sure what “…transfers by Investors to affiliated funds” means either.

Drag Along

The important point here is that both a majority of the Preference shares and of the Ordinary shares is required to enact it. Had it only been a majority of Preference shares then it would be drastically one-sided.

Restrictive Covenants and Founders Undertakings

Other documents come into play here, for example the employment contracts. An interesting point is that the founders shouldn’t work on anything else during their ‘business time’, which mean any side projects might need permission from the investors. It’s reasonable to expect founders to commit themselves to the business but they would need to check the employment contract to make sure there isn’t anything untoward. The last sentence referring to immediate dismissal for cause seems a little harsh.

Founder Shares

Apparently the shares held by founders would be subject to reverse vesting. In effect, whatever shares you thought you ‘owned’, you don’t really have anymore. Reverse vesting means that although you have your shares the company will repurchase them from you (at practically no cost) if you leave or are fired. That ‘right-to-repurchase’ diminishes over time. This doesn’t sound like a good deal for the founders but it’s understandable that the investor wants to keep the founders committed to the company.

The interesting part are the suggested figures. 25% of the founder shares vest after a year and the rest on a monthly basis over another two years. This seems a bit excessive to me. Firstly, I don’t see any reason why more of the shares can’t vest immediately and the rest on a monthly basis thereafter. Secondly, I think a three year time frame is quite long and although I’ve seen it for employee options it doesn’t seem appropriate for founders at an early stage. (edit to add: I’ve heard from friends in San Francisco that four year vesting with a one year cliff is pretty standard out there)

For the kinds of startups I’ve dealt with, I’m generally of the opinion that if a founder leaves, then the company is effectively over. Provisions for clawing back shares, like the one one here, seem largely pointless to me. In any case, I think founders will have to look carefully at what the vesting looks like and see if it works for them

Expenses

Option 1 just plain sucks. I find it slightly ridiculous that the investor expects the company to pay for their legal fees and the amount isn’t even capped. Option 2 is slightly better in that everyone pays their own fees but it also includes a caveat that if the founders decide to withdraw, then they’ll have to pay the investor’s legal fees to date. That would seem fair enough but notice there’s no caveat for the case where the investor decides to change their mind. I’d certainly push for that to be added too.

Brief comparison with Series Seed

The Seedsummit term sheet is itself based on the Series Seed documents so I thought it would make sense to do a quick comparison between them.

There are a handful of things I noticed which were:

Series seed only mentions a non-participating liquidation preference (Seedsummit includes an option for fully participating)
In the Series Seed document, the investor’s legal expenses are paid for by the company but they’re capped ($10,000).
Series Seed’s founder vesting is over 4 years (but gives no suggestions).
No anti-dilution provisions in Series Seed term sheet

If you got this far then congratulations on your patience!

You can either go back to the summary post or upvote and comment on HN

Tech Company Org Charts - and other cool comics

Amir Chaudhry — 2011-07-01T00:00:00+00:00

via bonkersworld.net

Just found this and I think it’s great. There are some other fun comics on the site too. Go check them out.

Did Google+ Hangout just kill Skype?

Amir Chaudhry — 2011-06-30T00:00:00+00:00

Disclaimer: I still haven’t tried Hangout, not enough people in my circles yet.

Skype has been great for conference calls and video calls but it’s nearly always been a hassle with more than 2 people. It’s analogous to the way in which phones work i.e great when only two people need to talk but confusing to set up if there are more.

Although it’s different, Hangout might obviate the need to use Skype video again. Having multiple people, with cameras, all in one ‘room’ and effectively with one-click sounds fantastic. Of course, I haven’t tried it yet but I can imagine not needing to use Skype video again.

Skype will still have it’s place (for now) since I find it useful to see who’s online and ask quick questions via IM. However, even that benefit might disappear in time.

I’m curious to know what other people think.

Edit: Some discussion now on Hacker News.

Solar 3d Printer. Fantastic video

Amir Chaudhry — 2011-06-26T00:00:00+00:00

Solar 3d Printer video

via vimeo.com

Wow. I’d love to know how strong those pieces actually are.

On my facebook wall tonight

Amir Chaudhry — 2011-06-09T00:00:00+00:00

Ok, so I’ve been busy for the last couple of days. Last night was the CUE Grand Finale and tonight was the CUTEC conference and dinner. Both were fabulous and I guess I should write about them soon (It’s been a while since I actually posted new stuff here).

Let the damage limitation begin :-P

Lego is Amazing

Amir Chaudhry — 2011-05-06T00:00:00+00:00

I just spent the afternoon ~~playing~~ doing valuable research with Lego. The pics above show my amazing creation but they don’t do justice to the complexity of it. I really didn’t think I’d need the instructions but I was so wrong.

This was actually part of a research study that will be done with 4-5 year old children. Essentially, the kids will be asked to follow instructions to build objects and the researchers are interested in the self-regulatory processes during constructive play … woosh … Me play with Lego. Me happy.

Apparently, the bus above is considered too complex, which doesn’t surprise me since it took me a few hours to put it together. Most of that time was spent rummaging around in the box for the right parts (that’s part of the study design). I didn’t help myself by deciding part way through that I wanted it to be a right-hand drive bus instead of the left-hand drive that the instructions described (you might be able to spot the difference between pics 2 and 3). That was fun.

Lego itself is pretty interesting and there are some really strict tolerances required in its manufacture. If you’ve ever played with Lego, consider the following: the pieces are held together only by friction, yet you can create and dismantle fairly large and robust objects; if you build a wall of Lego, the plane of that wall is pretty smooth (no protruding bricks). If you’re manufacturing millions of these little things, you can only achieve that with some damn precise engineering. Apparently the moulds are made within a tolerance of 2 micrometers. For comparison, the width of a human hair ranges from 17-180 micrometers. It’s even more impressive when you consider that bricks from 1958 are still compatible with bricks made today.

Check out the Lego Wikipedia article for more info.

Just found MovieBarCode. Fantastic images by stretching and lining up movie frames

Amir Chaudhry — 2011-03-09T00:00:00+00:00

via moviebarcode.tumblr.com

The image above is from Requiem for a Dream. Apparently some of them are available as prints.

Also a news story at Wired

Life's Too Short for the Wrong Job

Amir Chaudhry — 2011-03-03T00:00:00+00:00

Fantastic set of pics for a German ad campaign. Make sure to check out the full set. (via: honeybeeinthecity.blogspot.com)

via honeybeeinthecity.blogspot.com

Richard Stallman on Free Software and your Freedom (lecture)

Amir Chaudhry — 2011-03-01T00:00:00+00:00

Lecture at Cambridge Computer Lab on 1 March 2011 - Sponsored by Software East (@markdalgarno)

[pls excuse the typos & poor grammar. I’m typing this on the fly and unlikely to edit it before posting. Caveat emptor]

[edit: It was a long talk and I missed some chunks. Overall, he spoke for just over 90mins. Longer than I was expecting]

Bio: Richard Stallman launched the development of the GNU operating system (see www.gnu.org) in 1984. GNU is free software: everyone has the freedom to copy it and redistribute it, as well as to make changes either large or small. The GNU/Linux system, basically the GNU operating system with Linux added, is used on tens of millions of computers today. Stallman has received the ACM Grace Hopper Award, a MacArthur Foundation fellowship, the Electrical Frontier Foundation’s Pioneer Award, and the Takeda Award for Social/Economic Betterment, as well as several honorary doctorates.
[taken from: http://www.theiet.org/local/emea/europe/switzerland/stallman-2010.cfm see his website at stallman.org]

Stallman enters room and hush descends immediately. Cue joke about slides and “definitely not PowerPoint”.

What is free software? It sw that respects your freedom. Think of free speech not free beer. A non free sw is digital colonization. It acts to divide and conquer people

A program is free if you have 4 freedoms
0 - To run it as you wish
1 - Have source to edit to run as you wish
2 - To share & distribute & help others, as you wish
3 - To distribute modified, copies if you wish

If sw does not obey these then it’s unethical. Notice that these are not about the technical issues. The difference is an ethical, social and political distinction. The use of propriety sw is a social problem and we should aim to eliminate the problem. ie eliminate propriety sw. Goal of free sw is to make all sw free so that users are free. If you use a program without freedom 2 then you end up in a moral dilemma. If someone says “Hey, great sw. Can I get a copy?”. In this case you should choose lesser of two evils, which is to violate the license and give your friend a copy. Being the lesser evil doesn’t mean it’s good though. When you’ve fully studied this dilemma what should you do? Option 1: don’t have any friends! That what the proprietary debs would have you do. Option 2: don’t have that sw in the first place so you don’t end up in the dilemma in the first place. Note on Piracy: They attack ships. With arms. Freedom 0 - Essential so you can control your computing. No one else should have that control. Only you. There are proprietary programs that have licenses that restrict your freedom of speech (ie website software that means you cannot publish stuff disparaging to sw vendor)

Freedom 1 - Being a victim of malicious sw is something that ppl are subject to all the time. For example back-doors, DRM, spyware, handcuffs. The sw is not there to serve you but is actually a prison guard. Eg in MSFT sw can install stuff on pc w/out user consent or knowledge. Once windows has installed then it’s no longer your machine. Apple is no better. They’ve even stolen the ability to install apps. They don’t even agree w free sw. Also a back-door by which Apple can delete sw installed on users machine. Even apps exist that detect Jailbreak and refuse to operate. All these are Malware. Products include: Apple iMoan, Apple iBad, Amazon Swindle - “Kindle was conceived to burn books.” (riffs on Amazon taking books off devices etc)

Even if devs of proprietary sw are good, they’re human and make errors. ie bugs. Without freedom 1 you are a prisoner. Freedom 1 isn’t enough. There too much sw and plenty of non-devs cannot edit it. That’s why we need Freedom 3. To contribute to your community and spread benefits. Without this freedom what a waste it would be for people to write changes over and over. Every user can take part in Freedom 0 and 2 since it doesn’t require any programming knowledge. Freedom 1 and 3 allow those who can to make improvements. With free software. Support is a free-market since everyone can study the code and master it. Hence people can get better support. Since support for proprietary sw is a monopoly, the support sucks.

The four freedoms together give us democracy. These are sufficient for people to have control over the sw. Without these the sw controls the ppl. Launched free software foundation in 1983. All operating systems were proprietary hence ppl immediately lost freedom when they had to install an OS. This was an injustice. As an OS developer, he could write the OS and then legally make it free. At the time, the free sw movement had no enemies at the time. Mostly because ppl thought the job was so big that it couldn’t get done. Following the basic design of UNIX made sense. Then I gave it a name which was a joke. [digs in bag]. GNU = Gnu Not Unix. (riffs on the name and pronunciation). History of Linux. Aside: What is a free sw license? Why do we need them?
Copyright law automatically limits anything that’s written so you have to explicitly grant the four freedoms. Even then there are distinctions. Eg copy-left licenses and non copy-left licenses. Specifically, user has to pass on the freedoms that have been passed on to him (copy-left). Torvalds made the Linux Kernel free under the GNU license.

The name matters. Linux does not have the same views on ‘free’ as GNU. Yet people using Linux think that most of the ideas came from Torvalds rather than GNU. Debate on Human Rights has continued for centuries. Yet the debate on free/proprietary sw has only gone on for about 20yrs. Therefore the debate on what human rights are applicable to sw have only involved the devs, who obviously have said “none”. Another problem. Now ppl have another term: Open-Source. The ideas that surround this didn’t include the ethical aspects of ‘free’. Easy to lose freedom if you don’t fight for it. There are now distros that contain proprietary sw and are no longer free. Free distros can be found at gnu.org/distros. Why did all this happen?
Because the idea of freedom wasn’t important to people. They didn’t appreciate it. We have to teach ppl to value freedom and demand it. Our greatest scarcity in FSF is not ppl to write sw but ppl to fight for the idea of freedoms. Freedom is now different since JavaScript in browsers means that things get installed in browser without your knowledge. ‘Like’ buttons even track ppl that don’t use Facebook. Ppl need to be educated. Even SaaS is becoming pervasive and is worse than proprietary since the data goes elsewhere. There’s no remedy for this, other than not to use it. Only a minority of things like this but need to be aware (eg GoogleDocs). Free software and Education
Schools must only teach free software. It’s cheaper (but should be considered secondary benefit). Free stuff from proprietary vendors is simply a drug to get them hooked. Only free sw gives ppl the opportunity to learn from code of large Programs. Even deeper reason though. For goodwill and the moral imperative of sharing knowledge. Every school from nursery to university must practice this. The issue should always be an ethical one. Biggest obstacle to free sw is social inertia. Websites for more info:
Gnu.org
FSF.org
Defectivebydesign.org (against digital handcuffs)

[dresses up] St Ignuseus of emacs [riffs on emacs and free sw].

Applause
[ends]

Sent via mobile

Comment on original post: Nov 21, 2011 - human said... thank u so much, This is the same lecture of richard in benghazi-libya at november 2010 :) i hope to get this lecture in a audio file

Session 4: Using audience insight to drive product success (BBC Case Study) #pcampldn

Amir Chaudhry — 2011-02-26T14:32:41+00:00

Nic Newman

Story of how they got it wrong and then put it right.

History: 2yrs ago, Weather site was pretty dated and had grand ambitions to update site.

Editorially led site
No product manager
No audience research plan
Drew wrong conclusions from research they did do

[video]

Product took over a year to get out. BBC recorded thousands of complaints. Within 2-3 months it was re-launched, solving most of the problems. Site retested in beta with people who complained the last time around.

In original version did lots of focus groups and used this to decide what to do. eg ppl said they wanted everything on one page, so they did (without checking that it was right). Second thing was beta test was on folks that were weather nuts so they wanted all the detail. Public wanted simplicity.

For version 2, checked people’s complaints and coded them so that there was a top ten list of things people didn’t like. Then went back out to focus groups. Found people who were really angry and tried to work with these people to improve. Pressure from above (director general et al) meant quick wins were important.

Beta test 2, tested on 1300 people and survey on their views in comparison with old weather site. Some quantitative data made a big difference when talking to DG about whether new site is going to work.

Key learnings:

Have audience insight plan a
Qualitative followed by quantitative test is key
Make it iterative … Audience research, usability and dev going concurrently
Regular testing to ensure that NetPromoter detractors and promoters both accept changes
Audience insights to inform communication plans of changes

Q. What kind of quantitative measures?
A. Surveys and click-tracking. Eg of 1300 people, how many get to the forecast page and how many personalize the homepage.

Q. Is it important to build the tracking into the homepage?
A. Yes, important not only in final site but also in the beta sites etc.

Q. Is there danger that some of the ppl who complain the loudest are not representative?
A. Absolutely. Danger is that others in company look at emails, comments, blogs and assume it’s representative. Good to have data to show that they’re not.

Session 3: What's your Product Mgmnt Persona? #pcampldn

Amir Chaudhry — 2011-02-26T12:30:55+00:00

[Missed session 2 due to Apple Mail + Wireless fail.]

Ian Lunn from Product Focus

Only handful of ppl put hand up to Q of “who thinks they’re a good PM” :)

Product Focus do survey on ProdMgmnt.

Ave PM has been a PM for 8.5yrs (up from 7.5yrs last yr)
44% of time spent on firefighting and 56% on planned activities “too much time on tactical”
lack of clarity on role/responsibility
51% of projects delivered on time (some ppl in room think that’s high)

Effective ProdMgmnt. Now we have a 2x2 matrix [notepad ain’t gonna help me w that]. Axes are knowledge/skills and actions.

8 Personas

Numbers Guy
Subject Matter Expert
Expert Compromiser
Troubleshooter: Wild West Style
Product Champion/Advocate
Voice of Market
Reliable Project Manager
Multi-CEO (bit of everything)

Biggest issue in ProdMgmnt in companies seems to be:

Lack of clarity on roles
lack of time for strategy
Ave salary is £61k base (cue murmurs from room…)

Job titles can be misleading. Can split work by activities.

Strategic PM Activities: Working out the right product
Inbound Activities: Helping business deliver the product
Outbound Activities: Helping business to sell the product

Q. Has the role of PM changed over the years as it becomes more recognised?
A. Still a way to go. Better recognised but need more exposure.

Session 1: Data Driven Decisions #pcampldn

Amir Chaudhry — 2011-02-26T11:44:00+00:00

Talk by Lucy from LoveFilm (@lucyjspence) Slides at http://slidesha.re/hFq1OB

Measure change through Experiments.

About half the room has done some form of A/B testing before. Different changes, pages etc etc. Next thing to try is multivariate testing (MVT).

Radical A/B testing. Taken to an extreme you can have different journeys (separate website, even?).

Statistical significance. Important to take note of since you need a certain volume of traffic before you can declare something as valid or not.

Relation to product lifecycle. Lifecycle affects the kinds of tests you can do. Early on, can only do simple tests since you don’t have volume to do valid MVT. As volume grows, then You can do more stuff and segment people. Eg separate ppl who come in via email campaign from other sources.

Advice:

Respond to results
Design for experimentation
Design to segment/identify people by preference behaviour.
Get the right tools for your org.
Get ppl involved and pulling same direction
Be prepared for things to go in the wrong direction from time to time. Don’t get dispirited.

Q&A:

Q Getting the right tools. Any recommendations?
A Google is a good start. website optimizer. Can let marketing ppl see what’s going on. It’s a monthly thing

Q. A/B testing is great for testing ideas you’ve already got but doesn’t help come up with new ideas. Does MVT etc help org come up with new ideas?
A. In some situations saying you can test things helps reduce risk. Depends on risk appetite of ppl too. It’s just a tool so it just helps provide more info on what’s happening.

[Ok, I need to work on my note-taking skillz. Missed a lot while trying to type.]

Sent via mobile

Comment on original post: Feb 26, 2011 - Cathy said... Great thank you for the notes - I was gutted that I missed her presentation... Sounds like it was a great one!

I'm at product camp London

Amir Chaudhry — 2011-02-26T11:14:43+00:00

Will try and write notes and post them up as I go. This could go horribly wrong. Whatever.

Sent via mobile

Moved desks and discovered spent munitions as well as a weapons stockpile

Amir Chaudhry — 2010-12-30T00:00:00+00:00

I just moved desks from one side of the room to the other. While messing with the cabling I also cleared out a bunch of nerf bullets from under the desk.

As if that’s not ominous enough, behind me is what appears to be the weapons stockpile (see pic above).

I’d like to think that the New Year will bring peace, love and harmony to this part of the building but that would be crazy talk.

I’m going to be Switzerland (or maybe the Red Cross).

Random things I learned at school

Amir Chaudhry — 2010-12-03T00:00:00+00:00

One of the useful but slightly odd things that stuck with me from school came from one of the teachers I really didn’t like. I never thought that particular class would have any relevance to the Rest Of My Life but here I am thinking about it.

That one thing was a phrase that the teacher kept repeating, week after week. Anyone from my school will know exactly who I’m talking about.

“Make a statement, back it up with a quotation, then explain fully.”

Ok, so the ‘explain fully’ part wasn’t entirely explained (the irony), but I’ve found the general principle to be helpful when I’m trying to get my thoughts on paper.

Make your point. Provide some evidence. Explain it clearly.

Why Obama got the Peace Prize

Amir Chaudhry — 2010-11-25T00:00:00+00:00

via imeveryone.com <p class="footnote">The ‘imeveryone’ link is now dead (April 2012). Updated the image link though.</p>

How to discourage people from messing with your [insert object here]

Amir Chaudhry — 2010-10-29T00:00:00+00:00

This is on @nick_wood’s desk. Yes, that’s real hair. Yes, it’s Nick’s hair. So far you can see that it works on iPads and cakes.

Not sure how much my therapy* will cost but it’s going on expenses :-P

*that's therapy and a dustbuster

Update: Turns out that it doesn’t work as well on cakes as I previously thought. I’d take another photo, but the cake is gone…

Collective Intelligence in Human Groups

Amir Chaudhry — 2010-10-04T00:00:00+00:00

Evidence for a Collective Intelligence Factor in the Performance of Human Groups

Anita Williams Woolley,^1,* Christopher F. Chabris,^2,3 Alexander Pentland,^3,4 Nada Hashmi,^3,5 Thomas W. Malone^3,5

Psychologists have repeatedly shown that a single statistical factor—often called “general intelligence”—emerges from the correlations among people’s performance on a wide variety of cognitive tasks. But no one has systematically examined whether a similar kind of “collective intelligence” exists for groups of people. In two studies with 699 individuals, working in groups of two to five, we find converging evidence of a general collective intelligence factor that explains a group’s performance on a wide variety of tasks. This “c factor” is not strongly correlated with the average or maximum individual intelligence of group members but is correlated with the average social sensitivity of group members, the equality in distribution of conversational turn-taking, and the proportion of females in the group.

Carnegie Mellon University, Tepper School of Business, Pittsburgh, PA 15213, USA.

Union College, Schenectady, NY 12308, USA.

Massachusetts Institute of Technology (MIT) Center for Collective Intelligence, Cambridge, MA 02142, USA.

MIT Media Lab, Cambridge, MA 02139, USA.

MIT Sloan School of Management, Cambridge, MA 02142, USA. </small>

via sciencemag.org

This sounds like great work and it’s one of the things I wanted to study just before I started my PhD. Interesting to see that performance correlates with social sensitivity as well as the proportion of females in the group. I’d be very interested to know exactly what kinds of tasks the teams were asked to perform.

I’ll have to wait till I’m next on campus before I can download the paper itself.

Discussion thread on HN

Fantastic Eyjafjallajökull time lapse movie

Amir Chaudhry — 2010-09-30T23:12:00+00:00

Fantastic Eyjafjallajökull time lapse movie

via vimeo.com

Another great time lapse movie. This volcano nearly scuppered my travel plans earlier this year but somehow I completely missed this footage.

Great work.

(hat tip: Kim R)

The future is now

Amir Chaudhry — 2010-09-30T17:22:00+00:00

via geekandpoke.typepad.com

Also applies to DMs getting screwed up.

PhD Vivas: The Dutch do it better

Amir Chaudhry — 2010-09-30T10:48:00+00:00

I was at a good friend’s PhD viva a couple of weeks ago and I was impressed with the ‘pomp and circumstance’ surrounding it. Basically, the Dutch do PhD vivas/ceremonies way better than the Brits, especially given how much blood, sweat and tears are involved in getting to that stage.

There were lots of photos that day but I’ve managed to capture most of the story in the selection of pictures above. There are lots of differences between what I saw and the UK system.

For example, it’s obvious that the event is a public affair although seating is strictly limited so it’s usually just friends and relatives of the graduand (he’s the tall guy in the middle of the first pic). There is a committee of examiners who can question the candidate, which includes his supervisor and the period of questioning lasts no more than 45 minutes. That’s it. At the end of the 45th minute, the master of ceremonies (can’t remember his official title) will stop whoever’s speaking and usher the committee out to deliberate. By contrast, in the UK, the event would be a private meeting with only two examiners (supervisor not present), lasting anywhere between 1.5 to 3 hours.

Having said all that, the comparison isn’t really fair. In the Dutch system, the viva is much more of a ceremony than an examination. The Thesis will already have been read and approved long before the public defence takes place. That’s including any corrections that have been requested. Therefore, barring allegations of plagiarism, the outcome of the viva is already known to everyone which lends a much more ceremonial air to the proceedings. That doesn’t make the questions any less tricky though. Once the committee returns from it’s recess, there are some proclamations and the degree itself is awarded (it’s in the blue tube). After that, there’s a drinks reception and few family/friends head off for dinner. Overall, a very civilised affair with the attention squarely on the graduate throughout.

Frankly, I thought the Dutch method of graduating was way better than the UK equivalent (the expense notwithstanding). After spending several years working on research followed by the effort of writing it up, I’ve always felt the UK ceremony was a bit of a let down. Over here, families sit in a large room full of strangers, and watch a procession of other strangers spend 30-45 seconds in front of a robed figure before picking up a sheet of paper on their way out of the back door. There’s a brief flurry of excitement as you see your (insert_relation_here) have their few seconds of attention followed by the realisation that you still have to sit through the remainder of the class before you can leave. Not much of a celebration of the achievement in my opinion (not that I should disparage it too much since I have yet to finish myself!)

Given the fundamental differences in how PhDs are examined between the UK and the Netherlands, I’m not sure what we could do to introduce more of an ‘individual ceremony’, though it would certainly be nice to try. If I ever get there, I’m making sure I mark the occasion properly … which reminds me, back to work. :)

Comment on original post: Apr 30, 2011 - Gill James said... You can have your Ph D superviser in the room at UK universities. I opted not to, but in retrospect wish I had had him there. Just back form the Netherlands and watcehd a Ph D defense. Fascinating!

This time-lapse montage will blow your mind

Amir Chaudhry — 2010-09-29T00:00:00+00:00

Time lapse montage

What? Were you trying to use your brain for something right this moment? Please stop that. Stop your multitasking, your conference calling, and your SMS messaging for 2 minutes and 24 seconds, just long enough to enjoy this fantastic time-lapse montage from Mike Flores. It’s worth it.

The video was shot with a Canon 5D MKII and various lenses, including the Canon 16-35mm f2.8 II, Canon 14mm f2.8 II, and Canon 24mm f1.4 II.

via geek.com

It’s the audio that makes this work so well. The scenes with people in aren’t that spectacular so they’re a little distracting, when compared to the epic nature scenes.

When girls say I love you...

Amir Chaudhry — 2010-09-27T00:00:00+00:00

When girls say stuff like “I love you, but I’m not IN love with you,” I just wanna poke her in the eye.

Girl: OW! You poked my eye!

Boy: No, I poked you IN the eye. And according to you, there’s a HUGE difference.

(image via The Artwork of Siobhan Alcaide)

Woah! Drinking tea prevents testicle shrinkage. Are you getting enough?

Amir Chaudhry — 2010-09-24T00:00:00+00:00

Wow. Random fact of the day.

Apparently the recommended daily amount of Manganese is around few milligrams per day (some sites say up to a max of 11mg per day). The average cup of black tea has 0.2-0.8 mg of Manganese. Other sources of the mineral include soy, rice and beans and coffee (though I couldn’t find how much there was in coffee)

I guess I'm going to have to play Starcraft 2

Amir Chaudhry — 2010-09-23T00:00:00+00:00

Had a slightly bizarre experience at lunch today. Pretty much everyone I sat with was talking about games they’d played in Starcraft II (see below). Words like fast expand, Cruisers and Zerg were being bandied about and for several minutes it felt like a different language (the terminology gets even more odd with things like cheese tactics).

I mean, I speak English so I knew all the words but the order they were strung together in was alien to me. Direct peer pressure rarely works on me (and they’ve already tried) but apparently being part of a conversation where I’m failing to understand anything is like crack. I have to go off, google stuff and try and get a basic understanding. In this case, if I want to continue having lunch with these guys I’ll probably have to start playing. Given my tendency for getting addicted to (good) computer games, this could end badly for everyone.

Luckily, (this time) the conversation did move on to other topics, including male nurses and how PAs could take over the world. Epic stuff.

Amir

folks were: @plc, @leemallabone, @rahulvohra, @matclayton, @robbiethng

What the Facebook Phone would look like

Amir Chaudhry — 2010-09-21T00:00:00+00:00

via esarcasm.com

“Not surprisingly, the phone is designed for a world in which “sharing is the social norm.” Not only will you be able to scroll through your friends list and call people by tapping a button, Facebook will also record who you called, what you said, and what you were wearing at the time. This information will only be shared with law enforcement agencies and carefully selected advertisers”

Edit to add:

Also found the following more recently (Oct 2010)

via seanpercival.com

Cambridge ousts Harvard as world's best university

Amir Chaudhry — 2010-09-08T00:00:00+00:00

via guardian.co.uk

“Cambridge was voted best for research quality in a survey of 15,000 academics. It has an outstanding pedigree: famous minds who pushed back the frontiers of knowledge there include Newton, Darwin and Wittgenstein. Cambridge took overall first place in the rankings, which also use citation counts from a database of academic publishing”

The pic is of the first court in Emmanuel College, as viewed from the entrance

Cooking recipes as Gantt charts. Genius!

Amir Chaudhry — 2010-09-02T00:00:00+00:00

via kangasbros.fi

Fantastic idea! I used to try doing this with recipes to make cooking easier but it always felt like too much effort.

If there was a way to add your own recipes and have them presented back in this format, that would be really cool.

Paul Graham joins Twitter

Amir Chaudhry — 2010-08-28T00:00:00+00:00

He got the name @paulg and @avibryant sorted it out for him. Seems like the update hasn’t filtered everywhere yet since I found the following at http://twittercounter.com/paulg?manual_update=1

Looks like the previous owner was a Paul Gardi. I wonder if he has any idea how many followers his old handle now has (or will have soon).

Base Jumping meets Free Diving (kind of)

Amir Chaudhry — 2010-08-27T00:00:00+00:00

Free-diving video

via youtube.com

Awesome video. The summary says it’s an artistic project and I think it came out really well.

Almost makes me want to dive. Almost, but not quite. :)

(hat-tip: Martina - she keeps sending me cool stuff)

Gary Arndt: Things I've Learned From Traveling Around the World for Three Years

Amir Chaudhry — 2010-08-24T14:37:59+00:00

via huffingtonpost.com

Fantastic write up of a three year journey around the world. I’ve heard of this guy before but never took the time to look him up.

Click on the pic above to go to the main story.

Maz Jobrani: Did you hear the one about the Iranian-American?

Amir Chaudhry — 2010-08-24T12:28:40+00:00

Maz Jobrani

via youtube.com

So funny. This was at TEDxGlobal in Oxford.

A Periodic Table of Visualisation Methods

Amir Chaudhry — 2010-08-13T00:00:00+00:00

Upvote on HN (for those who care).

Found this great table of visualisations. Below is an image but you have to visit the page at visual-literacy.org and hover over the ‘elements’ to see the examples. Neat.

Awesome present for a cool nephew

Amir Chaudhry — 2010-08-03T00:00:00+00:00

How cool is that! Came out even better than I’d hoped. :-D

Easily Aggregate Outlook, Google & iCal calendar feeds

Amir Chaudhry — 2010-07-13T00:00:00+00:00

Summary

For those who don’t want to read the rest, here’s what you do.

Visit the free service MashiCal & create a calendar
Add each of your iCal feeds to your MashiCal calendar (adjust settings as appropriate)
Copy the subscribe link (right-click and ‘copy link’ on the aggregated calendar)

You now have a link to an aggregated iCal feed of your calendar!\ Below is the rest of the post for those with more patience ;)

Why would anyone need to do this?

If you have multiple calendars, sharing free/busy information becomes painful. One result is that too much time can be spent scheduling meetings.*

If I could combine all the calendar feeds into one free/busy feed then I’d only have to share that one calendar with colleagues. So far none of the applications I use are able to do this intelligently, and that’s without the additional complexity of having multiple calendars in each application.

Enter MashiCal.

Using MashiCal to merge .ics feeds

I was already using BusyCal from BusyMac so my Apple iCal calendars sync with Google quite nicely. Therefore, I only need to worry about getting my Google and Outlook calendars into one feed. (NB It’s possible to publish a feed directly from Apple iCal so you don’t need to buy any software)

Getting a feed from Google

To get your ics feed from your google calendar, visit the calendar’s settings page. Make sure the calendar is shared otherwise the feed will not work.

Getting a feed from Outlook

In Outlook 2007 you’ll need to right-click on the calendar you want to create a feed for and “Publish to Internet”. When you’re prompted to share the feed you just created you’ll be able to see the URL you need.

Aggregating the feeds

Now that you have the iCal feeds you want to aggregate, visit MashiCal and put them together. The interface is quite simple and MashiCal will produce an Agenda view and provide a link at the top of the calendar that you can share.

Be aware of the calendar settings for each feed you are entering since timezones and refresh schedules may be relevant. Once it’s done you’ll have a aggregated iCal feed that you can share with others. You can see what my free/busy feed looks like here. I’ve also collated my calendars into a general free/busy page too.

I hope this was useful. I wrote this since it took me a while to discover MashiCal and thought it might be useful for others too.

*I’m aware there are services that try and solve this problem (e.g MeetWithMe and Tungle.Me come to mind). However, I’m more interested in having an aggregated free/busy feed which people can subscribe to.

Comment on original post: Apr 17, 2011 - Zack said... This is cool. The scenario that really needs to be solved is... 2 working parents needing to view each others work schedules as well as personal appointments, maybe personal hotmail or gmail calendars at home and each has an outlook calendar on different work domains. And maybe throw in a teenager's schedule into the mix. Some details you want all to see and some you want to keep private but busy. Now take these calendars and figure out how to view them on each parent's phone. I've been playing with outlook connector which is great when you're at a pc, but the phone experience is missing and you really need that when you're on the road. Has anyone figured out the aggregate when multiple people are involved, and how to do it cleanly on phones?

CfEL Ignite programme - Celebration Dinner

Amir Chaudhry — 2010-07-03T00:00:00+00:00

Photos from the final dinner of the CfEL Ignite programme. At Corpus Christi College.

My first 3-way

Amir Chaudhry — 2010-07-01T00:00:00+00:00

BCPA Monthly Draw – June

I’m the guy in the white helmet.

lipsum

The above was my first attempt at a proper 3-way skydive. If you look closely it becomes obvious that I’m being dragged around. I’ve clearly got some work to do before I can keep up with these guys by myself. We did get 7 or 8 points though!

About the video

The BCPA is the national student body that promotes skydiving. Every month there’s a draw of three formations. The idea is to complete the sequence of formations (and repeat) as many times as possible within the time allowed (35sec).

June was the last draw of this academic year and the formations in the video are Sputnik, Opal and Cluster. The divepool these formations came from can be found here

Learning Markdown

Amir Chaudhry — 2010-06-30T22:57:00+00:00

My first attempt at markdown

Having come across markdown some time ago, I’ve decided to give it a try. This post is basically my attempt at using as many of the tags as possible. At least those I’m likely to need.

Reasons

More control over how how posts look
To be able to emphasise things in plain text
Because I prefer using plain text email

As Gruber says:

Markdown’s syntax is intended for one purpose: to be used as a format for writing for the web

I’m hoping I can get used to it. The last thing I need is to give myself another roadblock to writing.

edit: I didn’t get the list stuff correct the first time around

This is my posterous

Amir Chaudhry — 2010-06-30T22:15:00+00:00

I’ve finally set up a blog after procrastinating for several months.

Still not sure what I’ll be doing with it but since it seems to run via email I expect I’ll actually use it a lot more than other services.

Next post will be a link to my 3-way skydive video. :)

Edit: This used to be Posterous, until I moved to Jekyll and GitHub Pages in April 2012