The Daily ACK

Getting Started with Arduino and iOS

2012-05-15T18:39:00.000+01:00

To celebrate the release of my new video course Getting Started with Arduino and iOS, O'Reilly Media are offering 60% off when you buy it along with my book iOS Sensor Apps with Arduino.

60% off when you buy both the video and book together

It's the MAKE: Hardware Innovation Workshop this week, as well as Maker Faire Bay Area this weekend, and I'll be hanging around both talking about this and other things and well as doing some live demos. So come and talk to me if you see me, and you want to know more about connecting your iPhone to the open hardware world.

Location Enabled Sensors at Øredev

2012-05-15T15:39:00.000+01:00

Back in November last year I spoke at Øredev in Malmö, Sweden, about location enabled sensors, and the video of the talk has just been put up onto the web by the organisers.

Øredev - Location Enabled Sensors for iOS from Øredev 2011.

If you're interested in the topics discussed at Øredev, all the talks will eventually make it onto the website including another talk I did at the conference on visualisation which hasn't made it online quite yet.

Interviewed at O'Reilly Where 2012

2012-05-02T09:53:00.000+01:00

I forgot to post a pointer to this at the time, but while I was out at O'Reilly's Where conference I was interviewed by Mike Hendrickson about location privacy, data leakage and data exhaust.

A fireside chat with Dale Dougherty

2012-04-25T11:15:00.000+01:00

While I was out at the O'Reilly offices in Sebastopol earlier in the month I sat down with Dale Dougherty to talk about how to make iPhones and iPads talk to the open source world.

iOS5 Week at O'Reilly

2012-03-31T09:11:00.000+01:00

It's #iOS5 week at O'Reilly Media, and they're offering 50% off most of their iOS titles in ebook, including my own Learning iOS Programming (2nd Edition) and Basic Sensors in iOS.

If you've been thinking about picking up some #iOS5 goodness from O'Reilly, then now is probably as good a time as any...

(More) Migratory Data

2012-03-22T09:37:00.001Z

This article was originally published on the O'Reilly Radar.

Big data isn't just about multi-terabyte datasets hidden inside eventually-concurrent distributed databases in the cloud, or enterprise-scale data warehousing, or even the emerging market in data. It's also about the hidden data you carry with you all the time; the slowly growing datasets on your movements, contacts and social interactions.

Until recently, most people's understanding of what can actually be done with the data collected about us by our own cell phones was theoretical. There were few real-world examples. But over the last couple of years, this has changed dramatically. Courting hubris perhaps, but I must admit it's possible some of that was my fault, though I haven't been alone.

Interview with Mac Slocum at O'Reilly Strata.

The data you carry with you

You probably think you know how much data you carry around with you on your cell phone. You'll certainly be aware of it if you've ever lost your phone, or had it stolen, or it's just plain stopped working. But there is a large amount of data in the background that isn't surfaced in the user interface.

We know about what I generally call primary data: calendars, address books, photographs, SMS messages and browser bookmarks. These are usually user generated, and we'd be pretty unhappy if we lost them. There is also the secondary data that the phone generates about us: call history, voice mail, usage information and records of our current and past locations. Most of what I'd call secondary data is surfaced to us in our phone's user interface. We generally can't change this sort of information without resetting the phone to a factory fresh condition; it's generated by the device for us, it's not something we generate ourselves.

But there is also what I refer to as tertiary data. This is data that, similar to the examples I mentioned above, is generated about us, rather than by us. Mostly, this data consists of cache files — data that is entirely necessary to you using the device, or significantly improves your user experience, but you don't necessarily know is there. At least until some hole is found in the operating system to expose that data layer to you. That's happened before, after all.

An obvious example is tucked in your photographs. Every picture you take is geotagged and date stamped, and if you publish your pictures to a photo-sharing site without stripping that information, you're leaking data. Back in 2007, when geotagged photographs of newly arrived helicopters at a U.S. Army base in Iraq were published to the Internet, they allowed insurgents to determine the exact location of the helicopters inside the compound and conduct a mortar attack. Four of the AH-64 Apaches on the flight line were destroyed in the attack.

Where Conference 2012 — O'Reilly's Where Conference, being held April 2-4 in San Francisco, is where the people working on and using location technologies explore emerging trends in software development, tools, business strategies and marketing.

Save 20% on registration with the code RADAR20

Data leakage

Recently, there have been a number of high-profile cases of data leakage, and the one that has raised the most controversy, at least until the next time, is the social network Path.

Upon opening the Path application on your phone, it automatically uploaded your address book to Path's servers so it could find "friends" that you might want to connect to without asking for explicit permission to do so, or even implicit permission for that matter. Path has since apologized and updated its application so that it now asks permission before pushing your address book to its servers.

This was not data theft, but data leakage. You asked the application to accomplish something and didn't really ask yourself how it was doing it. While there are technical solutions that don't involve uploading your address book, the laziest solution is probably what you should have expected. I can almost hear the developers, "... we'll just upload the address book for now and switch to hashing later on when we have time."

There has been a lot of comment that somehow the whole Path thing was unexpected. Realistically, that's not the case. It's not an isolated circumstance, either. To the best of my knowledge Hipster and other apps also tapped your address book behind the scenes without asking permission. Interestingly, there are other, less obvious, culprits. Applications that make use of Chillingo's "Crystal" game service, like Angry Birds, will in some circumstances also upload your address book. While there is a button to push, it is, at least for me, misleadingly labelled and doesn't suggest what's going to happen next.

Data leakage like this is not really a solvable problem at the user level, at least not in real-time. Having multiple permission boxes pop up at regular intervals is a bad design choice; users stop reading them, they lose importance and become ineffective. Just try using Microsoft Windows and you'll understand exactly what I mean. Modal interrupts should be reserved for vital time-critical issues. They're already used far to prolifically in iOS. Run the Mail application with multiple mail accounts configured when you're not connected to the network and that'll become instantly obvious. You'll be bombarded by error messages.

I did have a thought that you might be able to deploy a customized web proxy directly onto your mobile device and have all web requests directed through it. The proxy would sift through the outgoing network connections in a (semi-)Bayesian manner looking for data that you don't want transmitted and stop the application cold before it sends it to the remote server. Basically, it's acting as a reverse spam filter, or a smart firewall, depending on how you want to think about it.

I think that something like this could well be far more effective at stopping data leakage than the current solution, which Google has used on Android: Permissions pages when you initially install an application are all very well, but most people don't read them, and when you're installing an application you're not really thinking about why it might need certain permissions. However, you can be very clear about what data you're interested in not leaving your phone. One configuration page for the proxy, rather than multiple ones, every time you install an application. Like modal dialogs on the iPhone, you subconsciously start to ignore them, to your peril.

Location, location, location

Of course, I can't really talk about data leakage without mentioning the kerfuffle surrounding location and data privacy that happened just about this time last year. Unsurprisingly, the file in question still exists, despite some of the press stories; the existence of the file was never the problem. A cache of that nature is fairly necessary if you want to have reliable and timely location services on your phone. However, the file is now actually just that, a cache, and it is regularly swept clean by the operating system. It's also not included in your usually unencrypted backups to your laptop, which was perhaps more of a problem than the fact it wasn't being cleared out in the first place.

A visualization of iPhone location data. Click to enlarge.

What Apple was doing was taking a piece of tertiary data, generated about you by the device, and then exposing it on a platform (laptop or desktop) where accessing that data was easier. There are a lot of people who know how to navigate a file system on a computer, but a lot fewer who would know how to get the same data directly from the phone itself. It was a classic case of data leakage: data moved from a secure(ish) environment on the phone to a less secure one on the computer.

Data exhaust

Back in the days of floppy disks, the lines of ownership were pretty clear. If you had the disk, the data was yours. If someone else had it, it was theirs. Things these days are much blurrier. That tertiary data — data that's generated about us but not by us — doesn't just build up on your mobile devices of course. Other people are building datasets about our patterns of movement, buying decisions, credit worthiness and other things. The ability to compile these sorts of datasets left the realm of major governments with the invention of the computer.

We're all aware of this, and there's even a provocative buzzword to describe it: data exhaust. It's the data we leave behind us, rather than carry with us.

In the U.S., data from grocery store loyalty schemes has been used by security services to search for terrorist suspects. Turns out the number of toilet rolls you buy can be quite telling.

Which does make me think, instead of being afraid of the data exhaust, perhaps we should embrace it. In the U.K., the biggest retailer is the supermarket Tesco. Like many, I spend a good fraction of my income there, and like almost everyone I know, I have a Tesco Clubcard. This is a loyalty card that has a record of (almost) every purchase I make, from toilet rolls to roast chicken.

I'd actually pay good money for a copy of my own Clubcard data, so long as it was actually in a machine-readable format, not on paper. Although for Tesco, the data is only really interesting in aggregate; it's the fact that they have millions of Clubcard records that makes the dataset useful to the company. To me, a history of my purchases would be useful data.

Of course, people have already started selling our data exhaust back to us. Think about your credit report, for instance.

Keep your friends close, and your enemies closer

It's not just your own data exhaust that you have to worry about. There was an interesting paper recently by Adam Sadilek of the Department of Computer Science at the University of Rochester. It talked about how geotagged tweets could be used to locate individuals, even if they themselves didn't geotag their tweets — it was enough that their friends did so.

Geotagged messages on Twitter during a typical
weekday afternoon in New York City.

The paper found that only a couple of weeks' worth of location data on an individual, combined with location data from their two most-sharing friends, was enough to place that person within a 100-meter radius with 77% accuracy. That rises to nearly 85% when you combine information from nine friends.

Even someone who has never shared their location at all can be pinpointed with 47% accuracy from information available from two friends. That goes up to 57% when you include nine friends.

There is a great debate going on right now, which is really only starting to surface into the mainstream press, about how we share data. Despite social networks becoming mainstream, the recent privacy debacles in the mobile space say a lot about how users perceive information privacy. I think Sadilek's paper presents even more compelling evidence.

For instance, I'm finding Google's new Instant Upload feature, where photos taken on my phone are automatically uploaded to Google+ behind the scenes, a lot spookier and more worrying than I thought I would. It's especially interesting that I'm feeling that way, as I'm using Apple's Photo Stream without thinking or worrying about it that much.

I'm trying to figure out whether it's because the privacy trade-off — in Apple's case sharing my photos between all my devices, and in Google's case making my photos more-or-less instantly available for sharing in Google+ — is more obviously in my favor with Photo Stream, or it's for other reasons.

The interesting thing here is that Photo Stream and Instant Upload are, at least behind the scenes, effectively identical. Both are cloud services and your photos are stored in a data center somewhere. The master copies of your photos have essentially been moved to the cloud, rather than residing on your device.

However, because of the context these two services operate in, I have no problems with one, and I'm finding the other an uncomfortable fit. I think there's a big lesson there for people dealing with personal information. When you're sharing someone's information, even with their informed consent, the context is important about how they think about the implications surrounding that sharing.

Building platforms

So, all of this got me thinking. There are large personal datasets about me, and you, and everyone, being built up by large companies. But we're also building up datasets about ourselves, in our own control. What happens if we mash them together? Can we actually do something productive?

I'm currently running an interesting experiment with my credit card and my iPhone. I'm scraping my bank's website to grab transaction data in near real-time onto one of my servers. Each transaction comes with a postcode. This is like a U.S. zip code, but it normally specifies a much smaller neighborhood, perhaps down to a single street or smaller in a major urban area.

Watching my credit card transactions in real-time.

On my iPhone, I'm running an application that continually monitors my location using the Significant Location Change service, so my phone knows my location to better than 1km (perhaps much better in a crowded city) more or less all of the time.

Every time a new transaction occurs, I forward it via push notification from the back-end server to my iPhone. Now, my iPhone knows both the location where the transaction took place and where I actually was at the time. If those locations don't match, then this indicates there might have been a fraudulent transaction and it flags it for me with a notification.

The interesting thing here is that I'm using data that my credit card company doesn't have, and hopefully will never have: my actual physical location when the transaction took place. They couldn't possibly provide this service to me because they simply don't have the data I have.

Of course, there are false positives. Online transactions in particular stand out. Most of these are tagged with a postcode of the headquarters of the company I'm dealing with. However, my next development step will be to give my back-end server code access to my inbox and allow it to scrape for online transaction receipts. This should reduce the false-positive rate down to something vanishingly small, and I should be able to deal with those left over with some sort of machine learning. After all, there's a human-readable string attached to each transaction that details the retailer and sometimes other useful information.

A thought to ponder

A thought to ponder in the dead of night: In the near future, the absence of data is going to be increasingly unusual. If you think the data exhaust you leave behind yourself is wide and varied, then just you wait, because we're at the banging-the-rocks-together stage right now.

If your data exhaust becomes assumed, what happens if you turn your phone off for an hour or two one night? What if you're accused of a murder during that time period, and you can't prove where you were? Perhaps in the future that's going to be sufficiently unusual that it's automatically suspicious. Innocent until proven guilty may underlie our current legal system, but that's because our current legal system was codified in a very different era, one that was data poor rather than data rich. Perhaps in the future, the absence of data will imply guilt.

Migratory data

2012-03-07T06:19:00.001Z

I spent most of last week at O'Reilly Media's Strata Conference on Big Data in Santa Clara, where I was talking about the data you carry with you. Something I've started to call migratory data.

Big data isn’t just about multi-terrabyte data sets hidden inside eventually-concurrent distributed databases in the cloud, or enterprise-scale data warehousing, or even the emerging market in data. It’s also about the hidden data you carry with you all the time, about the slowly growing data sets on your movements, contacts and social interactions.

Until recently most people’s understanding of what can actually be done with the data collected about us by our own cell phones was theoretical; there were few real-world examples. But over the last couple of years this has changed dramatically.

Interview with Mac Slocum at Strata
Mac Slocum caught up with me about half way through the week to talk about the data on your cell phone, and other mobile devices, along with the possibilities for making use of that hidden data to reveal things about our lives that we might not realise ourselves, about data privacy and data leakage, and about where all this might lead.

Teardown of BiKN

2012-02-15T12:18:00.000Z

A couple of days ago I received a BiKN for iPhone case and tags, with just over a week between ordering and it arriving on my desk, considering the big body of water in the way, I was pretty impressed by the rapid delivery.

BiKN is advertised as a smart case that allows you to "Find your stuff". Effectively it's a wireless based sensor network which makes use of tags and uses your iPhone as the hub of the network.

BiKN for iPhone.

I was also fairly impressed with the packaging, which had be designed so that the instructions on how to charge and use the tags and case were just there. You didn't have to go looking. Everything also seemed to come fully charged, which is always nice.

The inside of the box cover has the instructions.

Plugging my iPhone into the case the standard iOS prompt to download the app associated with the accessory from the App Store popped up. Clicking on it took be directly to the BiKN app as you'd expect. It all worked as advertised.

However there is very little information about how the system works on the BiKN site, so I took one of my tags apart to find out. I was curious about whether they were using an 802.15.4 mesh network, active RFID, or something else. It turned out my hunch was correct and BiKN are using a 802.15.4 network between the tags and the phone case. For those of you in the Maker world, that's the same underlying protocol as the Digi XBee chips use. However BiKN us using a Jennic chipset instead of the, almost ubiquitous, Digi one.

Taking a BiKN tag apart.

There are really only two identifiable ICs on the tag's board. The first is a 4MB MXIC MX25V4006E Serial Flash package, providing a Standard Serial Interface x1 or x2 I/O, Single I/O or Dual I/O, at single 3V or 2.5V power-supply voltage.

One side of the BiKN board.

However the really interesting find is visible when you lift the battery. The tag is running from an Jennic JN5148-001 wireless micro-controller. It is a 32-bit RISC processor, with both 128kB of ROM and RAM onboard to support the 802.15.4 networking stack an on-chip user applications. It looks like the BiKN uses a JenNet-IP based networking solution. This is essentially a enhanced 6LoWPAN network so it's extremely generic. I'm going to be surprised if tags are the only hardware to get BiKN-enabled in the future. This is a serious bid to build an (at least semi-) standards-based Internet of Things from Treehouse Labs.

The other side of the BiKN board.

It's a nicely set out board and interestingly, if you look at the side holding the flash package, there seems to be exposed Rx and Tx lines on the board. Which sort of hints that some sort of serial access might be available. I'm presuming these are going to be a serial pass-through for the micro-controller, but I could be wrong.

The only real criticism I have at this point is the build quality. In fact I'm very disappointed with the quality of the materials that they've used. The case feels cheap and plastic, and the tags are bulky and likewise have a cheaply-made plastic feel. These don't measure up the iPhone build quality at all. I was expecting a much nicer rubberised look-and-feel.

I don't normally use a case with my iPhone, the phone itself is very hard to scratch and lives in my pocket with my loose change without any (well much to speak of) damage. I was hopefully I could live with the BiKN case well enough so I could leave it on my phone all the time, but it's bulky and cheap plastic feel has ruled that out. The case will have to live on my desk and that means I'll have to swap the phone in and out of it when I want to use BiKN. Which probably means I'll use it a lot less that I otherwise would have done.

My only other problem at this point is that, about two times in five, when I open the BiKN app it fails to register the presence of the hardware case. Those times it does register, the case battery life shows to be fairly healthy, so despite the suggestion to charge the case to rectify this, I don't think that's the problem. I've been in touch with the company and they're suggesting this might be a general problem that they're going to address with a forthcoming firmware update, so hopefully that's just an initial teething problem that's going to go away.

Their choice of micro-USB connector for the case, rather than a 30-pin pass-through, is also a bit irritating. I can see why they did it, having been involved around the edges of Apple's MFi programme in the past, I'm presuming using the patent laden 30-pin connection could seriously reduce their margins. But I've got lots of spare 30-pin cables around, and not many micro-USB ones. Charging the phone, and the case, is therefore somewhat problematic.

Finally then, I'm going to be really interested to see if they come out with a publicly available SDK, as I think that'll be the make-or-break for the product. It's moderately interesting without it, but with a decent SDK so developers can integrate it into their own apps? That's much more interesting. At this stage of the product they should be looking for network effects, but they should be looking for them from the developer community, not end-users. A public SDK would go a long way towards filling what seems to be a gaping hole in the product right now.

The reality distortion effect, 3 months on...

2011-08-03T10:26:00.005+01:00

One of the things I've never talked about in this blog, mainly because I talked about it enough elsewhere, is the iPhone Tracking scandal and the bizarre reality distortion effect that went on during those couple of weeks back in April.

Mac Slocum finally managed to corner me at OSCON last week and got me to talk about things now everything has settled down.

Interview with Mac Slocum at OSCON
One of the things I mentioned during the interview as one of the really positive things to come out of the iPhone Tracking debacle is the visualisations from the crowdflow.net group. If you haven't already seen them, I'd encourage you to take a look at what they've managed to pull out of the data.

CREDIT: Crowdflow.net

The movements of 880 iPhones in Europe during April 2011

Talking about connecting iOS to the real world

2011-08-03T10:15:00.001+01:00

I seemed to spent a lot of time in front of the camera while I was at OSCON last week, amongst other things I talked about connecting iOS devices to the real world using the new Redpark serial cable for iOS.

Demonstration of the cable

Interview with Mac Slocum talking about the implications

OSCON 2011

2011-07-26T19:18:00.007+01:00

This week I'm in Portland OSCON 2011 along with OSCON Data and OSCON Java. If you're not here in person O'Reilly is streaming keynotes and interviews live from the conference floor.

The Live Stream Schedule

But if you are around, come see me talk on Thursday when I'll be discussing connecting iOS to the real world and the Internet of Things.

Connect your iPhone to the real world

2011-07-18T17:10:00.014+01:00

The arrival of Google's Accessory Development Kit (ADK) for Android, which allows you to connect your Android handset to an Arduino-based development board was seen by some as the beginning of the end for Apple's restrictive Made for iPod (MFi) program.

Today we discovered Apple's response to Google's ADK, and while it's still inside a crunchy MFi wrapper, the program is now a bit gooey in the middle, as today saw the release of the Serial Cable for iOS from Redpark.

The cable is a fully MFi approved external accessory that allows home-hobbiests to talk to external hardware, no jailbreak required. On one end of the cable is a dock connector that plugs directly into your iOS device. On the other, is an RS-232 serial port that you can easily connect to anything that speaks a serial protocol.

Suddenly connecting your iPhone to the real world became a lot easier, easier in fact than using Google's ADK.

I've been working with the pre-release version of the cable for a couple of months now and I've put up some sample code to get you started. Including a rather nifty Universal application for the iPhone and iPad which will let you directly control an Arduino board. I've dubbed it the "Paduino."

The "Paduino" application.

Paduino Xcode Project for iOS (470KB)
Arduino Sketch for Arduino (1KB)

Because someone had to...?

A simple "Push the Button" example.

Xcode Project for iOS (250KB)
Arduino Sketch for Arduino (1KB)

Also in the works, but not quite ready yet, is a book which will walk you through how to use the cable and how to integrate your iPhone or iPad into the Internet of Things.

Finally if you're at OSCON next week I'll be talking about the cable and how to use it on Thursday. We're hoping to have an early-release copy of the book ready by then.

I can't wait to see what people can do with this...

Apple and the Web Free Cloud

2011-06-16T15:27:00.000+01:00

This article was originally published on the O'Reilly Radar.

The nature of Apple's new iCloud service, announced at WWDC, is perhaps more interesting than it seems. It hints very firmly at the company's longer-term strategy; a strategy that doesn't involve the web.

Apple will join Google and Amazon as a major player in cloud computing. The 200 million iTunes users Apple brings with them puts the company on the same level as those other platforms. Despite that, the three companies obviously see the cloud in very different ways, and as a result have very different strategies.

Amazon is the odd man out. Their cloud offering is bare metal, contrasting sharply with Google, and now Apple's, document-based model. To be fair, Amazon's target market is very different, with their focus on service providers. If you're a Valley start-up looking for storage and servers, you need look no further than Amazon's Web Services platform.

Google and Apple's document model contrasts sharply with Amazon's service-stack approach. Both Google and Apple have attempted to abstract away things, like the file system, which stand between the end user and their data. An unsurprising difference perhaps, Google and Apple are consumer-facing companies that are marketing to the final end user rather than the people and companies who aim to provide services for those users.

But that's where the similarity between Google and Apple breaks down. Google sees the cloud as a way to deprecate general purpose computers in the hands of their users. In the same way that their new Chromium OS is built for the web, their cloud strategy is an attempt to move Google's users away from native applications so that their applications and data live in Google's cloud of services. Perhaps coincidentally, this also gives Google the chance to display and target their advertising even more cleverly.

Apple's approach is almost entirely the opposite. They see the cloud as a way to keep the general purpose computer on life support for a few more years until touch-based hardware is really ready to take over. Apple's new cloud platform is built for native applications, in an attempt to pull users into native apps designed for their platforms. This method also gives Apple the chance to sell hardware, applications, and content that will lock users into their platform even more firmly. This is the basis of the often remarked "halo effect."

At least on the surface things seem to be simple — the "why" of the thing is not in question. However it's what hasn't been said, at least openly, that raises the most interesting questions.

Web 2.0 Summit, being held October 17-19 in San Francisco, will examine "The Data Frame" — focusing on the impact of data in today's networked economy.

Save $300 on registration with the code RADAR

Apple is fundamentally platform orientated. It's deep in their company genetics. The ill-fated official cloning program from the mid-'90s, which was brought to a screeching halt by the return of Steve Jobs, seems to have set a deep fear inside the company about letting someone else control anything that might stand between the company and direct access to their customers.

At least to me, nothing confirms that mindset more than Apple's return to designing their own processors in-house in Cupertino. Apple has a long history of using its own custom silicon, but it's been more than five years since Apple has done so. With the move to Intel, the hope was to delegate nearly all of Apple's custom chip development. Unfortunately, that proved to be a stumbling block when Apple built the first generation iPhone. The Samsung H1 processor in the original model wasn't quite what Apple wanted, even though it was what had been asked for, and I think the return to custom silicon probably brought a sigh of relief in some corners of the company.

The link between custom chips and the cloud may seem tenuous at first glance, but I think Apple's return to designing their own silicon is telling. Almost as telling as spending half a billion dollars on a custom data center to support their new iCloud service. Both moves show the company is now committed more than ever to controlling the verticals. From the chips inside the devices to the data centers their customers' data ultimately resides on, Apple is committed to controlling the user experience, and the web has no place in that.

You might argue that this is because the web is "too open" and that threatens Apple's platform. However, the continuing argument over openness, or lack there of, isn't really relevant. Despite Google's protestations to the contrary, neither of these two companies is particularly open. The very document-based model they're both advocating in their cloud architectures precludes a truly open system. It's such an obvious straw man argument that it's not actually that interesting.

What is interesting is that there was little or no mention of the web, or HTML5, during Apple's WWDC keynote. I think you'll see far less emphasis on HTML5 from Apple in the future, unless someone asks to do something with Apple's platform the company disapproves of, and then the traditional answer of "Well, you can always do that in HTML5" will be rolled out again.

Apple has finally put their cards on the table. They have not yet bet the company on iCloud, but it's telling how deep the integration into both iOS and OS X appears to be. They have for too much invested in iCloud for it to fail, if only in reputation. Whether the first incarnation lives up to its promises out of the box is still to be seen, but success isn't out of the question. Despite MobileMe, Apple does know how to build large-scale reliable backend services. You only have to look at the App Store itself for an example.

So in the future don't be too surprised to see Apple integrate iCloud even more tightly with both iOS and OS X. For the same strategic reasons, don't be shocked to see more custom chips appear — I expect to see the arrival of ARM-based MacBooks and the transition away from Intel for Apple's laptops. That's because for Apple, It's all about the platform.

The next, next big thing

2011-05-19T16:32:00.003+01:00

This article was originally published on the O'Reilly Radar.

In my old age, at least for the computing industry, I'm getting more irritated by smart young things that preach today's big thing, or tomorrow's next big thing, as the best and only solution to my computing problems.

Those that fail to learn from history are doomed to repeat it, and the smart young things need to pay more attention. Because the trends underlying today's computing should be evident to anyone with a sufficiently good grasp of computing history.

Depending on the state of technology, the computer industry oscillates between thin- and thick-client architectures. Either the bulk of our compute power and storage is hidden away in racks of (sometimes distant) servers, or alternatively, into a mass of distributed systems closer to home. This year's reinvention of the mainframe is called cloud computing. While I'm a big supporter of cloud architectures, at least at the moment, I'll be interested to see those preaching it as a last and final solution of all our problems proved wrong, yet again, when computing power catches up to demand once more and you can fit today's data center inside a box not much bigger than a cell phone.

Thinking that just couldn't happen? You should think again, because it already has. The iPad 2 beats most super computers from the early '90s in raw compute power, and it would have been on the world-wide top 500 list of super computers well into 1994. There isn't any reason to suspect that, at least for now, that sort of trend isn't going to continue.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% with the code OS11RAD

Yesterday's next big thing

Yesterday's "next big thing" was the World Wide Web. I still vividly remember standing in a draughty computing lab, almost 20 years ago now, looking over the shoulder of someone who had just downloaded first public build of NCSA Mosaic via some torturous method. I shook my head and said "It'll never catch on, why would you want images?" That shows what I know. Although to be fair, I was a lot younger back then. I was failing to grasp history because I was neither well read enough, nor old enough, to have seen it all before. And since I still don't claim to be either well read or old enough this time around, perhaps you should take everything I'm saying with a pinch of salt. That's the thing with the next big thing: it's always open to interpretation.

The next big thing?

The machines we grew up with are yesterday's news. They're quickly being replaced by consumption devices, with most of the rest of day-to-day computing moving into the environment and becoming embedded into people's lives. This will happen almost certainly without people noticing.

While it's pretty obvious that mobile is the current "next" big thing, it's arguable whether mobile itself has already peaked. The sleek lines of the iPhone in your pocket are already almost as dated as the beige tower that used to sit next to the CRT on your desk.

Technology has not quite caught up to the overall vision and neither have we — we've been trying to reinvent the desktop computer in a smaller form factor. That's why the mobile platforms we see today are just stepping stones.

Most people just want gadgets that work, and that do the things they want them to do. People never really wanted computers. They wanted what computers could do for them. The general purpose machines we think of today as "computers" will naturally dissipate out into the environment as our technology gets better.

The next, next big thing

To those preaching cloud computing and web applications as the next big thing: they've already had their day and the web as we know it is a dead man walking. Looking at the job board at O'Reilly's Strata conference earlier in the year, the next big thing is obvious. It's data. Heck, it's not even the next big thing anymore. It's pulling into the station, and to data scientists, the web and its architecture is just a commodity. Bought and sold in bulk.

The overflowing job board at February's Strata conference.

As for the next, next big thing? Ubiquitous computing is the thing after the next big thing, and almost inevitably the thirst for more data will drive it. But then eventually, inevitably, the data will become secondary — a commodity. Yesterday's hot job was a developer, today with the arrival of Big Data it has become a mathematician. Tomorrow it could well be a hardware hacker.

Count on it. History goes in cycles and only the names change.

The secret is to bang the rocks together

2011-05-14T22:24:00.014+01:00

This article was originally published on the O'Reilly Radar.

Every so often a piece of technology can become a lever that lets people move the world, just a little bit. The Arduino is one of those levers.

It started off as a project to give artists access to embedded micro-processors for interaction design projects, but I think it's going to end up in a museum as one of the building blocks of the modern world. It allows rapid, cheap, prototyping for embedded systems. It turns what used to be fairly tough hardware problems into simpler software problems.

CREDIT: Arduino.cc

The Arduino UNO.

The Arduino, and the open hardware movement that has grown up with it, and at least to certain extent around it, is enabling a generation of high-tech tinkerers both to break the seals on proprietary technology, and prototype new ideas with fairly minimal hardware knowledge. This maker renaissance has led to an interesting growth in innovation. People aren't just having ideas, they're doing something with them.

Goodbye desktop

The underlying trend is clear. The general purpose computer is a dead end. Most people just want gadgets that work, and that do the things they want them to do. They never really wanted computers. They wanted what computers could do for them.

While general purpose computers will live on, like the horse after the arrival of the automobile, these systems will be relegated to two small niches. Those of us that build the embedded systems people are using elsewhere will still have a need for general purpose computers, as will those who can't resist tinkering. But that's the extent of it. Nobody else will need them. Quite frankly, nobody else will want them.

We'll be saying a big hello to all intelligent lifeforms everywhere and to everyone else out there, the secret is to bang the rocks together, guys. - The Hitchhiker's Guide to the Galaxy, Douglas Adams

The humble Arduino is the start of that. The board has multiple-form factors, but a single-programming interface. Sizes range from the "standard" palm of your hand for prototyping, down to the size of your thumb for the almost-professional almost-products now starting to come out of the maker renaissance. From wearable versions like the Lilypad, sized and customised to be stitched into clothing, to specially built boards launched into space onboard the new generation of nano-satellites build on a shoe-string budget by hobbyists, to Google's new Android Open Accessory Kit.

CREDIT: NASA

The ANDE deployment from STS-127 in July 2009.

Every interesting hardware prototype to come along seems to boast that it is Arduino-compatible, or just plain built on top of an Arduino. It's everywhere.

Maker Faire Bay Area will be held May 21-22 in San Mateo, Calif. Event details, exhibitor profiles, and ticket information can be found at the Maker Faire site.

Things are still open. They're just different things.

There has been a great deal of fear-mongering about the demise of the general purpose computer and the emergence of a new generation of consumption devices as more-or-less closed platforms. When the iPad made its debut, Cory Doctorow argued that closed platforms send the wrong signal:

Buying an iPad for your kids isn't a means of jump-starting the realization that the world is yours to take apart and reassemble; it's a way of telling your offspring that even changing the batteries is something you have to leave to the professionals.

I'm philosophical about the passing of the computer. What we're seeing here is a transition from one model of computing to another. We've seen that before and there were similar outcries for the death of the mainframe, as there has been for the death of the desktop. There is plenty of room for closed platforms, but the underlying trend is toward more openness, not less. It's just the things that are open and the things that are closed are changing. The skills needed to work with the technology are changing as well.

What the Arduino and the open hardware movement have done is made hard things easy, and impossible things merely hard. Before now, getting to the prototype stage for a hardware project was hard, at least for most people, and going beyond a crude prototype was impossible for many. Now it's the next big thing.

.Astronomy 3

2011-04-06T14:32:00.002+01:00

This week I'm at the .Astronomy 3 conference in Oxford. A wierd mashup between astronomy, cutting edge computer science, and social media.

The .Astronomy3 Trailer from Markus Poessel.

Shunned by their colleagues for not doing enough research, an elite team of computer geeks has gone into the Oxford underground. If you can find them, maybe you can hire... the .A Team.

Location Enabled Sensors for iOS

2011-03-22T12:01:00.000Z

Yesterday evening I gave a talk on "Location Enabled Sensors" to the Bristol branch of the BCS, the slides from the talk are embedded below.

Updated Visualisations for Japan

2011-03-22T11:30:00.005Z

Download: Fukushima-Data.zip (1.1MB) updated at 22-03-2011T11:44Z

With thanks to Gemma Hobson and her ongoing efforts at data entry, we've now extended the duration of the visualisation to cover a full four days. From 17:00 on the 16th of March through till 16:00 on the 20th of March.

Environmental Radioactivity Measurement, 17:00 16th March - 17:00 20th March -- For data from the Fukushima site itself see the "Readings at Monitoring Post out of 20 Km Zone of Fukushima" data sets online. Data collection from Mlyagi (Sendai) ceases at 17:00 on the 17th of March and does not resume. Several other smaller duration data drop outs also occur during the monitored period.

It can be seen that radiation levels near Fukushima are somewhat higher than the original visualisations, whilst the previous data showed peak around 0.15 µSv/h, measurements are now peaking at 0.25 µSv/h. The typical minimum and maximum values across Japan are plotted below on the same scale for comparison.

Environmental Radioactivity Measurement,
Typical Minimums

Environmental Radioactivity Measurement,
Typical Maximums

Additionally, the map embedded below again shows the environmental radioactivity measurements with respect to the typical maximum values for that locale.

Environmental Radioactivity Measurement,
Ratio with respect to typical Maximum Values

This shows that enhancement around Fukushima is now spiking around four times typical maximums, the previous visualisations showed enhancements of only around twice typical maximums.

Finally, earlier today Sarah Novotny pointed me in the direction of an excellent visualisation by Paul Nicholls showing the series of magnitude 4 to 6 aftershocks that are still rocking Japan. Over 670 to date since the original magnitude 9 earthquake.

Japan Quake Maps by Paul Nicholls

Update: I've just been pointed to this interesting visualisation of the original magnitude 9 earthquake in Japan. It was created using GPS readings from the Japanese GEONET network, which should not to be confused with the similarly named GeoNet network in New Zealand.

The video shows the horizontal (left) and vertical (right) displacements recorded when the earthquake struck. The resulting ground movement ripple propagating through the country is very evident.

Radioactivity Measurement in Japan

2011-03-19T00:04:00.032Z

This article was updated to reflect the latest data at 16:29 20/Mar/2011.
With thanks to Pete Warden and Gemma Hobson.
Download: Fukushima-Data.zip (1.1MB)

Over the weekend I came across some data on levels of radiation in Japan collected by the Japanese government, and helpfully translated into English by volunteers.

Unfortunately the data was also somewhat unhelpfully stuck in PDF format. However between us Gemma Hobson, Pete Warden and I transcribed, mostly by hand, some of the more helpfully formatted files into CSV format (16KB) making it acceptable to Pete's OpenHeatMap service. The map embedded below shows our first results.

Environmental Radioactivity Measurement,
17:00 16th March - 17:00 18th March

For data from the Fukushima site itself see the "Readings at Monitoring Post out of 20 Km Zone of Fukushima" data sets online. Data collection from Mlyagi (Sendai) ceases at 17:00 on the 17th of March and does not resume. Several other smaller duration data drop outs also occur during the monitored period.

As you can see from the visualisation environmental radiational levels change fairly minimally over the time course of the day. Most measurements are steady, and within the historic ranges, except around the troubled Fukushima plant where readings are about double normal levels.

Things become more interesting however when we look at the historic baseline data, the two maps below show the typical range of background environmental radiation in Japan. The first shows typical minimum values, while the second shows typical maximum values, put together they illustrate the observed range for environmental radiation across Japan.

Environmental Radioactivity Measurement,
Typical Minimum

Environmental Radioactivity Measurement,
Typical Maximum

Finally the map embedded below shows the environmental radioactivity measurements with respect to the typical maximum values for that locale. From this visualisation it is evident that the measured values throughout Japan are normal except in the immediate area surrounding the Fukushima reactors where levels are about double normal maximum levels.

Environmental Radioactivity Measurement,
Ratio with respect to typical Maximum Values

However when analysing this data you should bear in mind that the normal environmental range in that area is actually fairly low compared to other areas in Japan. Currently the levels of radiation at the plant boundary are actually still lower than the typical background levels in some other parts of the country. Levels also seem fairly static over time, and do not seem to be increasing as the situation progresses.

Unless the situation significantly worsens, which admittedly is always possible, human habitation in close proximity to the plant will not be affected in the medium term. From talking to people on the ground in Japan, and by looking at the actual measurements across the country, a very different picture seems to be emerging than that reported by the Western media which seems highly skewed, and heavily politicised, by comparison.

I think everyone should take a deep breath, step back, and look at the evidence which is suggesting that this is not another Chernobyl in the making. It's may not even be another Three Mile Island. If the remaining functioning reactor units are decommissioned following this incident it may well have more to do with politics than the science.

Update: A Radiation Dose Chart from the people that brought you xkcd.com. The extra dosage you would pick up in a day while in a town near the Fukushima plant is around 3.5 µSv.

Radiation Dose Chart

For comparison a typical daily background dose is around 10 µSv, whilst having a dental X-ray would expose you to an additional 5 µSv. The exposure from a single trans-continental flight from New York to L.A. is of the order of 40 µSv.

Update: This visualization compares the energy mix and number deaths related to each of the main sources of energy worldwide - coal, oil, natural gas, nuclear, hydro and biomass.

Deaths per TWh for all energy sources

The abandonment of technology

2011-03-03T15:09:00.000Z

This article was originally posted on the O'Reilly Radar

Right now the Space Shuttle Discovery is in orbit for the last time, and docked with the International Space Station (ISS). On its return to Earth the orbiter will be decommissioned and displayed in the Smithsonian's National Air and Space Museum. Just two more shuttle flights, Endeavour in mid-April, and Atlantis in late-June, are scheduled before the shuttle program is brought to and end.

Tracy Caldwell Dyson in the Cupola module of the International Space Station observing the Earth below during Expedition 24 in 2010.
(Credit: Expedition 24, NASA)

Toward the end of last year I came across an interesting post about the abandonment of technology by Cameron Locke. A couple of months later on I read an article by Kyle Munkittrick who argues that the future is behind us, or at least that our current visions of the future are outdated compared the current technology:

The year is 2010. America has been at war for the first decade of the 21st century and is recovering from the largest recession since the Great Depression. Air travel security uses full-body X-rays to detect weapons and bombs. The president, who is African-American, uses a wireless phone, which he keeps in his pocket, to communicate with his aides and cabinet members from anywhere in the world ... Video games can be controlled with nothing but gestures, voice commands and body movement. In the news, a rogue Australian cyberterrorist is wanted by world's largest governments and corporations for leaking secret information over the world wide web; spaceflight has been privatized by two major companies, Virgin Galactic and SpaceX.

I've been thinking about these two articles ever since, and Discovery's last flight brought these thoughts to the front of my mind. On the face of things the two posts espouse very different view points, however the underlying line of argument in both is very similar.

The future is already here and we may be standing at a crucial decision point in our history. Forces are pulling us in both directions. On one hand the rate of technological progress is clearly accelerating, on the other, the resources we have on hand to push that progress are diminishing at an ever-increasing rate. In a world of declining resources, and increasingly unreliable energy supply, you have to wonder whether our current deep economic recession is a sign of things to come. Will the next few decades be a time of economic contraction and an overall lower standard of living?

Big problems, unaddressed

At the 2008 Web 2.0 Expo, Tim O'Reilly argued that there are still big problems to solve and he asked people to go after the big, hard, problems.

And what are the best and the brightest working on? You have to ask yourself, are we working on the right things?

I think Tim was right, and I don't think much has changed in the last couple of years. I'm worried that we're chasing the wrong goals, we're not yet going after the big, hard, problems that Tim was talking about. Solving them might make all the difference.

While not all big problems are related to the space program by any means, the successful first launch of SpaceX's Falcon 9 rocket last December has given me some hope that some of our best and brightest aren't just throwing sheep at one another or selling plush toys.

Despite this, I see signs of a growth in pseudo-science, and an inability of even the educated middle classes to be able to tell the difference between it and more trustworthy scientific undertakings. There is also a worrying smugness, almost pride, among many people that they "just don't understand computers." While some of us are pushing the boundaries, it appears we may be leaving others behind.

We abandoned the moon, then faster flight. What's next?

The fastest westbound trans-atlantic flight from London Heathrow to New York JFK was on the 7th of February 1996, by a BA Concorde. It made the journey in just under 3 hours. Depending on conditions, the flight typically takes between 7 and 8 hours on a normal airplane. As a regular on the LHR to SFO and LAX routes, I spend a lot of time unemployed over Greenland. I do sometimes wish it was otherwise.

A few weeks ago I watched an Airbus A380 taxi toward take-off at Heathrow, and I felt a deep sense of shame that as a species we'd traded a thing of aeronautical beauty for this lumbering giant. Despite the obvious technical achievement, it feels like a step backwards.

I'm young enough, if only just, that I don't remember the Moon landings. However when I was a child my father told me about how, as a younger man, he had avidly watched the broadcast of the Moon landings, and I have a friend whose father was in Mission Control with a much closer view. In the same way I can tell my son that we once were able to cross the Atlantic in just 3 hours, and that once it was possible to arrive in New York before you left London. I do wonder if things go the wrong way — and we enter an age of declining possibilities and narrowing horizons — whether he'll believe me.

The return of the Personal Area Network

2011-03-01T15:03:00.000Z

This article was originally publish on the O'Reilly Radar.

The recent uprisings in Egypt and across the Middle East have caused some interesting echos amongst the great and the good back in Silicon Valley. Despite the overly dramatic language, I find Shervin Pishevar's ideas surrounding ad-hoc wireless mesh networks, embodied in the crowd-sourced OpenMesh Project, intriguing.

Back in the mid-years of the last decade I spent a lot of time attempting to predict the future. At the time I was convinced that personal area networks (PAN) were going to become fairly important. I got a few things right: that things were going to be all about ubiquity and geo-location, and that mobile data was the killer application for the then fairly new 3G cell networks. But I was off the mark to think that the convergence device, as we were calling them back then, was a dead end. Technology moved on and the convergence devices got better, thinner, lighter. Batteries lasted longer. Today we have iOS- and Android-based mobile handsets, and both platforms pretty much do everything my theorised PAN was supposed to do, and in a smaller box than the handset I was using in the mid-naughties.

It's debatable whether the OpenMesh Project is exactly the right solution to create secondary wireless networks to provide communications for protestors under repressive regimes. The off-the-shelf wireless networks Pishevar talks about extending operate on a known number of easily jammable frequencies. Worse yet, there has been little discussion, at least so far, about anonymity, identity, encryption and other security issues surrounding the plan. The potential for the same repressive regimes the protestors are trying to avoid being privy to their communications by the very nature of the ad-hoc network is a very real threat.

Where 2.0: 2011, being held April 19-21 in Santa Clara, Calif., will explore the intersection of location technologies and trends in software development, business strategies, and marketing.

Save 25% on registration with the code WHR11RAD

For these reasons I'm not absolutely convinced that mesh networking is the right technical approach to this very political, and human, problem. Perhaps he should be looking at previous work on delay-tolerant networking rather than a real-time mesh protocol.

However, Pishevar's suggested architecture does seem robust enough to cope with disaster response scenarios, where security concerns are less important, or if widely deployed to provide an alternative to more traditional data carriers. Pishevar isn't of course alone in thinking about these issues: the Serval Project uses cell phone swarms to enable mobile phones to make and receive calls without the conventional cell towers or satellites.

The Bluetooth standard proved too complicated, and at times too unreliable, a base upon which to build wireless PAN architectures. The new generation of ZigBee mesh devices is more transparent, at least at the user level, possibly offering a more robust mesh backbone for a heterogenous network of Wi-Fi, cell and other data connections.

With the web of things and the physical web now starting to emerge, and with wearables now staring to become less intrusive and slightly more mainstream, the idea of the personal area network might yet re-emerge, at least in a slightly different form. The resulting data clouds could very easily form the basis for a more ubiquitous mesh network.

I'd argue that I was only a bit early with my prediction. The technology wasn't quite there yet, but It doesn't mean it isn't coming. The 2000s proved, despite my confident predictions at the time, not to be the decade of the wireless-PAN. Perhaps its time is yet to come?

Ignite London #4

2011-02-16T10:06:00.001Z

On my way back from Strata Conference I stopped off in London and gave a 5 minute Ignite talk at Ignite London on "Who owns your data?."

Who Owns Your Data? - by Alasdair Allan, Ignite London 4, 8 February 2011

Update: Edd Dumbill just posted an interesting article on data as a currency that makes some similar points to my Ignite talk.

Interview on Machine Learning

2011-02-03T22:39:00.000Z

This week I've bee out at Strata, and earlier today I was interviewed by interviewed by Mac Slocum for the O'Reilly Media YouTube channel about my talk on Machine Learning in the Real World and other related topics.

Machine Learning in the Real World

2011-02-03T06:18:00.004Z

Today I gave a talk on machine learning in the real world at O'Reilly's Strata Conference. Can machines help us make better decisions? During the panel we looked at how machine learning is applied in industry and academia to optimise the use of resources and help with decision support.

Cloud Status v4, for the iPhone

2010-11-04T19:51:00.001Z

It has been a while since the last release of Cloud Status, my Cloud Computing monitoring application for the iPhone and iPod touch. So I'm really happy to announce a new release, version 4.

Predictably with such a long time between updates there have been major changes. Along with a number of minor bug fixes, and an updated UI, the application has been fully updated to support iOS4 and the iPhone 4's retina display.

Reporting for MS Azure and the Rackspace Cloud has been added in addition to the existing status reporting for Google App Engine, Google Apps and Amazon Web Services.

The best part? This new release of Cloud Status is now available for free thanks to the generous support of WatchMouse, the application and website monitoring people.

The Daily ACK

Getting Started with Arduino and iOS

Location Enabled Sensors at Øredev

Interviewed at O'Reilly Where 2012

A fireside chat with Dale Dougherty

iOS5 Week at O'Reilly

(More) Migratory Data

The data you carry with you

Data leakage

Location, location, location

Data exhaust

Keep your friends close, and your enemies closer

Data sharing

Building platforms

A thought to ponder

Migratory data

Teardown of BiKN

The reality distortion effect, 3 months on...

Talking about connecting iOS to the real world

OSCON 2011

Connect your iPhone to the real world

Apple and the Web Free Cloud

The next, next big thing

Yesterday's next big thing

The next big thing?

The next, next big thing

The secret is to bang the rocks together

Goodbye desktop

Things are still open. They're just different things.

.Astronomy 3

Location Enabled Sensors for iOS

Updated Visualisations for Japan

Radioactivity Measurement in Japan

The abandonment of technology

Big problems, unaddressed

We abandoned the moon, then faster flight. What's next?

The return of the Personal Area Network

Ignite London #4

Interview on Machine Learning

Machine Learning in the Real World

Cloud Status v4, for the iPhone