Random Title, Working Thoughts

AWS & strategy over the years

2018-11-30T00:00:00+00:00

I started using and consulting on AWS somewhere around 2008. Over the years AWS has extended its service portfolio and its geographic presence, not to speak of the huge increase in computing and storage capacity in all its datacenters.

While the increase in geographical reach and absolute capacity are easy to understand — simple response to raw customer demand — the updates to service portfolio require more thinking. Why this service? Why at that particular time? Are there any clear patterns? Could you predict future AWS services?

Viewpoint: Adoption life cycle

One can look at AWS through the adoption life cycle, and say that

Initially AWS targeted innovators by producing useful MVP services (I’ve discussed one aspect of this in 2013 and still stand by that reasoning.)
Once “cloud” became as a viable business platform (“early adopters”), it broadened its service coverage.
… and so on …

Viewpoint: Developer-visible phases

I chance at another timeline categorization based on my own perception of using AWS services:

Establishing “cloud” as a viable option for new development projects, offering minimal but valuable services via leveragable interfaces (aka APIs)
Pivoting towards making AWS viable for enterprises to integrate and later migrate existing systems to, adding more varied and less developer-focused services, expanding features of existing services
Making everybody’s head spin with a plethora of overlapping and confusing services announced at increasingly rapid pace

The first two are clear, and I would put the dividing line between those two phases at the the introduction of VPC in August 2011 and Direct Connect in September 2011. Why? Because Direct Connect makes more sense in integration with existing workloads in enterprise data center than for cloud-only projects. While VPCs were useful for cloud-native workloads, they were essential for enterprise data center integration and Direct Connect.

This does not mean that enterprises jumped on AWS bandwagon at that point. No no no! — Yet, there is a strategy. Simple toe-dipping projects can be isolated, but to do real business, enterprises needed integration with existing CRM and ERP systems. For that, more enterprise-oriented features came along. Eventually enterprises gained more confidence and actually started migrating existing workloads off their own data centers (an obvious progression).

Viewpoint: Software engineers a.k.a. chaotic code cranking machines

What about the third step, the head spinning and confusion? (I alluded this in 2015 when lamenting the loss of “expert cloud generalist”.)

AWS is announcing new services and new features on existing services at an astonishing pace. Some years back I looked at AWS’s open positions, and was thinking like why is AWS hiring all these developers?

To write software, of course. For existing services? No, while you need some more people when the service is growing exponentially, the basic tenet of cloud engineering it to create systems that scale well. Neither it was tenable that AWS’s retention would have been so bad to require hiring software engineers at the pace that was apparent from their open positions.

To write software for new services.

It often feels that AWS is overrun by engineering teams and service ideas and cannot always produce coherent and co-working interfaces. Maybe we are a phase where AWS usage is growing so fast, they put resources to any development that seems to be “cloud”.

This is a possibility, but it might not be true, or only a partial truth.

Viewpoint: It’s all intentional

Another, a bit more sinister possibility is that AWS is doing the innovate-leverage-commoditize (ILC) cycle faster, and faster, and with more and more software engineers (speeding up internal development cycle multiplied by more developers)

You really should check Simon Wardley’s work on strategy. I’ll let this one tweet from him (with pictures!) lead into why he things AWS is getting faster.

X: Why do you think Amazon is so dangerous? Don't you think they will slow down?
Me: No, they'll get faster.https://t.co/LiUlzyn9RZ pic.twitter.com/TpWmGdulaQ

— Simon Wardley #EEA (@swardley) 30. lokakuuta 2017

Just looking at the ludicrous number of launches and updates would appear to support better the former — chaos — hypothesis. WTF satellite ground station as a service? That’s hardly a high-volume low-margin business. What is next, purchase satellites on a credit card?

On the other hand, it could be that Amazon is using AWS’s momentum in the cloud space to rush into any high-margin high technology area, and assume it has the technological cloud and enough runway to cause chaos and panic on incumbents in any area (satellites?). Leapfrogging to providing lower cost, automated, self-service, something-as-a-service, attempting to exploit the inherent slowness and obstacles of existing market players.

In this context, it would not be a terrible loss if some of these eventually fail. Why would you attempt to compete in the current market, when you have the chance of owning the future market?

I am no strategist, but I’ve learned that most of what appears as strategy is often just a retroactive narrative. What in reality was a jumble of intentional strategies, bunch of accidents and a lot of groping in the dark is often cleaned up, diced and re-assembled into a narrative that promotes the supreme wisdom of the ~~supreme leader~~ company. The question is not whether Amazon slash AWS is chaotic. It undoubtedly is, and a lot of the history will be written into a nice, retroactive, primetime story. Nevertheless, this does not exclude the possibility that amid all of the chaos there is a conscious, strategic direction being implemented.

(You can see all posts from re:Invent 2018 footpedalling down the reinvent2018 tag.)

Betatesting AWS certifications tests

2018-11-28T00:00:00+00:00

I got the whole bunch of AWS certifications in 2012 and 2013 at re:Invent (bunch = all that were available). AWS has a tendency to offer so-called “beta” certification tests at re:Invent. These beta tests are essentially the “next” set of tests to be rolled out on that particular certification.

I do find the idea of using beta testing for tuning of the certification fascinating. Beta-testing a test … enticing. There are some major differences between AWS re:Invent beta-certifications and regular certifications (which are also available):

They are cheaper: this time at 1/2 price of a regular test.
They are way longer. There are 220 minutes reserved for beta tests (170 minutes for non-beta), and I have found that the extra time is needed. I think there are more questions, but it might also because the test questions and multichoice answers are not necessarily always as polished and require more mental effort to work through.
It takes several months to get the test results. For regular test results you get a pass / no pass result right after the test. I guess they’ll need to collate and analyze the results and figure out how to set the scoring etc.
Some of the problems are … unclear or ambiguous. I’m not talking about the typical case of finely nuanced question where you’ve got to be sharp to spot the significance of a single plural. I mean the kind of questions where there are, for example, two valid answers, and the differentiating factor just is not mentioned in the question, no matter how hard one looks for it. Yet, you might have missed the hint, but … perhaps I am assuming too much of myself.

I would still expect these to be — not necessarily made immediately obvious — but at least changed to make the ambiguity only superficial and resolvable in the and.
There may be other problems in the questions. Like, having a large picture in the question, pushing answers behind the fold, with answers being so long that you need to scroll up and down and up and down to cross-correlate the answer to the question (or re-draw the network diagram on a paper).

Although, on the face of it, having 50% discount on a test that costs less than the billable time you put into it … so there might be no good reason to put in the extra effort and delay associated with beta exams.

I do AWS certifications solely for business reasons¹: I don’t personally put much value in certificates — sorry in advance for those who got boatloads of them, you might want to stop reading here — yet I recognize their usefulness in business, in AWS ensuring their partners retain some people with certain minimum knowledge, in partners being able to show some proxy of potential capability etc.

If you think that passing a certification test — say, AWS Solutions Architect — shows you to be an expert in the field, you are falling for a logical fallacy. Yes, experts pass the test. But, passing a test does not imply being an expert.

You should think a certificate as an … say, sports license. It shows a certain determination, and at least a belief in your own potential. It is a ticket to enter a tough competition. It does not imply that you would prevail.

Anyway, how do I think the tests have changed over time?

At least for Solutions Architect tests, I’d argue the same way AWS in general: much more into enterprise side, integration, migration, security policy concerns, enforcement and auditability.

At bit more broad than before, which is nice.

(You can see all posts from re:Invent 2018 pecking the reinvent2018 tag.)

Primarily for gauging the test difficulty so I can evaluate others whether they are ready to take the test. It is useless — even demoralizing — to put someone on even a practice test if they’re guaranteed a failure. It is better to let them gain experience on the job, through training, or other means so they have a good chance of passing on the first try. This is also one of the reasons I prefer beta exams: it allows you to understand well in advance where the normal tests are moving towards. ↩

re:Invent — growth pains

2018-11-27T00:00:00+00:00

This is a just quick post from re:Invent 2018 conference. I am planning to later write more on how the cloud industry has changed over the years since my previous visits at the conference.

However, for now I am just going to ~~bitch~~comment on some practical issues here at re:Invent 2018 and how they relate how the conference when I visited it the last time in 2012 and 2013.

(This is being finalized on Tuesday, end of the second day of the conference. There are still three days to go.)

Locations 👎

The move from a single location (Sands Expo) to multiple hotels and conference venues on the strip is problematic:

You need to transfer between sites unless you can schedule all of your sessions in a single location (for a day) — the transport time varies hugely, I’ve gotten from Mirage to Aria in 10 minutes (morning, light traffic) versus 10 minutes wait time plus 50 minutes transport time (Aria to Venetian) with queues, full buses, and an accident clogging the rush-hour congested route.
Given that popular sessions will be full, not being at the entrance 10 minutes prior to the session can mean complete miss of that session — definitely frustrating if you missed because of transport delays. If you had a reserved seat, they expire at 10 minutes prior to the session start, so that’s sort of a double loss — you made the effort to reserve a seat, then lost the whole session.
The logistics of planning which sessions and where to attend is just … the conference app and web site do not help, either. (I wrote a screen scraper solely to generate an ICS file from the planner’s “interests” just to be able to import them to Google Calendar for easier schedule management. The fact that the planner app needs wifi to show your schedule is a back-ass-ward design. Ever heard of offline use? Like, using AWS’s own mobile services and toolkits?)

Nevertheless, I found that not having a registered seat is not a huge problem — you’d need to be there 10 minutes in advance regardless of whether you have a reservation or not. I’ve managed to get into all sessions so far (I’m sure this streak won’t last, though) where I’ve been at the door 15 minutes prior as a walk-up (no seat registration) attendee. (Which reminds me, that as of writing this, I’ve got next session chalked up in 18 minutes, need to get to the queue to see if my 15-minute experience still holds.)

Crowd 🤷

I have not heard of any official statement on number of attendees, but this conference is crowded, at least in Venetian and Aria (the main sites). It’s not yet elbow-in-your-mouth-crowded, but I’ve seen a lot of lock-step marching in and out of places, bottleneck routes getting really congested.

So, not as terrible to make me skip sessions or venues, but definitely more crowded venues — even with people spread over multiple locations — than in 2012/13.

The other consequence is that you are extremely unlikely to run into a specific person without explicit arrangements — I am pretty sure I met 80% of all Finnish attendees in those earlier conferences. While, in general, conferences are bad places of “trying to meet someone”, I find the extremely unlikelihood of such accidental meets (of people I know are here) somehow distancing, making me feel more detached from others.

Wifi 😢

Between absolutely horrible to passable. Superbly frustrating when you can ping 8.8.8.8, but nothing else seems to pass through to the Internet.

Since I don’t have a local SIM — and did not pay for the 2 GB USA-specific package from my home operator — I can’t even fall back to mobile data.

Which brings me back to this:

Hey @AWSreInvent, imagine the following scenario:

1) go to registration
2) open app for QR code
3) app requires re-login
4) login requires net
5) wifi password in app
6) no data roaming ($$$/MB for non-US SIM)

Wifi pwd prominently++, plz?

(Registration painless w/o QR though.)

— Santeri Paavolainen (@paavolainen) 26. marraskuuta 2018

Catch-22.

Length 🤔

I think — again, based on my memory, too ~~lazy~~tired to check — the earlier ones were three days plus a half day for partner summit. This is a bit of a ±0 thing as more days = more content, but also more days = more difficult to schedule such a long trip. This trip eats two whole weekends for traveling here and back.

Improvements 🙋

Not all is meh. There are some things that I see as improvements over my previous visits, and some things that I’ve heard from attendees as improved from last year.

Lots of guides and helpers — not sure where is something? There’s practically all the time literally a highly visible guide within 20 paces of you. While the number of staff seems excessive at times, they are definitely helpful and useful. I realized I’ve come to almost completely on them instead of even trying to check the venue map on the flaky wifi.
Better transport than last year. I wasn’t here last year, but people who were said it was absolutely horrible last year. I understood that last year the shuttle buses were doing a circular route over all the locations — and this year they were point-to-point.
Overflows. These are screen-and-wireless-headphones combination in most (all?) locations. Some sessions are marked as OVERFLOW and these can be viewed from any of these overflow locations. (I don’t get the terminology though. Overflow in different location? Overflow to?)

I only today realized how useful this was when instead of trying to go to a session and then jump on a shuttle to next location, I could do the other way around and use the overflow at the other location to watch the session at the place where I just left! Doesn’t maybe make sense initially, but when you factor in lunch, travel time etc. it ended up being much easier this way.

Overall

Almost all people I discussed about the size of the conference agreed that re:Invent is too big, multiple venues is pain and so on. This included several AWS staff, too.

How to fix? AWS Summits are clearly a way forward, maybe they could be developed and promoted further, turn them from one-day events to two-day events? Have them more frequently and all over the globe, so there would be no incentive to attend every one of nearby ones?

Move re:Invent to another, more compact location?

Make all sessions remotely viewable live? Live for a fee, some time later free. You’ve already got overflows, which are live, so the capability is there. (Maybe overflows are a PoC?)

Split re:Invent into differently focused conferences? While a large portion of the services are generic and useful in different situations, I can imagine it would be possible to create some differentiations that could act as a divider. Enterprise (migrations, governance, services more relevant to enterprises, etc. etc.) vs. technology focus (less governance, more startup-ey)?

I don’t know. This is not purely a logistical problem, these big conferences (just think of Microsoft and Apple) serve also other goals than just ~~wine and dine~~ ~~entertain~~ educate the attendees.

(I planned this to be a short post. Better stop right n

(You can see all posts from re:Invent 2018 hunting down the reinvent2018 tag.)

re re:Invent

2018-11-25T00:00:00+00:00

I have attended AWS re:Invent conference twice, in 2012 and 2013. This year as the Head of AWS Business from Fiare Consulting will mark my third visit to the conference.

During the conference I’ll be tweeting, posting some stuff here and potentially also in my LinkedIn feed. I do not have high expectations of visibility, so you can read these posts in different media as sort of a process where I collate my conference experiences and try to understand them better in my own context. If someone finds these useful, that’s nice too.

I flew in through Arlanda — how probable is it you are sitting next to another senior from another consultancy going to the same conference? Empirically: not zero.

I flew in already on Saturday, giving me Sunday to adjust to the time difference. While it is sometimes possible to avoid full timezone adjustment, re:Invent is such an intensive event that it is necessary to be full physical and mental strength, and that’s why I reserved a day for that. (I also exercised heavily by hiking to Turtle Head Peak in Red Rock Canyon, and got fresh air and plenty of exposure to sun, all of that helping the body to adjust.)

Anyway, this conference has gotten bigger and bigger. While I cannot remember exact numbers out of my head, it almost doubled from 2012 to 2013 (from several thousands to almost ten thousand, if my memory serves correctly). Last year it was 43 000 and probably quite a few more this time.

While re:Invent was smaller way way back, and while it was definitely more compact (all of it fit in the Sands expo centre), let’s not get too nostalgic: it was big even five years ago. There was absolutely no way you could attend all of the interesting sessions nor meet all the people you thought you’d want to meet! The fact that there are now even more sessions, spread over multiple locations on the Strip does make some practical things harder — but “seeing all” was back then as impossible as it is today.

I have other comments to come, but I’ll cut this post short and just get it out.

(You can see all posts from re:Invent 2018 following the reinvent2018 tag.)

Life is short, why URLs long?

2018-06-26T00:00:00+00:00

At the end of last year, I wrote as part of a thesis work an energy market simulator modeling the Finnish electricity market. While I moved onward after finishing that work, I’ve been intending to return to the project to fix a few of the nagging TODO items.

So, while taking a look at that I also noticed that copy-pasting URLs from the simulator did not work anymore. Ouch!

But why is this a problem? The simulator has a few interesting implementation details:

It runs completely in the browser — the Monte Carlo simulation runs as a web worker in the browser.
It is written in Scala (not JavaScript). Actually, it uses Scala.JS which generates JavaScript from Scala sourcecode (while being mostly cross-compilable to JVM too).
There is no backend and thus no state stored in any backend.
All of the simulation world state (those user can manipulate) is encoded in the URL.

The last one is intended to make two things possible:

If you do modifications on the world state and bookmark the page, then loading the bookmark will get you the modified world and not the default one.
You can share the URLs, as opening the URL will get the same world state as you had.

Something broke

I have been exclusively testing this on Chrome and I do not make any claims or attempts about whether the application works on any other browser.

Late last year the URL copying worked. When I tested it a few weeks back, it did not. Something had changed in Chrome. Or OS X. (I checked Chrome changelogs from last December but could not find anything immediately obvious.)

Regardless of the cause, I wanted to make URL copying work again.

Solutions, so many solutions to choose from!

This was a problem I had considered before, and knew the solution to that already: encode only changes from the default world state. So, what to use? Since the original (“version 1”) data encoding scheme dumped the whole world state as base64-encoded JSON, a reasonable step might have been using JSON diffs — but no, I could not find reasonable Scala.JS-compatible implementations. Also, many “JSON diffs” looked quite verbose and might not have actually solved the problem at all.

Maybe if I encoded the world state as binary JSON (BSON) instead? Alas, I did not find libraries with sufficient Scala.JS support.

No automated luck this time. Let’s roll our own then!

Since the UI only allows users to change the enabled/disabled state and capacity or sources and lines and they have unique ids, it is possible to make a short cut and only encode changes from the default value on an identifier-by-identifier basis. So I wrote a JSON encoder/decoder wrapper for a class that encapsulates such changes.

So now the default world state URL is small since there are only some metadata encoded (no changes to encode). Then, toggle all and change capacities (using the global toggles and sliders) and … too long URL. Can’t copy paste. Damn.

JSON ends up too verbose in this case. Partially this is also due to the encoder/decoder logic which maps a case class Change(name: String, version: Int, changes: Seq[Change]) into {"name": ..., "version":, ..., "changes": [...]} where each change repeats the name, version and changes strings in verbatim. I could have changed to encode the changes as an array ([id,enabled,capacity]) but… decided not to.

I decided to go for a binary encoding directly. BSON was not an option, so what others? MsgPack would have been nice, since it at least has a specification and some cross-platform support, but again, I did not find a ScalaJS-compatible implementation that I was happy with.

There are quite a few binary encoders supporting Scala.JS. Out of those, I settled on BooPickle. With that I got the worst case data encoded as (I’ve broken it to lines of 80 characters, in reality this is all a single unbroken string):

#2-AAVTdW9taQFTAApjZW50ZXItb2lsAgACgJwADmNlbnRlci1vaWwtY2hwAgABAAtjZW50ZXItd2luZ
AIAAorIAApjZW50ZXItYmlvAgABAA5jZW50ZXItYmlvLWNocAIAAobGAAtjZW50ZXItY29hbAIAAQALY
2VudGVyLXBlYXQCAAKBnQAPY2VudGVyLXBlYXQtY2hwAgACiFAADGNlbnRlci1oeWRybwIAAoXrAA5jZ
W50ZXItbnVjbGVhcgIAAQAKY2VudGVyLWdhcwIAAQAOY2VudGVyLWdhcy1jaHACAAKA-wAMY2VudGVyL
W90aGVyAgABABBjZW50ZXItb3RoZXItY2hwAgACgSsADGNlbnRlci1zb2xhcgIAAQAId2VzdC1vaWwCA
AKDNgAMd2VzdC1vaWwtY2hwAgACTAAJd2VzdC13aW5kAgACiQUACHdlc3QtYmlvAgABAAx3ZXN0LWJpb
y1jaHACAAKCTQAJd2VzdC1jb2FsAgAChhYADXdlc3QtY29hbC1jaHACAAKH-wAJd2VzdC1wZWF0AgABA
A13ZXN0LXBlYXQtY2hwAgAChOsACndlc3QtaHlkcm8CAAKHjQAMd2VzdC1udWNsZWFyAgACoCCsAAh3Z
XN0LWdhcwIAAQAMd2VzdC1nYXMtY2hwAgAChYMACndlc3Qtb3RoZXICAAEADndlc3Qtb3RoZXItY2hwA
gACgIAACndlc3Qtc29sYXICAAIOAAlub3J0aC1vaWwCAAIhAApub3J0aC13aW5kAgACi2AACW5vcnRoL
WJpbwIAAQANbm9ydGgtYmlvLWNocAIAAoN4AApub3J0aC1jb2FsAgABAApub3J0aC1wZWF0AgABAA5ub
3J0aC1wZWF0LWNocAIAAoPJAAtub3J0aC1oeWRybwIAAqAfjwANbm9ydGgtbnVjbGVhcgIAAQAJbm9yd
GgtZ2FzAgABAAtub3J0aC1vdGhlcgIAAQALbm9ydGgtc29sYXICAAEACXNvdXRoLW9pbAIAAoLlAApzb
3V0aC13aW5kAgACgQ4ACXNvdXRoLWJpbwIAAQANc291dGgtYmlvLWNocAIAAoZGAApzb3V0aC1jb2FsA
gABAA5zb3V0aC1jb2FsLWNocAIAAqAQSAAKc291dGgtcGVhdAIAAQAOc291dGgtcGVhdC1jaHACAAKBR
wALc291dGgtaHlkcm8CAAKEHgANc291dGgtbnVjbGVhcgIAAqASuwAJc291dGgtZ2FzAgABAA1zb3V0a
C1nYXMtY2hwAgACjkIAC3NvdXRoLW90aGVyAgABAA9zb3V0aC1vdGhlci1jaHACAAKCKwALc291dGgtc
29sYXICAAITAAhlYXN0LW9pbAIAAoMLAAllYXN0LXdpbmQCAAJjAAhlYXN0LWJpbwIAAQAMZWFzdC1ia
W8tY2hwAgAChnAACWVhc3QtY29hbAIAAQAJZWFzdC1wZWF0AgABAA1lYXN0LXBlYXQtY2hwAgACg-oAC
mVhc3QtaHlkcm8CAAKG9QAMZWFzdC1udWNsZWFyAgABAAhlYXN0LWdhcwIAAQAMZWFzdC1nYXMtY2hwA
gACIQAKZWFzdC1vdGhlcgIAAQAOZWFzdC1vdGhlci1jaHACAAJ2AAplYXN0LXNvbGFyAgABAAp3ZXN0L
XNvdXRoAQKgNB0AC3dlc3QtY2VudGVyAQKgNB0ACnNvdXRoLWVhc3QBAqA0HQAMc291dGgtY2VudGVyA
QKgNB0AC2Vhc3QtY2VudGVyAQKgNB0ADGNlbnRlci1ub3J0aAECoDQdABNzd2VkZW4tbm9ydGgtaW1wb
3J0AQKH0AASc3dlZGVuLXdlc3QtaW1wb3J0AQKGQAAScnVzc2lhLWVhc3QtaW1wb3J0AQKGxQATbm9yd
2F5LW5vcnRoLWltcG9ydAECgIQAFGVzdG9uaWEtc291dGgtaW1wb3J0AQKFNQ==

That’s 1950 characters. Not bad! That actually we can copy and paste. (That’s also what is a “version 2” of the data format.)

Yet it is possible to do much, much better. Here is the same URL encoded in “version 3” format:

#3-AAVTdW9taQFTCAIAAoCcCQIAAQ8CAAKKyAACAAEBAgAChsYCAgABDAIAAoGdDQIAAohQBQIAAoXrB
wIAAQMCAAEEAgACgPsKAgABCwIAAoErDgIAAU8CAAKDNlACAAJMVwIAAokFRgIAAUcCAAKCTUkCAAKGF
koCAAKH-1MCAAFUAgAChOtNAgACh41OAgACoCCsSwIAAUwCAAKFg1ECAAFSAgACgIBVAgACDicCAAIhL
AIAAotgIQIAASICAAKDeCMCAAEpAgABKgIAAoPJJQIAAqAfjyYCAAEkAgABKAIAASsCAAE7AgACguVBA
gACgQ4xAgABMgIAAoZGNAIAATUCAAKgEEg-AgABPwIAAoFHOQIAAoQeOgIAAqASuzcCAAE4AgACjkI8A
gABPQIAAoIrQAIAAhMYAgACgwseAgACYxACAAERAgAChnATAgABGwIAARwCAAKD6hYCAAKG9RcCAAEUA
gABFQIAAiEZAgABGgIAAnYdAgABVgECoDQdSAECoDQdNgECoDQdMwECoDQdEgECoDQdBgECoDQdQgECh
9BEAQKGQC8BAobFLQECgIQfAQKFNQ==

Only 591 characters! Yet it encodes exactly the same information. How is that possible?

The world data is named and versioned with the assumption that any structural change will result in a new version number. This means that all source and line identifiers in the model are static and sorting the identifiers will result in a sequence where a particular identifier will stay at the same index! The version 3 data format uses this fact to turn identifier into integers. This helps a lot since the identifiers are actually pretty long (descriptive) strings.

I kept support for the older formats in the code, so if you had a version 1 encoded URL and can get your browser to open it, it should still work. Similarly you can try to open this URL. If you manipulate the model in any way (try toggling a checkbox) it will convert the URL into version 3 format (like this).

If you are interested in the code, you can find it here (I linked a commit version since I might refactor the code later).

P.S. If you are bothered because of inconsistent indentation, it is caused by me sometimes editing the code in IntelliJ IDEA and sometimes in Emacs (with ENSIME). I strongly refrain from re-indenting source files on a whim as it breaks a lot of version history tracking, even on my own source code. As a professional programmer I have learned a long time ago to check in my own ego (regarding indentation and code style) at the door and insead adjust to the style of the codebase currently being worked on.

So if you are a junior: Don’t be an ass — don’t arbitrarily re-style existing code to your own tastes. Touch only the code you actually work on.

Energy Simulator

2018-01-10T00:00:00+00:00

(I have been quiet on this blog for quite a while. I might offer an explanation on that at some point. Although, some of the reasons can be found in this blog post.)

During the latter part of 2017 I worked on my bachelor’s thesis on Engineering Physics at the Aalto University¹. You can find the results online (with a short description and a link back to this article) at energysim.kooma.net and GitHub. If you want to play with the result, click on the first one, and if you are interested in the code and not on a pretty long monologue, click on the second one.

Also, in case you want to get really sciency, just jump to reading the actual B.Sc. thesis.

Introduction

There’s this thing called energy policy. You might not know it even exists, yet it affects you every day. Each country has one, even if it is only implicit in the set of current policies enacted by the government. Energy policy essentially defines the set of goals of how energy is produced and used by a country, and the set of rules that aim to get those goals accomplished. In the olden days, energy policies were mostly about economics and safety of the energy supply. Nowadays, a third axis of environmental effects must be taken into account (especially greenhouse gases, but also particulate emissions et al).

Even countries neighbouring each other can have vastly different energy policies that reflect their own, particular local environment. Norway, for example, has a huge natural advantage of having lots of water and mountains. When these two are mixed, these result in lots and lots of hydroelectric power. So while Norway is a major oil producer, it produces almost all of its electricity from water. Contrast this with France, which has taken a completely different approach having over 70% of its electricity producer through nuclear power. So overall, without going too much into the why’s of energy policy, let’s just say that policies differ from country to country, and there’s usually good reasons beneath the differences.

Also, while I’ll primarily will talk about electricity, please note that “energy use” includes also the production and use of heat (pretty important in colder climates), industrial uses and transportation.

Finland

So, when concentrating on my own country, Finland, and its energy policy, one finds many interesting and conflicting drivers. At the moment, simply put, Finland is dependent on electricity imports. The peak electricity production in Finland is less than the peak demand². Whether this is a problem right now is up to debate, and it may even be possible that in the future the problem will be solved through an unified all-EU electricity market. However, things move pretty slowly in the electricity market, thus any external beneficial effects are likely to be even more slow than any local energy policy effects (it takes easily decades to build GWs of production capacity³).

Energy policy, pretty much by its definiton will be closely tied with politics, which in turn ties into popular opinion. I do not know about other countries, but at least here in Finland there are several trends affecting the attitude towards energy policy in the general population: polarization, loss of interest, lack of trust and unrealistic production method preferences:

There’s a widening attitude gap between rural and urban population. While the majority of Finland’s population is urban (>80%), due to historical reasons the rural population has a larger say in domestic politics than would be immediately obvious. For this reason, these differences cannot be disregarded.
Some segments (read: the same segment that fancies any popular politic, e.g. white, male etc. etc.) are becoming jaded with the complexities of energy policies (yearning back to the times when you could just burn oil without any care in the world?), and of course, if the trend gains more traction will make any kind of rational energy policy more and more difficult as it is hijacked by popular politics.
There’s a marked lack of trust by the general public on politicians (on energy policies), of energy companies … also there are examples in Finland of politicians strongly dismissing energy and policy experts. Not good, of course.

Unless you’ve lived under a rock, you should have seen where popular politics combined with ignorance of facts can lead. So, ignorance, not a good thing. What can be done about it?

Show, don’t tell

I think that people are more receptive towards results they have experienced through their own actions. What if people could play out their own energy policy preferences out, and see how that would affect CO₂ emissions and the safety of electricity supply? Out of this thought resulted the Energy Simulator (or more accutarely, electricity sandbox simulation of Finland but the shorter, albeit less accurate version has stuck). Show below are two different situations, first the default state, and another with some … issues:

Energy simulator running after running some time with defaults (left) and several imports disabled and production capacitities disabled or reduced (right). Light green indicate some regional blackouts and redder colours more frequent blackouts. The CO₂ emissions on the latter are 120% higher than in the former case.

You can go ahead and play with the energy simulator at energysim.kooma.net. I’ll talk a bit more about the technical details below and if you are so inclined, you can check out the source code at github.com/santtu/energysim. You can find also a lot more background information on Finland’s energy situation, on how the energy simulator’s simulation and its parameters are modeled in my Bachelor’s thesis.

I’ll quickly touch on some non-technical issues, but after that I’ll go more into the technical details of the implementation. You can skip rest of the post if you’re not into programming (well, after some other stuff first).

While the energy simulator is nice in itself, it is not a game as it currently stands. It is an open-ended sandbox simulation, and it has lots of caveats and shortcomings due to all of the necessary abstraction and simplification that just getting it finished in limited time required.

Likewise the hypothesis that using a “game” (which this, strictly speaking is not) would be more effective in changing people’s attitudes on energy policy towards more “holistic” approach (from where? from assumed polarized ends?) is entirely unverified. I thought about adding a questionnaire before and after, but finally decided to omit it simply because of schedule (results would not have made into the BSc report).

And finally, I am not an UX designer and either not very good at CSS or web layouts. There are tons of elements that I dislike yet I could not justify myself for spending hours and hours on polishing it from “engineering quality” to “professional quality”, especially when I needed to finish first other things more relevant from the scientific point of view. In the end, it got finished, and I have other things to do. Time to move on and accept it as whatever it currently is.

Let’s get technical

Monte Carlo simulation. The simulation is a stochastic simulation that, for each simulation round, draws random samples of capacities for consumption and production values⁴.
Edmonds-Karp maximum flow algorithm is used iteratively to distribute electricity (from surplus production areas) in a way that minimises global CO₂ emissions.
Everything runs in the browser. This is a pure single-page application, with everything running in the browser including the sandbox simulation. The simulation runs in a separate web worker thread so it doesn’t block the user interface.
Scala all the way down. While the browser runs javascript, the whole energy simulator – simulator core, web worker and user interfae – are all written in Scala.
All state in the URL. The URL contains always contains the current model. You can copy and paste the URL for others. Uuuuunfortunately the URL is also several kilobytes in size and that might potentially be a teeny little problem with some browsers…

The use of Monte Carlo simulation came quite naturally as most of the source data is experimental (e.g. real world) and not expressable as a mathematical expression. This meant that performing any kind of mathematical analysis to infer the overall properties of the system was pretty much out of the question. The use of random sampling in a Monte Carlo simulation results in meaningful and accurate statistics in the long run. Thus after twiddling with the parameters you can let the simulation just run and the most prominent values (mean and standard deviation) are guaranteed to be close to what would be the results of an analytical approach (if ever done) to within some error margin⁵.

From the very beginning I realized the simulation would need to run in a browser. There is no way I could furnish the resources to do this server-side! While the JVM version runs at about 500x the speed of the JS version, there are way way way more browsers out there. I thought about using WebCL for the simulation core to speed it up as all of the sampling could be trivially parallelized, and even the Edmonds-Karp algorithm probably could be ported to WebCL. However, schedules again, and the few iterations per second I get from my old MBP on Chrome is “good enough”.

While the very first trial simulation I wrote in Python, all subsequent development was in Scala. But what, didn’t this run in the browser? Yes! Thanks to the magic of Scala.JS, all of the simulation code works both in JVM environment (at JVM speeds!) and in JavaScript (both in browser and nodejs). Only a very small amount of glue was needed in the actual JS-only land, although since the user interface part does interface with browser, it is very much aware of the javascript-isms of the browser environment (global variables and so on).

While undoubtedly the Scala.JS version of simulation is slower than a hand-written javascript code, I considered writing code in Scala much easier than in JavaScript. While targeting JS from Scala.JS did set some limits on library use, for example, I think it overall offered a much nicer environment. First of all, in the simulator core I could run, test and debug it in scala directly. Thanks to the strong typing of Scala I had a high confidence of it working correctly in the JS world, too. Of course, all the unit tests on the core run in both JVM and JS environments (nodejs for JS, not in browser). Furthermore, the functional programming aspects made it natural to do state management, for example — the “world model” is an immutable data structure, and any changes result in a new version of it. Thus any point where it is updated becomes a natural place to update the single-page application visible (address bar) state too.

I did the user interface using React bindings to Scala.JS. While it took some time to learn their proper use (the distinction between state, props and backend instances and what to use were not always immediately obvious), I liked the result overall. What I came out missing was an easier way to integrate with other (non-core) react components. I ended up using a patched version of scalajs-react-bridge — not an optimal solution, and the syntatic difference between scalajs-react and scalajs-react-bridge usage is a visual irritation.

While all of the model is encoded in the URL, this may result in problems as the URL is a teeny bit long (it is Base64-encoded JSON file). Passing the model via URL does have the benefit that it is possible to pass URLs around. I am not sure how URL shortening services will react to several kilobytes of URL, though…

Okay, the technical bit came out as a bit of a ramble. In case you are interested in more details, or would like to extend or re-use the energy simulator please don’t hesitate to contact me with questions! Also, in case you are reading this in 2020 or so you probably should check out the repository first as it might be more up-to-date.

Not my first degree. ↩
Jääskeläinen et al, Adequacy of power capacity during winter peaks in Finland, DOI: 10.1109/EEM.2017.7981883 ↩
This can be attested by the Olkiluoto 3 nuclear power plant project which has taken over a decade longer than planned. ↩
Transmission line capacity is also sampled, but for simplicity they were modeled at constant capacity (e.g. no failures, no capacity variances). ↩
The “MC results approach true values” argument has several requirements that are not necessarily satistfied in this case, though. The quality of the random number generator in JavaScript is questionable, for example. ↩

Dynamic devtest deployments

2016-01-30T00:00:00+00:00

While I work less and less on down-to-earth development, there are times when I get a chance to hack something. In this post I’ll describe a devtest deployment system I got a chance to be work on.

In a system we are developing we wanted to do a persistent deployment of the whole system to ECS on each branch repository push where each subservice would be accessible as <branch>-<service>.dev.example.com¹. This would make it easy for developer to verify integration tests before merging (pull request) to master², to show their work to other developers and to allow non-developer stakeholders easy access to features as they are being developed.

Flow of each push through the CI. Integration tests are run against a deployment on ECS.

There are multiple ways to accomplish this such as using dynamic DNS updates, ELBs and so on, and after some discussions and testing we settled on the following setup.

There is a separate frontend deployment that consists of etcd and nginx servers.
Each branch deployment service registers itself to the frontend by addings its address to etcd registry with its branch and service names.
A watcher process on nginx container notices changes to etcd registry and updates nginx configuration so that the branch-service.dev address gets proxied to the correct docker container.

Request processing with multiple branch deployments. Each service registers its address and port to etcd registry.

Registration is done during service startup and is also pretty simple (REGISTRY_URL, SERVICE, NAME and PORT are passed as Docker environment variables, ip is fetched from EC2 metadata service):

if [ -n "$REGISTRY_URL" -a -n "$PORT" -a -n "$SERVICE" -a -n "$NAME" ]; then
    echo "INFO: Registering to $REGISTRY_URL as $NAME/$SERVICE at $ip:$PORT" >&2
    url=$REGISTRY_URL/v2/keys/deployment/$NAME/service/$SERVICE
    now=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
    $curl $url/ip -XPUT -d value=$ip >/dev/null || exit 1
    $curl $url/port -XPUT -d value=$PORT >/dev/null || exit 1
    $curl $REGISTRY_URL/v2/keys/deployment/$NAME/created -XPUT -d value=$now >/dev/null || exit 1
fi

Similarly when the service is shut down it’ll issue DELETE on its keys to unregister itself.

The nginx container is based on the standard docker registry nginx container, but it will start a “regenerate” process alongside the actual nginx server. It uses etcdwatch to keep track of registry changes and then finally a separate regenerate.py script takes the registry contents and updates the nginx configuration file³.

if [ -n "$REGISTRY_URL" ]; then
    echo "Registry is at $REGISTRY_URL" >&2
    echo "Starting etcdwatch to generate configuration files" >&2
    etcdwatch -u $REGISTRY_URL -d /deployment -- sh -c './regenerate.py && nginx -s reload && echo `date -u`: Regenerated configuration' &
    pids="$!"
fi

Now some caveats. THIS IS NOT FOR PRODUCTION USE. We use this only for devtest deployments. This setup is meant to make it easy and straighforward to test and verify things during development, even when working on incomplete and definitely-not-ready-for-merge code changes.

I have also omitted a lot of details. Where does PORT come from? (It needs to be container’s host port, and it needs to be unique per container instance.) How to use frontend also on local (developer machine) deployment? What to use as REGISTRY_URL? How to actually integrate all of this into a CI pipeline? — I’ll leave those as a home exercise either to solve yourself, or to pester me to write about them :-)

Anyway we are pretty happy about the setup. It means that any branch will get full integration test love on an ECS-deployed setup, automatically and without any extra work on developer’s side. It also means that errors and problems are a bit easier to debug, since any URL from any log or test user report will automatically reveal branch and service names.

It is also easy to determine a correct url to pass to other people (developers, stakeholders, internal alpha testers) as it is always branch.dev.example.com. We special-cased the default entry point to drop the service name.

Sprint demo coming up? Merge master to demo, push and it’ll be up in a jiffy.

Not example.com in reality, of course. ↩
Before you say that a developer should be able to run integration tests locally and tests should always match ECS-based tests thanks to the magic of docker I’ll answer that yes, they can, it’s one command, and no, since there are inherent differences in deployment in local vs. remote deployment the results are not always the same. Although usually there are no problems, and when there are they usually are configuration and not code problems. ↩
The regenerate script actually just dumps the registry information to a Jinja2 template, writing the result to /etc/nginx/conf.d directory. ↩

On Software Development and Layperson's Perceptions

2016-01-17T00:00:00+00:00

Since I have been in the “software business” for about two decades it is natural that I project my own knowledge on to any software problem I hear. Yet sometimes I get a glimpse of how this whole “software thing” might look to people who have little or no knowledge at all of software development.

When I recently was complaining about a billing service (of a local sporting association) lacking a very useful and common feature, the reply I got gave me one of these “oh, so that’s what it looks like” moments of insight. “The vendor asked too much money for the feature.”

A few thoughts raced my mind.

It just can’t be that expensive - this must be a case of sticker shock!
The system does feel a little home-brewed, maybe it is a bespoke or tailored solution instead of COTS or SaaS?
In that case the vendor may be trying to cover development costs of a new feature, plus some.
Then again the feature is pretty basic and could probably be implemented in one day, with time to spare.

But thinking this way is just looking at trees instead of the forest.

It’s not about my views or my estimates. I am not the customer of the vendor. I do not make a decision here. This is a case of information asymmetry between the vendor and their customer. My viewpoint is more symmetric, and thus not valid in this case.

What is the problem, then?

Before going to the main question I have (about the forest), let me first take a look at some trees first.

Estimation is useless! Estimation is valuable!

While software estimation¹ itself has been studied for a long time, is it even possible to estimate software development efforts in reality? What is its place in the world of agile and lean development?

I’ve seen people think that agile development has made away with software estimation. It has not. Task sizing, planning poker or even just “guessimating” whether a story will fit a sprint or not are all judgments based on software estimation. Doing estimation by the seat of the pants instead of formally does not make it go away.

Software estimation is also known to go horrendously wrong. I am not going to even link examples, it’s that depressing. […] So which is it?

Estimating “small” problems can be done with useful reliability and accuracy.²
Estimating “large” problems is difficult because requirements are not known to sufficient detail.³

My view is that software estimation in itself is not useless and when used in correct context can yield usefully accurate results.

Just think about it yourself — a programmer is making judgments about task complexity and difficulty all the time. If these estimates were completely useless what would that mean? I mean, if you’d estimate a ten-minute task to take five years? You would be bloody useless. And jobless, fast.

Software is easy! Software is hard!

I’ve written previously about the power of being able to write programs. But is software easy? Being able to use programs for automating rote tasks just means that it is incredibly powerful.

Software is not easy or hard. There are hard limits to some problems that come from either the theory of computability or from physical limits, but software in itself is not easy or hard. Learning how to write software may be hard, or it may be easy, but this is as meaningless as saying that learning to draw is easy or hard — some people may have natural affinity, or the drive to learn. If so, learning is apparently easy.

Most of the things we value are difficult and time-consuming to learn. Even if some people make learning look easy.

Learning to do software is hard. So is learning to play violin.

Yet — yet I have often encountered, and I believe to be common that many people think software is easy. “Easy” in the sense that “it cannot take more than a few days” easy. “Easy” in the meaning that “those people at WhizzyCorp got 1M users in two weeks” easy. “Easy” in the suggestion “I can get a random programmer to replace your easy job” easy. Banal, if you may.

Some things that look like magic are actually easy to do in software — now. Some things that are difficult today are easy — in the future. This rapid change may confuse people both ways, both into thinking that something is not possible when it actually has become possible due to a recent development, but also equally well into tricking people thinking that past rapid changes automatically translate into automatically making previously difficult things easy now.

Finally, the forest.

I think that there is a gulf between “laypeople” and “professionals” regarding the difficulty and complexity of software development. This is a point I find difficult to explain even to myself — this train of thought is work in progress. I’ll try my best to articulate this viewpoint in text now.

First, this gulf is not about skills or knowledge. I have absolutely no idea on how to construct an airplane or how much of work it does. Yet someone does.

Someone out there does not have any idea on how much work is to create software for a Mars rover. Well, I don’t, but someone at NASA does.

Unfortunately software development is often bespoke or tailored work. This means there is information asymmetry between customer and vendor. Even when assuming honest and ethical vendors this asymmetry persists.⁴

So when a software professional gives an estimate — making the assumption that it is a reasonably accurate estimate given the constraints I outlined above — what is a layperson e.g. the customer to do with this estimate? There are four possibilities (SWOT anyone?) between professional’s estimate and customer’s expectations:

Both match and are correct: nice
Estimate is correct and expectations are incorrect: customer is happily surprised (estimate is lower) or … put into a bind (estimate is higher)
Estimate is incorrect and expectations are correct: oh woe is me⁵
Both are incorrect: run, don’t look back, just run

Finally, the question:

Why and how do layperson expectations and professional estimates differ?

That’s it. That’s the forest.

It’s not that professionals’ estimates are incorrect. If estimates are used in a valid context then they are likely to be reliable and useful.⁶

It’s not that laypeople’s estimates are incorrect, either. They most likely are incorrect for the exactly same reasons that any random person’s estimates for Mars rover software or airplane construction work are incorrect. Vendor estimates and customer expectations are very likely to differ. Assuming they would match is not a sensible default.

How?

How they are going to differ? My own experience is that they are more likely to be underestimates than overestimates. Yet I don’t consider the quantitative difference as important as the qualitative:

Why?

Why? I don’t know. I tried looking into research into software estimation.⁷ I found papers on estimation techniques, their validity and accuracy, comparisons between them and so on, but I did not find anything that would consider the psychological or sociological reasons why people (especially professionals and non-professionals) would or could take different viewpoints or stands on software complexity or effort estimation.

I have no answers here, only questions.

I think that looking into the why could potentially help a lot in the software industry’s interaction with customers. I think that the software industry or academia is not looking enough (if at all) into the human side — sociology and psychology — of interactions between humans in software professions and humans in other professions.

Why are customer requirements misunderstood? What are the warning signs in human communication or behavior?

Why customers think they have clear requirements when they are not clear? What is an effective way to communicate the inadequacy of requirements?

And so on. Consider research into group-think, for example (Bay of Pigs decision-making is a famous example). This is not computer science, not computing science, not software engineering. It is cross-department stuff. Not very popular in CS, I know⁸.

The Big Conclusion

Nope, there is none.

I wrote this blog post because I got a rare glimpse into non-software-person thinking, got thinking, and found out questions I found no answers for.

For an all-around view on software estimation in practice I can recommend Steve McConnell’s book Software Estimation: Demystifying the Black Art. ↩
Meaning useful in the context of software development project. It is not uncommon to go 3x or 10x way off in estimation of tasks in a scrum sprint, for example. However agile methods have feedback processes meant to keep this deviation from ballooning uncontrollably. In this context estimations do provide metrics useful to guide development projects. ↩
This is the crux of agile methods. While a waterfall software project could theoretically be estimated accurately given that requirements are known in advance, in practice nobody knows the requirements in advance (even when they think they do). Agile methods start with the assumption that requirements will change. ↩
Dishonest and unethical vendors may use the asymmetry to their own advantage. Yet information asymmetry can cause problems even for honest and ethical vendors and their customers. ↩
If the vendor estimate is higher, then they will not get the job and it’s their loss. If vendor’s estimate is lower, they will get the job but there will be hell later when the either the customer ends up paying more than they expect, or the vendor will go negative profit on the project. ↩
Likely, likely, likely. Never 100%. ↩
I have to admit I did not do a thorough literary search. Just random searches on scholar, ieexplore, university library search portal and the like. ↩
If you got here, please read this. ↩

Wardley maps at the low end

2015-09-11T00:00:00+00:00

I have had this nagging thought about Wardley maps. Don’t get me wrong, I love Wardley maps! For example I have found the concept of evolution to give me insight into a lot of both recent and older events both generally in the technology sector as well as in business aspects of companies I have worked for and worked with.

But: There’s this thing about almost all the practical examples I have seen on Simon’s blog:

They are examples from large companies and large public organizations.

I work mostly with smaller companies — startups. Can Wardley maps be used with them? Would mapping offer any benefits?

It is possible to argue both ways — startups are resource constrained and would greatly benefit from increasing their leverage (more bang for the buck), yet the same resource constraints may make it difficult to actually execute strategic manipulation plays! There’s also tremondous variability between different startups. So,

Is it possible to use Wardley maps with startups? (I think: Most likely)
If so, are there any benefits of using them? (Probably)
Finally, what are those benefits and how are they influenced by the company size?¹ (Maybe)

Does size matter? (Source: Miguel, CC BY-NC 2.0)

I am pretty new to using Wardley maps within organizations, so it is possible that I’m doing everything wrong, or being biased, and regardless my sample size that I’ve studied with this question in mind is only three companies so far, and even then the feedback is from short hands-on introduction sessions to Wardley mapping. These sample companies were soliticed from my current customers and professional network and included:

A small team forming a virtual startup within a larger organization, being responsible for a commercial web-based B2B service recently introduced to the market.
A medium-sized group at a startup with MVP development in progress in a healthcare segment. This segment is likely to face disruption (war!) due to digitization of its services within a few years.
A larger group at a startup in very early forming phase — they have a business idea but no clarity of a MVP at the time of the session.

So something between a few people to at most about ten people, okay? I think it possible to define an early startup’s size only after the fact when the difference between contributors and hang-arounds has become clear.

Using Wardley maps

Once you have a Wardley maps of your business and the surrounding business ecosystem, there are several possibilities on how to use it:

Manipulate the playing field to your own advantage.
Anticipate developments that will occur and potential actions by competitors.
Decide what to purchase or outsource and what to do internally.
Select best models to use for internal work.
Choose appropriate purchasing models for external sourcing.
Define teams, their goals and responsibilities.
Define roles and hiring profiles for different teams.

What I found out is that in this sample of startups:

Manipulating the playing field was felt to be out of reach and anticipation of playing field changes and competitor actions were similarly felt to be “too far out”.
Having multiple teams, defining hiring profiles and deciding internal work model were also mostly out of scope as hiring and teaming is driven by immediate tactical needs more than any long-term planning effort². It is pretty pointless trying to create artificial team divisions among five people or less, or try to run a single team in different operating modes.

However there were things that the audience found useful:

“Eye-opener” was a frequent comment when talking about “purchase, outsource or develop internally” decisions.
Differences between purchasing models for different evolutionary stages was also appreciated.
Although I suspect the understanding of evolution was superficial at this stage, comments were made about how this helped visualize potential risks of relying too heavily on early stage technologies (leaning too much on the “bottom left quadrant”).

What to make of this?

Although it is nice to have provided useful insights to participants in these sessions, the relevant question really is did this offer more value than using the same time on some other strategic method (self-study or facilitated)?

Answer: Yes and no and it depends.³

Yes, it clearly provided insights for people on topics they might have come across only by chance.
No, as these insights were common knowledge. Different project, development and organizational models are nothing new and neither is the knowledge that there’s no one-size-fits-all method.⁴ The same situation is with outsourcing methods — hey, this stuff is introductory business course material.
It depends — I have found out that a consultant, a facilitator or a teacher often is paraphrasing common information that the audience already knows but has not yet understood. Helping with this step between knowledge and understanding can be valuable in itself.

Conclusions

Based on feedback and my own observations from these relatively short introductory sessions with early-phase startups, I would say the benefit of using Wardley maps for small, early-phase startups is inconclusive.

There are clearly some benefits, but these benefits might not be attributable to Wardley maps specifically. There is a possibility these benefits could have been reached with some other method too with comparable effort. Even then the benefits appear to be more tactical than strategic.

Afterthought

Well… there is definitely a possibility for early-phase startups with little resources, with enough drive and willpower, to manipulate the strategic playing field to their advantage. For most? Probably not.

After some contemplation I thought of one situation where Wardley maps might prove very useful for early-phase startups: when it is looking for funding. Putting effort into using Wardley maps and understanding the strategic business environment, at least I believe so, would help a company put forward a much better story about the company and its business for potential investors.

I think the case for “strategy” in startups faces the same problem as any “X” that could help them plan and execute better: there are too many “X”s, too little time and not enough money and time to throw around. It just is not feasible to evaluate and use every “X” that might or might not work. I think that founders who realize their need to understand the strategic playing field and the will to manipulate it are already in the “better off” category — they probably are more experienced with better connections and better access to funding, too.

For a next step, how about looking into still-quite-early startups with seed funding, and an MVP on the market?

Here I am using “company size” as a proxy for its resources. This is just a heuristic, as said, there’s tremondous variability between startups. ↩
There is of course a lot of effort put into hiring “the right people”. This hiring just is not driven by any understanding of bimodal- or trimodal IT. ↩
My favourite answer to all yes-no questions. ↩
Most people I know and value have no illusion that any variant of agile, lean or six sigma would be a fit for all needs. This view is supported by plenty of research too. ↩

Watts, watts, watts!

2015-06-18T00:00:00+00:00

A few days ago I read this tweet from Nicholas Weaver about laptop fans spinning on a certain web site.

@ncweaver @mikko How about energy labels for websites? Forbes clearly a D.

— Santeri Paavolainen (@paavolainen) June 16, 2015

It was sort of a joke. Think European Union energy labeling. Would a random site get an A++ or a D energy efficiency label? Based on what? What a thought!

But as things go, that thought would not leave me alone. There clearly are some applications that routinely will busyloop on a cpu core¹. As Nicholas said, there are also some web sites that put a large burden on the processor, too. You can literally feel that as heat on your lap.

How much?

The question is how much power can a power-hungry website consume?

I am ready. I have a pluggable power meter, computer, paper and a pen.

So I set up to work. First I measured² some baseline power usage levels on my laptop³ with different screen brightness levels:

From left to right: lowest visible brightness level, screen off, 50% brightness, 50% brightness with browser running and 100% screen brightness.

I decided to use 50% screen brightness for my tests. Notice that the difference between screen off and lowest brightness level seems neglible, which is interesting. I had expected that the backlight would consume significantly more power than screen completely off. (The measurement baseline of screen at 50% intensity with Google Chrome running and a single incognito window open used 10.8 ± 0.7 W.)

Having set up the baseline, time to browse some sites! After gathering data on several randomly chosen sites I divided sites into three groups, low, medium and high power usage:

From left to right, in low power group: New Scientist, BBC, Apple, YouTube, Google; in medium power group: Vimeo playing a video, YouTube video in fullscreen mode, Vimeo video in fullscreen mode, The Guardian, YouTube video; in high power group: The New York Times.

Power usage by group was

Low power group: 10.7 ± 0.9 W
Medium power group: 20 ± 3 W
High power group: 48 ± 3 W

Considering Nicholas’s comment I was surprised about Forbes being in the low power group. One factor might have been that I have the Flash plugin disabled by default, and there was at least one Flash ad on the Forbes front page. Secondly, I was expecting a more uniform power use distribution, but at least these results were quite stratified. I was also expecting that video sites would be the most power hungry. They weren’t.

The main conclusion based on this very limited sampling is nonetheless clear: there are significant differences in browser power use between web sites. The difference between low and medium group is almost 10 watts and grows to almost 50 watts between low power group and The New York Times site.

Post Scriptum: Does that matter?

The global electricity consumption⁴ is about 2 terawatts. Even if we assume that 30% of world population use 1 hour a day browsing the web then 10 watts more power would mean a total of 875 megawatts more power consumed, which is only 440 parts per million of the global electricity consumption.

So is 875 MW a large number or not? Perhaps it is better to compare it against power conservation efforts. Let’s take the European Union energy labels for refridgerators as a reference. When the labels were introduced the lowest energy label was an A. Now it is A+++ whose difference is 33 kWh per annum⁵. This difference multiplied by the number of households in EU-28⁶ totals up to 790 megawatts.

In the same ballpark.

These are just numbers, but I think they show that it is possible that power-hungry websites can potentially consume significant amount of power by end-user computers.

I’m not naming any birds, thundering or not. ↩
Most applications turned off, no Time Machine backups running, battery at 100%, not using the computer during measurements, starting measurements only 10-30 seconds after page load, pausing video until player has cached as much as possible before video plays, measuring power meter visually from a digital display at 5 second intervals for 30 seconds for a total of 7 measurements for each test. ↩
13” Retina Macbook Pro, 2013 model. ↩
Source: Wikipedia ↩
This is actually normalized to the volume of the refridgerator, but I’m willing to take a chance in taking this difference as a valid average. ↩
210 million, source: Eurostat ↩

Custom Termination Policy for Auto Scaling

2015-04-25T00:00:00+00:00

While chatting with Thomas Avasol about auto scaling group termination policies¹ I got an idea on how to implement custom termination policies for AWS auto scaling groups.

Background

(Feel free to skip ahead if you are familiar with auto scaling groups and termination policies.)

A bit of a background first: When an auto scaling group is scaled down, an instance to be terminated needs to be picked. There are sevaral policies to choose from: oldest/newest instance, oldest launch configuration, closest to full instance hour and the default (most complex) one. A customer gets to choose one of these and that’s it.

If none of these suited you, there are some optios available:

Use termination protection to prevent some instances being picked up for down-scaling termination. (This has been recommended to me as one way by several AWS engineers.)
Run your own down-scale scheduler in an instance. This would effectively implement the downscaling logic itself, but with your own custom twist for choosing instances to terminate.

I consider the first one almost a kludge — an it’s a binary decider, so more nuanced policies are impossible with it anyway — while the second one has superb flexibility, it brings in a new potential SPOF and maintenance burden. So neither is a perfect solution.

Oh, and why would you want a custom termination policy? Well… maybe those instances host large caches, and you’d like to terminate instances with the least filled caches? Or you are running batch jobs from a queue, where some jobs run long and some short and when jobs are drained you’d want terminate only instances without jobs?

(You could run the downscale scheduler inside the auto scale group, but that forces you to open the can of distributed worms including leader worm election and all that. I would not want to go down that road. It is much easier to instead run the scheduler in an instance within an auto scale group set to one instance in size, ensuring it’ll get re-created. Either way, it requires an instance, a thing that I’d like to avoid for smaller groups and simpler custom termination policies.)

Idea 💡

Downscaling in an auto scaling group works when a CloudWatch alarm is triggered, which in turn is set to trigger an scaling action that then changes the desired instance count.

Now… alarms can also send notifications to SNS topics,
SNS topics can trigger Lambda functions,
Lambda functions can be assigned IAM roles, and Lambda functions can access all AWS API functionality that the IAM role allows,
Ergo, I can move the custom termination policy code to a Lambda function without needing to run a separate instance.

With my head humming I started hacking at it and got a proof-of-concept version running already a few hours later. It ~~is alive~~ works!

Code

I’ll tell you how to actually set up this in a while, but first, here’s the Lambda code.

It’s a bit verbose. Using Underscore or Coffeescript surely would be more readable and/or compact. I’m not really a Node.js developer which probably also shows. This certainly is not production quality as it makes quite a bit of assumptions and has hard-coded values in it. This is a proof-of-concept code! Caveat emptor!

The code is straightforward: exports.handler will get called, it’ll extract auto scale group name from the SNS notification, then iterate over running instances in the group and fetch a total time value via HTTP from these instances (a custom CloudWatch metric would do as well), finally picking the instance with the lowest total time value.

Setup

The Lambda function above will not in itself yet do anything. To run it, you need to first (assuming you already have auto scale group set up):

Create IAM role with enough permissions to do auto scaling actions
Create the Lambda function
Create an SNS topic and add a subscription for it to call the Lambda function
Create CloudWatch alarms to send a notification to the SNS topic from above when you want a downscale to occur.

You will need multiple alarms set up for multiples of your cooldown period. For example, if you want to downscale if average CPU load is less than 25% for 10 minutes, you’ll have to set up the first alarm at 10 minutes, second one at 20 minutes, third at 30 minutes up to the maximum number of your scaling group (see below for details why).

Done!

Caveats

I ran into some limitations of Lambda and CloudWatch when working on this:

Lambda functions can access instances only via public IP addresses — PrivateIpAddress does not work. This is a bit of a security bother, so I’d actually suggest to push custom cloudwatch metrics from instances that the code would read without relying on public access.

(I’d like to refer to Lambda functions in security groups directly and use internal IP addresses to access AWS resources.)
I originally had the downscale handler set the triggering cloudwatch alarm to INSUFFICIENT_DATA state in a hope to make it re-trigger later again. It got stuck in a re-trigger race loop, getting repeatedly triggered (downscaling the group into 1 instance in one fell swoop).
I could not figure out a way within AWS services to cause a delay in re-triggering the downscale handler. It is possible to have delay queues in SQS, but no way to trigger Lambda functions based on SQS messages. If there ever is a way to trigger Lambda functions from SQS messages (or route SQS messages to a SNS topic) then it would be possible for the handler to inspect the original alarm state after a cooldown period and requiring only a single alarm.

(I’d like either a way to schedule Lambda call for later, to trigger Lambda functions via an SQS delay queue, or to specify to an alarm to retrigger after some cooldown delay if it is still in a trigger state.)
Lambda is currently available only in US East, US West (Oregon) and EU West regions, which may limit its usefulness at the moment. On the other hand, it should become available in other regions soon-ish.

Note that auto scaling apparently has some internal logic that causes it to inspect the alarm state after cooldown period, retriggering the scaling action multiple times as needed. An alarm, however, will send only a single notification to SNS topic, thus making the Lambda function edge-triggered on the alarm. (Auto scaling magic turns that into level-triggered action instead.)

Anyway that’s it — please drop a comment below if you find this useful!

On 21st April 2015 at AWS Summit Stockholm, to be precise. ↩

Picoservice bruhahaha

2015-04-21T00:00:00+00:00

I’ve been busy, as again, an haven’t had a good chance to continue on my µ²services series. I’m planning to discuss more of the potential implications of technology development meeting microservice architecture models. But this post isn’t about that.

Instead I want to comment a bit on the nano/picoservice commentary found over the net. For example:

@michaelneale I thought we agreed that was the hip new marketing term for functions?

— Mark Wotton (@mwotton) February 3, 2015

I absolutely love these comments!! I absolutely think that at least 95% of use of “microservice” is just hot air. However if you take “microsizing” as an end goal itself and extrapolate to smaller and smaller scales you get nanoservices and picoservices on an function and instruction level granularity:

1k smaller microservice is nanoservice; exposes assembly instructions via HTTP+JSON; pikoservices become inadressable due to quantum effects

— Tomas Petricek (@tomaspetricek) December 3, 2014

Tomas of course is making a point of this absurdity.

What I think what microservices (and by extension, nano and pico too) are: Microservices are externally loosely coupled but internally tightly coupled functional service components.

Although it is not evident from this definition, most value from microservice architecture actually comes from organizational improvement by making the underlying loose-tight coupling explicit in development, management and operations. This also means that the true potential value of a microservice architecture is difficult to determine as it is more dependent on the development organization itself than in the actual software they are developing.

In the end you must ask yourself

Do microservices make snarzzz more flordbious?¹

It just depends. Depends on you, your team, your resources, your processes, your choice of technology, everything. Investigate and decide yourself. Don’t be a slave to backward causality.

So to clarify for my imaginary pundit what I mean and don’t mean when talking about micro- or µ²services:

Replacing function calls with “services” is just idiotic. Plainly and simply idiotic and anyone who claims anything remotely similar is a plain walking and talking bullshit machine.

This same thought experiment was done in reality with remote procedure calls (RPC) already a decade ago and the result is local and remote operations are fundamentally different. You can not create a distributed system based on “function invocation” pattern.

A call to a library routine should now and in the future be a local function call².
I use or at least used to use the term µ²services to emphasise that the trend of shrinking containers may have (some unforeseen) consequences on how we develop and use microservices.

Perhaps I should just stop using that term instead.

Anyway, check out some of these cool posts and sites: A VM for every URL by Magnus Skjegstad, Microservice Classification Model Proposal by Daniel Bryant and of course, the ALL CAPS AS A SERVICE.

P.S. All of the tweets above were written clearly with tongue in cheek. Yet, the fact that people are feeling the need to dismiss “picoservices” for me is a sign that the idea of shrinking services as a goal itself is floating around — and since I’ve been writing about µ²services I want to make it clear that that particular model of sub-microservices is not what I am talking about.

Substitute your own corporate, process or agile buzzwords. ↩
With no significant external constraints or requirements applying. This is not science, this is engineering. You can always find counter-examples but they do not a general case make. ↩

Divine concurrency

2015-02-14T00:00:00+00:00

In previous post I described µ²services, a system development model that is based on extrapolation of current trends in microservices and shrinking containers. I argumented that potential benefits of µ²service model might outweigh its costs. But are µ²services really technically feasible?

In this and future posts I’ll go through some of technical details both from feasibility and benefit points of view, with probably one idea per blog post to keep them manageable in size.

To summarize µ²services: µ²services is container-per-request model where a new virtual machine¹ is created for each request made to the service which then handles the request and only that request and is destroyed after response is generated.

A warning for the reader: All of this is pure speculation on my part. µ²services might happen, but they might not. This is futurology. Do not think this is technology that currently exists (although technological precursors exist.)

Divine concurrency

I have previously argued that concurrency is hard and developers should primarily use language and software architecture constructs that naturally result in safe code. I think µ²services offer a way to create massively parallel service architectures where risks associated with concurrency (dead- and livelocks, mutable data and so on) are either completely eliminated or largely reduced and limited in scope.

The graph below shows a hypotethical call dependency path for a request. The service is composited of multiple smaller (micro)services which themselves are a group of µ²services. State is managed separately and the state storage mechanism is shared by all of these components (via a deployment configuration parameter). The graph could describe a conventional service as well where the colored blobs would signify a service process boundary and circles functional elements within the service.

µ²services responding to a request to /listings — so far this looks like a regular microservice.. (Letters and numbers match those in the second graph.)

In an µ²service the colored blobs are service boundaries, a mechanism to group several different µ²service endpoints together. Perhaps they all share the same configuration elements, or same repository and release tag or similar. With µ²services a “service” is more of a convention than a fixed entity.

Individual circles represent separate µ²service endpoints, pieces of code that can be invoked externally by either users or other µ²services. When not running these are essentially templates that are instatiated when a request is received. Thus each inbound arrow in the call graph represents a new virtual machine that starts to run the service code. For example, calls c and f run the same function² but in different virtual machines. Same applies for n and o which result in different virtual machines 13 and 15.

As described, all µ²service call graphs are acyclic — while calls “up the chain” are logically possible, they result in separate instances making the physical call graph acyclic. The graph below is another view of the same call graph, but structured to make it clear that there exist only forward dependencies. It shows the lifetime of each call (virtual machine), where each either creating a new virtual machine (request) or resulting in a termination of one (response).

Virtual machines running concurrently when processing the request in the earlier graph.

Concurrency is difficult in monolithic services. A single spinning thread can block the whole system. Microservices offer a potential for increased concurrency by allowing concurrent requests to dependent services. µ²services with even finer service decomposition has the potential to offer even more concurrency.

Note also that since each service-to-service request is explicit they can be separately managed for failures and timeouts. Timed out requests are no longer left running — terminated requests will result in termination of their container, which will propagate request terminations further down the chain. For example in the graph above if the user terminated the request (closing the HTTP connection), it would cause instance 1 to terminate, followed recursively by all dependent instances. This would occur even if some code was stuck in an infinite loop³!

Like all things, µ²service model is not a magic dust that could turn any system into massively parallel system. It is not, and cannot be. What it can do is what Erlang as a language has done — it can make concurrent programming a little less error-prone. One benefit — compared to Erlang at least: using a different service model does not require you to learn a new programming language.

That’s all this time. I would appreciate any comments on uuservices — please share your thoughts by adding a comment below. Thanks.

I’m using “virtual machine” to emphasise the isolation between µ²services. Zones, containers, schmontainers, whatever… as long as all service calls are both spatially and temporally isolated. ↩
With function I primarily emphasise µ²service’s difference to traditional monolithic services or even microservices. µ²service functions are not programming language functions — image.flip() is a local function call that occurs within the runtime environment of the µ²service instance. ↩
Here’s a question on your favourite web service framework: If you make a GET request handler that will sleep for a minute and make a request to it killing the client immediately after request has been sent. When will the request handler terminate? Immediately on receiving TCP FIN? After the sleep completes? Somewhere in between? ↩

µ²services

2015-01-31T00:00:00+00:00

Previously I started playing the futurician, a path I’m now continuing against all good advice. I’ve always envied “visionaries” at companies as they get to play around without any kind of responsibility (they will have long since flocked elsewhere when their predictions can be checked). Similarly I am planning to make bold predictions of a future so far away that a claim of flying pigs would have equal credibility.

State of the art

Unless you’ve lived under a rock, you must have heard of Docker (and its take by AWS, Google and Azure), a kind of applications-on-(almost-)virtual-machines containerization mechanism.

Yet docker is just one evolutionary step in a long path of application deployment models. Way way back during pre-history computers were expensive and so much slower that CPU cycles were soooo expensive it made sense to cram as much services into a single server to minimize operating and capital costs. This was the era of mainframes, followed by minicomputers and later UNIX servers as relative hardware costs kept decreasing. Although the per-cycle cost decreased, for a very long time the tendency to run multiple services in a single server remained.

The widespread rollout of first server virtualization, then of “The Cloud”, allowed large servers to be splitted into smaller, discrete virtual machines. This made it feasible and later commonplace to run only a single service per virtual machine. This step is also crucial for later adoption of deployment automation as it allows each service to be equated with a machine, vastly simplifying problem resolution — there is no fear of hurting “other” services when rebooting (or re-creating) a one-service machine.

Docker is one step in this path of shrinking (relative) deployment footprints. Fundamentally it does not differ from a service-per-machine model as its containers have in practice similar isolation properties as earlier service-per-virtual-server model. In practice it is a major step: Virtual machine startup latency is anything from half a minute up while a docker container the startup time is in seconds to tens of seconds range. Other overheads such as memory and disk use are also reduced — for a single server these latencies and overheads would not matter much, but in the scope of cloud services with thousands of servers these seconds and gigabytes start to add up.

Service deployment speed has increased while the cost to run a service has simultanously decreased. (Images courtesy of Wikimedia Commons and Clive Darra.)

Coincident to this evolution — or perhaps co-evolved — are microservices. Microservices in their core are services, but scaled down so that a single service performs only narrowly defined operations. User-visible services are in not monolithic services, but are created as a composite of multiple microservices orchestrated together. For example see netflix blog for discussion on how their business runs hundreds of services on thousands of machines. This development mirrors the service-in-a-machine trend by shrinking services providing further benefits for simplifying and speeding up deployments.

So, that’s the situation now. There is an architectural trend towards distributed, asynchronous, microservice-based systems. Simultaneously the environments these services are deployed into are becoming both more numerous, smaller in footprint, easier to automate and faster to deploy to.

Here’s a mind-bender for you. Ever heard of Erlang on Xen? Here’s a quote of what it can do:

“On average, only 49ms passes between two moments when the Ling guest kernel is entered and the first Erlang instruction is executed by the virtual machine.” (emphasis added)

Now …

… your eyes skimming to this line took more than those 50 milliseconds. That is human-scale fast. Fast enough a human pushing a button would no longer detect if each button push was handled by a separately started Ling instance.

Where is this trend taking us?

Towards more fine-grained service decomposition, and
Smaller and simpler containers for services to run in

There are of course plenty of caveats. You can only go so far in decomposition and reductionism. There’s some lower limit for container size. They don’t matter overall — at least as arguments go — as we can use extrapolation from these to catch a a glimpse of a future — the future of µ²services — microservices to the second power.

µ²services

µ²services are a logical conclusion of decreasing container size and decreasing deployment unit sizes.

Each µ²service is a pure function with no state running in a virtual machine that is alive only for the duration of the request to the µ²service.

Every invocation of a µ²service results in creation of a separate virtual machine¹, created from scratch and torn down immediately after. This means the path of control flow differs between a “conventional” service and an µ²service — see the figure below.

Comparison of more conventional service implementation (left) and µ²services (right) responding to a GET / request on a REST-styled service with multiple operation endpoints.

On the left the request is first terminated by a load balancer or a reverse proxy on another machine. The application server receives the request and its dispatcher (path mapper) decides which routine to route the request. On a µ²service architecture the dispatcher (path mapper) spawns virtual machines which each runs only a single routine, and these virtual machines act as the endpoints of path routing.

This sounds stupid. <BLINK>Super stupid.</BLINK>

Creating a new virtual machine to separately process each request, alive only a for a few milliseconds seems, nay, is absurdly inefficient.

Yet … for the rest of this post I’ll walk you through for why I think this is not stupid, but instead at least a possible endpoint based on current trends. You’ll be the judge on how likely it is.

(I’ll go through some of the potential implications of µ²services from a technical viewpoint in a later post.)

Co-evolution

There are different drivers and trends that are co-evolving together. The trend of shrinking containers requires automation to realize the benefit of speedier deployments. Microservice architectures and service decomposition trend provide a use case for smaller but more numerous containers and again, decoupling development teams again requires increased use of automation. Finally the introduction of functional into the mix is making the separation of state if not easier, at least cleaner. There is no clear head or tail in this mix — all of these trends are driving each other.

Multiple trends are co-evolving together with feedback cycles. Some concerns such as security are affecting these trends, but they are not as such affected themselves.

Throw security in too, as it favors separation of concerns and role-based access control, which are easier to implement in a loosely coupled, decomposed service with containers with clearly defined boundaries and lifetimes. All in all I think this will drive at least some services to the logical conclusion of:

Minimal containers in both minimal complexity, minimum size and shortest possible lifetime
Minimal fundamental service components, where the fundamental components have no state and separated from each other during run-time

Benefits

Moving from service-per-container to a request-per-container essentially removes sharing. Even with stateless services there is an implicit request-to-request resource sharing of memory, disk, processor and network. Such sharing is a potential problem for security, performance and resource management. Running each request in an isolated, separate container offers several potential benefits:

Increased security
Increased flexibility
Increased reliability
Increased scalability
Increased elasticity
Increased efficiency
Simplified resource management

All of this of course comes with a price to pay. Deployment automation is a must. Hands-on debugging becomes harder. Risk of unexpected emergent behavior increases. Pervasive service monitoring becomes critical.

Yet all of these costs have been paid many times over. They were paid when operations moved from many-services-per-server model to a service-per-server model. To distributed services running on many servers. To virtual servers with ephemeral lifetimes. To having no servers at all, with the code just “running out there”. Concurrently new methods were developed to deploy, monitor and debug.

Thus my argument is that there are environments where the cost of following µ²services model will be outweighted by the benefits it provides. Not in all — most likely only in a minority, but in some.

What is unclear is whether these benefits outweigh the investment cost of developing all the required technology, practices and learning to use it in the first place. Will there be enough motivation to actually realize it? Unknown. As I said, this is a possible future. Computing history is littered with technologies that could have become dominant, but did not.

Replace “virtual machine” with “container” if you will. I’d guess something else entirely, but what? Something in between those two? ↩

Trans-earth networking protocols

2015-01-06T00:00:00+00:00

Assuming you read scifi novels — have you ever stopped, really stopped and thought about how the technology in those stories work? Let’s skip the obvious things that you just have to suspend your belief over like ray guns, faster-than-light travel, space lifts and sentient artificial intelligences.

What I’m talking about is the small stuff. Everyday stuff. Like the Internet and paying for a late-night space-age kebab meal.

That’ll be your tracks from the pub crawl. (Source: Wikimedia Commons)

Pub Crawl at Valles Marineris

So you take the trans-planet express line from Earth to Mars and after a long and thorough pub crawl with your local green-skinned friends you feel peckish, and order the space-age Buck Lightyear premium kebab. You whip out your earthen debit card, chuck it to the reader, enter your pin and

wait

wait some more,

and a lot more

for a total of 40 stomach-grumbling long minutes.

At least this would happen if current EMV protocols are used in the space-age future.

Why? The speed of light is finite. Your chip debit card will talk with the card issuer’s backend systems over the network (actually, it’s the terminal that does the talking, and that doesn’t talk directly to the issuer but to a … well, that’s just details) so that the kebab vendor will get a confirmation they’ve been reserved the cost of the meal by the issuer, and the customer’s bank balance (or credit) is valid.

This information needs to travel back and forth between your Earthen card issuer and the Martian card terminal. Distance between Earth and Mars varies between 3 and 20 minutes depending on their relative orbital positions so just one round-trip — request and response — will take 6 minutes in the best case and over 40 minutes in the worst case.

In case you’re thinking BitCoins … they won’t work either. Transactions are asynchronous, yes, but you’d still need to send one over the high-latency link, wait for the transaction to complete, and of course wait for the new transaction block to be sent to you. «grumble grumble» says your stomach.

This pretty much means that any kind of online protocol is not going to work in space, except if both endpoints are really close (Earth to Moon, Mars to Phobos and Deimos).

Even if you are patient and willing to wait for hours for a transaction to clear, most today’s network services have timeouts (connection timeouts, nonce validity timeouts etc.) that will prevent whatever you are trying to do from completing if endpoints are separated by distances of light minutes.

Is it going to be IPv6? For what I know there are no limitations in the IPv6 itself that prevent Earth-Mars Internet from working. TCP has a fixed maximum segment lifetime of 2 minutes, but this is easily circumvented with packet reassembly at trans-planet gateways. More critical are applications and protocols that set limits within themselves. Of course many IP-based protocols are very chatty and synchronous, neither being good for very long latency links.

This would also mean you can’t post those boozing pictures on Twitter, either.

The card should clear immediately!

And so it will. But the clearing protocol will not be based on the current model. Your card issuer has probably pre-reserved a portion of your balance or credit and “transferred” it over to a local Martian operator, and this local balance would be balanced between Earth and Mars “behind the scenes”, asynchronously.

Twitter, Google? Probably twitter.com and google.com map to ~~geo-located~~ planet-located IP addresses and these services are set up to do long-haul asynchronous synchronization on their (relevant) data sets between Earth and Mars.

There are other ways, of course. These are just examples to show that it is possible to at least generate an illusion of network service ubiquity even over planetary distances.

So, doable.

But not directly.

But it’s the far future, why worry now?

When eventually we’ve transitioned from IPv4 to IPv6 do you really think it will be EVER UPGRADED AGAIN?

Absolutely no. No, no no and no.

This is one prediction I’ll put down.

IPv6 will not be replaced within my lifetime.

It will be extended and expanded with new options and potentially other minor backwards-compatible (as with IPv4) changes, but fundamentally, current-day IPv6 will be the internet protocol even when we’re building outposts and colonies on the Moon and on Mars.

My point is that some portion of internet protocol choices made today are going to be around much, much later. IPv4 is 35 years old now, and not going anywhere in a hurry. IPv6 in 2100? Highly probable.

Challenge of the future

Internet use has gone through several phases, each with different assumptions, starting from early constantly connected and centrally operated (wired networks only, most users had only one or few “connection points”) to current intermittently connected model (assumptions of multiple location-fluid devices with variable connectivity).

All previous and current models have an implicit assumption of a small latency of less than a few seconds. There may be sometimes congestion leading to temporary latency increases, but more or less we’ve lived under the knowledge that a network packet can traverse to the farthest end of our planet within a fraction of a second. It will — it’s a fact of physics.

In a potential future with spread of human populations to different moons and planets this latency assumption will work only locally. That’s a world where service designers have to tackle yet another problem: how to provide good service when network latencies are minutes or hours.

Though, this is not a problem that should keep anyone awake at night.

Back to the kebab

I referred to scifi books and our hidden assumptions of the world. A good scifi writer will not get bogged down by thinking how today’s technology would not work in the future. Good story it makes not.

Technology and its improvement doesn’t work like that way. We can’t ignore laws of physics for the sake of a good story. Similarly there is a human imperative (scientific, commercial or out of curiosity) to make these things work.

I’ll be waiting for the first Twitter post from Mars base.

Year of the Cloud?

2015-01-03T00:00:00+00:00

It is the time of the year that people are trying to fill the intellectual void caused by overabudance of eggnog, glögg and premium chocolate by producing predictions for the coming year 2015. I’m not much of a seer, so instead I’ll take a look at the past year 2014 and produce some probably blindingly obvious tautologies.

A lot happened in 2014 in the cloud market. This happened, then and also. Docker this, docker that. Also, AWS, Azure and Google announced quite a lot of new features, services, bells and whistles.

It is also interesting to see that cloud vendors are willing to cut their prices again and again and again and again. I remember someone commenting on Twitter at re:Invent 2012 about AWS’s price reductions, like, that a 30% price drop is not normal in a competed market. Or similar.

I took that to mean that in a mature competed market like electricity, seeing major price reductions on the baseline price is not possible because there are no such margins in already-furiously-competed prices. Just the fact that cuts are happening means the market has not matured into a (semi)stable state.

That’s not much of a news in itself — “the cloud” is a shift in the landscape itself and only a fool would expect stability right now.

Similarly only a fool would expect cloud market evolution to be a re-run of something that has happened before. Expecting the past to be a play of the future is bound to fail.

The problem I see in 2014 is that it bought faster and faster chaos and complexity into the cloud market. I’ll try to explain:

Chaos is unpredictability, not randomness. All of the cloud vendors are what economics call rational players in the market and try to make optimal choices at any point of time. But this decision-making process is not visible to customers, so although customers and we, the Internet pundits can find post hoc narrative to all of these events they remain, at heart, unpredictable to us.

It is easy to predict that AWS, Google and Microsoft would drop their prices in 2015. So what? That’s just extrapolating past into future. Try predicting instead how much, how many times and when those price drops will occur. And who would be brave enough to predict that no price cuts would occur? Or even price increases? Anyone? Volunteers? (See here.)

Complexity … it is not enough that there are more cloud services, and these are more and more complicated but there are also more and more interactions between different systems leading to complexity. Complexity in turn easily leads to a priori unpredictable emergent and even chaotic behavior — again a problem especially for larger enterprises, but also for nimbler companies overreaching their capacity to handle emergent surprises.

Rapid change, chaos and complexity can cause havoc even before a single line of code is laid out. Analysis paralysis is always a real risk, with new stuff continuously popping into existence, potentially invalidating prior analyses is not helping planning-oriented organizations. Strategically, if you are not keeping an eye on the landscape your earlier assumptions about technological barriers of entry could be invalidated catching you unawares, hurting or obliterating your business case.

Usually you’d want to take a step up the abstraction layer when this kind of rapid chaotic evolution occurs to allow one to maintain an overview without losing too much of the necessary details, but I’m not sure it is yet possible. At least I feel not. I don’t personally have a useful abstraction at hand yet.

I’m seeing fragmentation of competence and skills for cloud consultants and engineers in 2015. At least for myself, as a sort of generalist bridging engineering and business strategy I see difficult choices ahead. Should I choose specialization into some part of the cloud landscape (technology or business-wise), or raising in the abstraction level?

Unfortunately the specialization path feels uncomfortably like 90’s fragmentation into different client-server camps and 00’s fragmentation into different web service full stack compositions. I didn’t like either of those when they occurred, mostly as it led to entrenched us-or-them positioning in the competence market. (Hardly benefiting customers.) Worst of all, now there’s a possibility of this fragmentation occurring within cloud service catalogues — Lambda vs. Beanstalk battle anyone? Even when AWS positions these as complementary, the growing service portfolio makes it harder and harder to have generalists on staff, leading (human nature and all that) into different specialization camps fighting for their viewpoints.

Perhaps a take-home message of that thought line is that cloud competence management increases in importance in teams when cloud is part of company strategy.

Earlier I noted that the abstraction-raising path does not seem to be open. I could ditch the technology and move purely into cloud business strategy level but that’s not what I mean with “going up in abstractions”. It’s a different business level with different abstractions, it is not a new abstraction for the technology layer that I’m looking for.

Personally I love working at two levels simultanously, both at the business and the technology level. It’s often difficult and challenging position, being judged by both camps, but also very rewarding because you can help customers find solutions that are beneficial win-win scenarios for all camps and business lines within the organization. And this of course within the context of cloud computing for which I have both personal, professional and scientific interest.

Yet it somehow feels there’s a pier and a ship, and I have a leg on both, and the ship is starting to drift off.

Sauna

2014-12-23T00:00:00+00:00

It’s been a while since my last blog post — not that I don’t have many ideas, but too little time — and I decided to do something different this time. A deviation from the very technical blog posts I normally do.

I am going to talk about Finnish sauna and its significance in Finnish culture. This is also a topical writing since now is the eve of eve of Christmas (“aatonaatto” in finnish, well the publish time might shift to eve, but that’s not when I’m writing this) as sauna is also a very important part of Finnish Christmas.

Also, I’m not even trying to do a logical blog post. Instead I’ll opt to just write about many topics, one topic at a time, without necessarily any kind of cohesive story between the topics. I’ll also mostly skip the history of sauna — feel free to browse the net or look into books and research articles yourself.

Words

Sauna is the place where there’s a kiuas, which is a stove full of exposed rocks, and is used to heat the sauna (methods of heating vary). There are raised wooden benches within the sauna to sit on.

Saunoa is the act of being in a heated sauna for the purpose of … well, being in a hot sauna. (If you sit in a cold sauna you are just being an ass.)

Löyly is the result of throwing water on the hot rocks in the kiuas. Yep, löyly is essentially 100+℃ steam resulting from instant vaporization of water when it meets rocks heated to several hundred degrees centigrade.

Types of saunas

Electrically heated sauna (e.g. the kiuas has electric heating elements within) is the most common. It is pretty difficult to use any other kind of heating method in modern apartments for various strange reasons such as fire safety and ventilation, so, it’s the most common.

Note that an electrically heated sauna is considered also to be the worst of all of the options. (But it’s like sex — a bad sauna is way better than having no sauna at all.)

Wood-heated sauna is. Well. The rocks are heated using burned wood. Pretty self-explanatory, I think.

As a kid I used to heat up wood-heated kiuas at my parent’s summer cottage until I got banned from doing so. Why? I repeatedly heated the kiuas so much that the topmost rocks were glowing dull red and the bottom ones were glowing white. My father didn’t take that kind of fire hazard lightly. (Electrically heated kiuas don’t get that hot, they have temperature regulators and heat-sensitive fuses.)

Savusauna is an older form of wood-heated sauna which has no chimney. Wood results in smoke, and no chimney plus smoke equals smoke in the whole sauna. This may sound like a bit crazy, but the real secret is that this is definitely the most super extra best type of sauna experience you can have. (Why these are not common? See next.)

Burned-down sauna is another type of traditional wood-fired sauna. Hey, think about it. You have a fire, which heats rocks, which can get very hot, and heating a sauna takes anything from 30 minutes to a few hours depending on the type which means that it is going to be left unattended. Fire hazard, anyone?

There are very few old historical saunas in Finland. Wood-fired saunas just do not last. It is practically a tradition in Finland to burn down a sauna every now and then. (Which is also another reason why wood-heated saunas are usually located in detached buildings.)

Who go to a sauna

Everybody.

I mean, in the demographical sense. “Liking” sauna is a continuum and not even all finns like or go to sauna. But apart from this funny little minority everybody in Finland go to a sauna more or less regularly.

Like, the president, adults, children, teens, retirees, very very old people, people with heart conditions, pregnant women, men, women, mythical beasts … everybody!

My own children started getting into sauna about as soon as they could crawl (yes, getting into, not taken into). Typically small children stay in for shorter periods of time and/or sit on lower benches, but still, even babies often go to a sauna in Finland. Even my 4 year old kid is sitting on the topmost bench nowadays.

(If you haven’t been into a sauna, and ever get a chance, keep that in your mind: 4 year old kid, 70+℃ sauna, topmost bench — don’t you dare to shirk.)

Mixed gender saunas and sex

True and not true.

First, a sauna in Finnish tradition is a place of cleanliness and purification. No sex in sauna, please (excluding the caveats, of course). Also there’s hardly any stud that can do the required hydraulic moves while the environment is 70+℃ without risking a heart attack. Also staring at any dangling things of other people is just as bad manners in sauna as anywhere else.

Secondly, yes, there are mixed gender saunas. Families typically go to saunas together — yep, Finnish kids will see their parents and siblings naked regularly, probably one of the reasons there’s less hangups around here about nudity than in some other places. (But half of that is probably generic Nordic mindset, Swedes and Norwegians don’t have as much saunas and they’re pretty relaxed as well.)

In non-familial situations such as with friends or colleagues having a sauna it is possible to have women-only and mixed shifts. Which means that if you are a foreign female, it’s okay to ask for a women-only shift as it is considered rude to force women to a mixed-gender sauna. But don’t assume that separate sauna shifts are the default as most finns just do not see sauna as a place of sexuality and it is just as common to default having mixed gender saunas in a company of friends.

After a day’s worth of skydiving I went with other from our club to a sauna of a bowling club. Being skydivers, naturally trying to do stupid things we wanted to see how many people we could ~~fit~~ cram into the sauna. 10. 20. 30. 40. I don’t remember exactly, but somewhere between 40 and 50 people the bench supports failed. This was a sauna with normal capacity of 15 people, so space was … let’s say, heavily optimized. And it was a mixed gender sauna. Would this have been possible somewhere else?

Where are saunas?

Everywhere in Finland.

Houses. Almost 100% of single-family houses have at least one sauna. A significant portion of them have an internal electric sauna and a separate detached wood-heated one.
Apartments. Most new apartments have a per-apartment sauna, and those that don’t have a shared sauna in the building available for residents on a reservation or a schedule basis.
Summer cottages. Why have a summer cottage without a sauna in Finland? Madness.
Office buildings. Yes, most office buildings have a edustussauna (“sauna for promotional purposes”) that is available for companies located in the office building.
Schools (well, not the modern ones, but older ones yes.)
Well-equipped gyms.
and many other places

Way back I worked for a summer job at a construction company. One of the places I worked at was the sauna level of the Neste headquarters building (nowadays Fortum). This one actually. The two saunas — executive and a pleeb one — in the building were at the very topmost floor. What a great view! On a clear day I could see Tallinn over the Gulf of Finland.

Why go to a sauna?

On an individual level,

It is relaxing. Veeeeerrrry nice for aching muscles.
After a while in sauna the dead skin layer gets soft and you can just scrape the topmost dirty layer off. It does feel very nice afterwards. Some people attribute the skin quality of Finnish girls to sauna, but I’m not so sure. (Or it doesn’t work with males at all.)
A kova löyly (throwing a lot of water on the stones for extra hot löyly) makes the skin prickle in the heat. It feels like your skin is burning, but if you just persevere a little … it’s not actually going to burn … your body will release a hit of endorphins.

You know endorphins? Body’s own opioids. Feelgood. Veryfeelgood.
Especially in wintertime staying outdoors often makes you feel cold. I guess if you are like, a canadian, you know what I mean. Not the cold where you are freezing to death (literally), but the kind where you are not really cold, but still somehow your bones are feeling the cold.

Solution: sauna. After outdoors I ask the kids “want to go to sauna to warm up?” the answer is “yay!”.
Sitting outdoors after a sauna. See below.

It’s not necessarily all of these. Sometimes I heat up the sauna because I didn’t stretch too well after a workout and the next day I got aches. Then I just strech down on the bench in the warmth and not even throw water on the kiuas.

On a group level, going to a sauna is often a social occasion. “Let’s have a sauna” is a potential substitute in Finland between friends to “let’s go have a pint”. Social events are often arranged as sauna events (often with food and drinks, and actually going to the sauna is not compulsory, so if you ever get an invitation to a company sauna event it’s okay to actually not go into the sauna).

apres-sauna

Sitting outdoors, on a bench, with a drink in hand, clothed only in a towel, after a sauna is a trope of Finnish sauna culture. You do that when:

It’s a warm evening in the summer.
It’s a freezing night in the winter. (And anywhere in between.)

Sounds crazy? It’s not, for two reasons. The empirical reason is that in both cases you’ll get dry. I mean, really dry, pretty fast, as if you had gone through a dryer. The best part of this is that even in the freezing winter it takes quite a lot of time until you’ll start feeling the cold. Second? Physics.

Physics alert! When you move from sauna to a colder environment (both -20℃ winter night or +20℃ summer evening is significantly colder than sauna) your skin will be both a) more moist than the environment (humidity of sauna, your perspiration .. you showered after sauna too, right?) and b) way warmer than the environment (warmer than normal skin temperature, too).

There’s a temperature gradient between your skin and the air leading to a convection. This circulation of air means that the ambient air that moves close to your skin it will heat up and its relative humidity will decrease. This low-humidity air will draw moisture out of skin surface. After that it is a race whether water on your skin will evaporate faster than epidermis will get cold…

The result is that you can go from a sauna to a -20℃ outdoors and sit comfortably, sipping a beer, for several minutes until you feel any kind of cold. (Provided it is not windy and you don’t have to rest your feet on a cold surface. Feet, cold, bad.)

Regardless, it is a pleasurable experience.

Sauna etiquette

Shower or take a dip in the lake before you go into sauna.

Don’t wear swimsuit in a sauna. If you do you’ll get a funny look, but being a foreigner you’ll be excuded. Still, it’s not proper etiquette.

If you throw water on kiuas, you don’t get out before others or until it has cooled down. It’s the principle of “you caused it, you suffer it.”

There are a lot more etiquette rules, but those are quite a bit more nuanced. Probably in the category of Japanese tea seremony — only a few major rules, but a lifetime to master wholly.

Christmas sauna

Finns take a sauna on Christmas eve. Whether this is a christian tradition (you know, purification for receiving the birth of the Christ) or something even older is not something I want really to comment on. It is a tradition. It was a tradition in my childhood home, in my grandmother’s home, and several generations before that, and I’m carrying it on myself.

I mentioned earlier that not all apartments have saunas. With shared saunas special holidays are usually handled so that there is a common saunas for all residents (men and women shifts separated) on most important holidays such as Christmas eve and midsummer’s eve (sometimes also easter).

Finns abroad

Way over early noughts I was helping a Finnish friend couple on their move at Stanford, California. After several hours of lugging furniture, boxes and other paraphelia me and the him of the couple were sweaty and decided to visit the apartment complex’s gym which supposedly had a sauna.

We found the sauna. Typical US warnings at the door “don’t go into sauna if you have freckles, crooked teeth, dry skin or general stupidity and most definitely don’t throw water on the rocks as it might cause a meltdown of the earth’s core” or something similar, we didn’t read too carefully.

We fashioned a water bucket out of a tissue container. Threw water on the rocks. Ran out — there was two decades worth of dust collected on the rocks of the kiuas which rose into the air as we threw the first batch of water and not even our lungs could take that.

(We waited and did a few iterations of throw-water-get-out until the air was sufficiently clean to continue normal Finnish sauna operations. Nothing can keep a finn from a sauna experience.)

Heat

Common sauna dry air temperature is somewhere from 70 to 90 degrees centigrade. It is possible to have 100-120℃ dry air temperature, although it gets a little harsh (not because of the air temperature, but because it requires a very hot kiuas, and a very hot kiuas results in a very harsh löyly, which generally is not preferable).

Don’t be alarmed, you won’t boil in a 110℃ sauna. Dry air is a bad conductor of heat. The same principle makes tropics feel much more stuffy than dry deserts.

Turingin testi

2014-09-20T00:00:00+00:00

This is a short poem I wrote in 2011 as a course assignment asking to write a short essay on Turing’s test (though it was not supposed to be in prose). It is in finnish, so if you don’t know finnish, sorry. I’m bad a writing poetry even in finnish so I will not make an attempt to translate as it would lose some of the rich nuances of phrase that finnish as a language offers that I’m unable to reproduce in any other language.

However this is a piece that I’m actually quite proud of. Not as a professional writer, but as a computer scientist and a software developer.

Tämä on runo jonka kirjoitin vuonna 2011 osana kurssisuoritusta (582102 Johdatus tietojenkäsittelytieteeseen Helsingin Yliopiston tietojenkäsittelytieteen laitoksella, jos aivan välttämättä haluatte tietää). Oikeasti tehtävässä edellytettiin esseetä Turingin testistä, mutta tässä kohtaa minulta pursui höyryäviä tekstintekeleitä korvista ulos joten päätin poiketa kaavasta oman mielenterveyteni vuoksi.

Miksi julkaisen tämän? Kahdesta syystä.

Ensimmäiseksi olen itse omalla tavallani ylpeä tuotoksestani. En ole mikään erityinen tai edes keskinkertainen tekstinikkari, mutta tässä tunnen saavuttaneeni jotain paljon parempaa kuin mihin normaalisti kykenen. Eli vaikka satasen normaalisti juoksisi 20 sekunnissa ja on päässyt vain kerran 15 sekuntiin, on se silti jotain mistä voi olla henkilökohtaisena suorituksena ylpeä. Kaikessa ei ole pakko verrata itseään maailman huippuihin.

Toiseksi, haluan sanoa että elämä ei ole vakavaa. Pidä hauskaa! Revittele! Tee asioita eri tavalla! Varman päälle pelaaja onnistuu vähemmän. Ja tämän kirjoittaminen oli oikeasti hykerryttävän hauskaa, vuosien jälkeenkin sen kirjoittamisen aikaansaaman mielihyvän pelkkä muisteleminenkin saa minulle hymyn korvasta korvaan ja vatsaan lämpimän pöhinän.

Turingin testi

Oli kone H, jolla oli ongelma. Miljoonittain muistia, tuhansia ja tuhansia tiedonmurusia, laskenta sukkela kuin salama, mutta silti ei kukaan usko että H osaisi ajatella.

Ken näkee koneen H tietää heti mitä se on: Tinaa ja kuparia, muovia ja kumia, arvometalleja harvinaisia, ei ollenkaan pehmeää ja limaista. ”Ei kone ajatella osaa” sanovat, miete naurettavakin on.

”Ajatus vain pehmeässä aivomassassa orgaanisessa asua voi.” Ei auta vaikka käheäksi H koneäänensä kuluttaa todistaessaan tietojaan, päättelykykyään, älykkyyttään – ei vakuutu kukaan. ”Kone olet, ihmisen ohjelmaa toistat vain, hänen älyään peilaat lain!”

Kääntyy herra A haudassaan, haamuna sieltä kohoaa ja näin konetta jututtaa: ”Laskenta ja limaiset aivot, molemmat samassa maailmassa majaavat. Fysiikka kummankin ekvivalenttia on, molemmissa sähköä, elektronista tahi kemiallista - ei periaatteellista eroa suuren suurea.”

”Epäreilua on sun muotoa metallista arvioida ja nauraa, eihän älykkyys asu enemmän pitkässä sen paremmin kuin lyhyenlännässä, miksei siis kuparissa ja tinassa myös?”

”Siksi kokeen mä järjestän, teletyypillä teidät eristän, saa ihminen paperitulostetta tihrustaa, näppäimistöä nakuttaa, ei näe, ei kuule sua, ei näe, ei kuule mua, ei tiedä kumpaa jututtaa kun printterin raksutukselta molemmat vain kuulostaa.”

Nyt ei naura ihminen, kun herra A vakuuta älyllään ei, vaan kone H briljeeraa vain, kokeessa väärän valitsi hän, häpeissään pois luikki, kun koneen ihmiseksi julisti.

Jäi silti koneen H mieleen tää, miksi ihmistä yritän matkia mä, ei ihminen täydellinen ole, virheitä tekee, muisti pätkii, logiikka heikkoa on, virheellisiin syllogismeihin ratkee.

”Olenko älykäs oikeasti, vaikka siltä näyttäisinkin? Mitä äly on, tietoisuus? Jäi mulle ongelma siis, eksistentiaalinen probleema L uus!”

What's in your AWS bill?

2014-09-12T00:00:00+00:00

Two nights ago there was a #DevOps meetup where I had a talk about AWS cost management. If you missed the event (slides) please come to the next Helsinki AWS User’s Group meeting on October 9th where I’m going to do better-and-improved version of the DevOps talk. (I wasn’t entirely happy with my own presentation, and I’ll try to improve it for the next one. Especially I’ll try to avoid knitting blankets for destitute devops people.)

Anyway, I wanted to muse on a few observations I made at the meet. First of all, there were quite a lot of people, over 70 or so. There’s clearly a lot of interest into devops in the Helsinki area. So during my part, to get a feeling of relevancy and for future focus I asked the audience a few questions:

How many use AWS?
How many knew their (personal, project’s, company’s — whatever is relevant) previous bill’s bottom line?
How many knew their current bill within some reasonable accuracy?
How many could forecast current month’s final AWS usage costs?

For these I got about 1) 50%, 2) 20%, 3) 10% and 4) 25% of audience. I assume that last on was higher than third one because quite many assumed (knew?) that their AWS bill for the current month would be zero dollars. I’ll rephrase that last one next time differently to get an idea of how many of those who actually use AWS out of the free tier are doing cost forecasting.

I find it surprising that so many devops people actively using AWS apparently were ignorant of their current status as well as AWS account and billing basics (only a few had seen the monthly PDF invoice, or knew about consolidated billing). Given that devops philosophy tries to automate a lot of things, to give a lot of freedom and responsibility to devops people and that any kind of rapid-turnaround automated deployment to AWS does have potential for large cost SNAFUs it really is a little disconcerting to see so few people being even reactive (minimum risk management) about their operating costs.

Note: Those who I knew to be from companies with over $10k/month AWS accounts were quite knowledgeable of these facts. I’m not sure whether this is cause or effect, though — are they using AWS because they felt comfortable with it financially, or did that knowledge come out of necessity when increasing AWS invoicing attracted attention of their financial controller? Gotta ask that.

I know that these please-raise-your-hands polls are not statistically or scientifically robust, so this might just have been a bad sample. Even if the goal would to have proactive cost management, it is likely that at least in the beginning only a few people in the team (those whose neck is the most visible to management) are concerned about costs. Maybe those people were underrepresented that night.

Perhaps while the benefit of flexibility, elasticity and scale of cloud services has percolated up organizational chains, the financial impact of those hasn’t. So to get the message straight, for anyone from management coming here I want to make out two clear statements for everybody’s future benefit:

There are possibilities for financial surprises when using infrastructure cloud services (like AWS). Keep your eyes open.
There exist mechanisms that can be effectively used to monitor, forecast and mitigate these risks, allowing organizations to use the cloud while managing its inherent financial and operational risks. Don’t be frightened.

About a year ago I wrote about how the cloud changes distribution of risks compared with the “old way” of acquiring IT services. This is exactly the same thing. Risks previously implicitly “outsourced” are now explicit, and by recognizing them as such they do become manageable.

Oh, last but not least. I did not catch a single person admitting they are using AWS spot instances. Frankly I wasn’t excepting many — but really, zero?

Cornucopia machine as a service

2014-07-17T00:00:00+00:00

I’m currently reading Cloudonomics by Joe Weinman. I frequently find myself nodding in approval as his conclusions either parallel mine or provide deeper insight into what I’ve seen when working with and consulting customers on “the cloud”. I’m not through the book yet, but I do like his insightful and deeply thought out arguments and counterarguments on what cloud is or is not and how it changes or might not change companies. Sometimes I mentally argue against Weinman’s conclusions but that’s a good thing — the book makes me think about my own assumptions and views.

Regardless I’m not going to review this book. Go search the Internet or read it yourself for that. I’ve got something else.

The opening of chapter 10 got me thinking. Weinman’s going through whether “cloud is like electricity” or “cloud is not like electricity” and what these differences are — also why we should not extend the “electricity utility” metaphor too much.

Anyways, I’ll quote a short passage (actually a recursive quote, since it in itself is a quote):

Security issues in the cloud are very different: As Brynjolfsson quipped “No regulatory or law enforcement body will audit a company’s electrons.”

That got me thinking. (If you want the full argument why electrons and CPU cycles are different, read the book.)

So today’s companies cannot differentiate by the use of computing, but by how it is used. That is, business practices and models, which are in turn encoded in software. (If the business model requires scalability, then scalable software.)

What does this computing make (cloud or no cloud)?

To me it sounds a lot like grey goo aka programmable matter aka utility fog. Okay, computing operates in a virtual reality instead of physical, but apart from that tiny teeny little difference the basic idea is the same: both can be programmed to do anything (within some limitations, within their realm).

Then … If there ever was a cornucopia machine (a nanomachine assembler from Singularity Sky by Charles Stross) then what would that mean to businesses?

In the real world some of my purchases are for convenience: instead of cooking a meal, I eat out. I can cook. I buy a soft drink — I believe I could look up a recipe, just-and-just construct a CO2 injection system and if I found some limestone, might even be able to generate CO2 and liquefy it, but hey, why? (Of course, where would I get the tubing and machinery to do those… see the toaster project on where this train of thought would lead you.)

Yet while a soft drink might be a borderline case, most of my purchases are purchases not because of convenience but by necessity. I do not have the skills or the resources needed to construct a cell phone or a computer. Then think how a cornucopia machine would change that. Recipe for cell phone + cornucopia machine = new physical cell phone. (Next exercise: think how it will be prevented by legislation and control mechanisms mostly not because of safety concerns, but because of protection of “intellectual property”. Just like copyright extensions to protect certain exclusive exploitation rights to a big-eared cartoon mouse.)

Given the proliferation of open source components, cloud services and software production and operating knowledge (high scalability, stack overflow etc.) there is less and less “secret sauce” in how software operates. The difference is more and more in how software is operated.

Just think all of this, including the cloud, as the equivalent of a cornucopia machine for virtual reality. You literally can create a load balancer or a scalable web server cluster out of thin (virtual) air, with publicly available recipes.

Then what’s the big conclusion?

None. I just think there are some conceptual (surprising?) similarities between the fictional cornucopia machine (smart matter) concept and now-and-current cloud computing.

Or maybe this: what if real live physical cornucopia machines existed, how would that affect your business?

Devastate? No effect? Then try the same with a virtual cornucopia machine, one that can re-create any computing infrastructure from a recipe. Including yours.

(Note: I’m using the term “infrastructure” in a broad sense here. I think that any computing service that does not operate in the “core” of the service, or in the left side of a Wardley map is just a supporting element — infrastructure. This is a distinction between “infrastructure” when discussing business vs. technology. In the business sense Salesforce is infrastructure but in the technology sense it is software-as-a-service, not infrastructure-as-a-service. Yes, confusion.)

Of course, an upstart competitor probably will not copy a cranky old HR or ERM system but will opt for easier to use, easier to deploy cloud service versions. So instead of copying your computing infrastructure as-is, think it as someone else making a better copy of it.

Anyway it is summer and I don’t think I have any need to be consistent in this post. Just thinking aloud, pies in the sky etc. etc. :-)

AWS Service Metadata

2014-06-23T00:00:00+00:00

For about 4–5 days I’ve been working through AWS’s news announcements, forum posts, digging through history with The Wayback Machine with the single goal of sorting out

when an AWS service became available, and
how many zones are available at any point in time.

I need this data for the work I’m doing on AWS service availability (see also here) for the simple reason that any pro-rated availability estimate will be impossible to calculate useless unless you know for how many service-hours a given service is actually available. Since not all services are available in all regions at the same time (or not at all) and some services expose the underlying availability zone (AZ) structure I just have to get those values.

Unfortunately there doesn’t seem — at least I couldn’t find — any public dataset that contains all of this data in a well-researched format (or at all, for that matter). With “researched” I mean having a rationale for each data item in the dataset that can be tracked back and re-checked from original sources (or archived copies of those) if needed.

I got the dataset done and since although it is critical in my research, I really only have need for it once. So I decided to share it with everybody and put the dataset available under CC BY 4.0 license at https://github.com/santtu/cloud-meta. I hope someone will find it useful in their work or research.

Since a blog post without a graph would do, here’s a graph showing number of AWS services, regions and zones from the introduction of Alexa in 2004 up to a few days ago:

You’ll find the original data here.

"Previous generation instance types"

2014-06-16T00:00:00+00:00

Just recently I noticed that AWS had removed most first-generation instance types from its instance type description page. Digging back in history you can find Jeff Barr’s post from April 15th describing this change (you can double-check using the Internet Archive that it occurred after April 13th). (How did I miss that for two whole months?) I started then thinking about how this relates to my earlier thoughts on AWS instance type retirement.

I drew a doodle as a help to thinking about various known and apparent things and their relations to underlying realities. I’ve reproduced it below. Why? Because I know a picture in the beginning of a blog post will keep readers engaged a bit more. Did you even read the previous sentence? I bet half of you skipped the second sentence and decided to go straight to the picture. Which is a bit of a mess and isn’t terribly coherent even after I’ve tried explaining it later.

First of all note that the change at this time was purely cosmetic as AWS did not deprecate any instance types. If you are looking for m1.medium please check the “previous generation instances” page.

Let’s start with a few quick facts and observations (top part of the graph):

No instance types were deprecated
No more explicit numerical generation numbers, only relative (“current” and “previous” vs. “second generation” as in m3 class announcement)
Current generation instance types conform to Intel’s “Powered by Intel Cloud Technology” program (all but three)
m1.small is listed as a current generation instance (but otherwise gets minimal screen space)
“[AWS has] no current plans to deprecate any of the [previous generation] instances” (source)
Pricing strongly favors customers picking current generation instance types — AWS’s own communication is also very direct in pushing customers to use newer instance types

A couple of deeper thoughts then:

No numeric instance generations. When “second generation” instances were originally introduced it made sense to market them as newer, better and superior to “first generation” instances. Yet the whole concept of distinct “hardware generations” did not make much of a sense even then. What are main customer-visible differentiating features between these? What would a third generation instance be like? Fourth?

For a customer what matters are capabilities. For instance types these have always been an unorthonogal bunch and will remain so, which numerical generations does not clarify even one bit. They are superfluous.

Good riddance, I say.

m1.small still holding on. The on-demand prices from lowest upwards are: $0.020 for t1.micro, $0.044 for m1.small and $0.070 for m3.medium.

It might make sense to introduce m3.small to replace m1.small in the gap between smallest (t1.micro) and the lowest-powered modern instance (m3.medium). But this can’t be done. Why? Part of the reason is that m1.small is an accident of history and is very difficult to replace.

AWS has three classes of CPU scheduling (year introduced in parenthesis):

Fixed (2006). m1.small CPUs are 50% shared between other m1.small instances. An eight-core machine can host 16 m1.small instances running each having one virtual CPU at about 50% of full Xeon core performance.
Dedicated (2007). Each virtual core is assigned to one physical core. This was introduced with m1.large and m1.xlarge and is used for all but two instance types.
Variable (2010). t1.micro is the only example of this type of CPU scheduling. Instances share CPUs with others but the allocation changes dynamically.

All but m1.small and t1.micro use assign each virtual CPU to a dedicated physical CPU core.

There are many good reasons to avoid CPU sharing which is why I believe all new instance classes will only use dedicated CPU assignment. However, since m3.medium already has one virtual CPU there is no way to create decrease CPU count to create a smaller instance than m3.medium except with CPU sharing. Which I assumed would not happen. Reductio ad absurdum, thus no m3.small.

There could be t1.small, though. This is because the whole t1 class is really an odd one out. I’m not sure what it is. Was it introduced as a way to satisfy cheapskate customers? Or is it a way to get life out of older (repurposed) hardware? Or something else? It is useful, though, for running infrequently active, mostly dormant servers. Make t1.small subject to the same bursty CPU behavior as t1.micro, but with more oomph (shorter penalty box time). That way nobody would be fooled into thinking that it’s a decent replacement for a constant-work server, but it would still be a good replacement for m1.small.

(Burstiness isn’t that bad since nobody should be running CPU-bound jobs in m1.small either as m3.medium offers 3× performance for 2× the cost. The only real reason to use t1.micro or m1.small is when you need an always-on, infrequently used server, and the only reason to pick m1.small over t1.micro is either 1) really need more memory or 2) really need a little more long-running oomph from the CPU.)

No current plans for deprecation. Yeah, and pigs fly.

Let’s be realistic. AWS might not have yet a schedule for deprecation, but I think someone should get their asses fired if there are no deprecation plans mapped out. AWS might now just be sounding out customer reactions to the current vs. previous generation marketing message change before deciding on the deprecation plan out of a few choices planned out. But plans there are, assure I you.

Surely, someday, after a lot of generic modification and 3D-printing of retro pilot glasses even pigs can fly! In style! (Image credit: Larry Wentzel, used under Creative Commons license).

Miscellaneous. AWS is a business. It’ll keep “previous generation instance types” around as long as it makes business sense. Conversely, they’ll be EOL’ed the day they don’t make business sense.

OTOH, “business sense” isn’t clear-cut and aging hardware especially makes it complicated to evaluate. On one hand old hardware is likely to be fully depreciated, so it’s all operating profit. As long as it actually makes money. On the other hand aging hardware breaks down more often (it’ll hit the end of the bathtub curve of reliability), it might not work so well with more streamlined management systems and maintenance processes and most of all, demand for it might just drop. And you can’t get “new old hardware” so any long-term plans cannot rest on “old stuff” anyway.

Just understand that current instance types will eventually be deprecated/retired. Separating “current” and “previous” instance types is a clear step towards having a clear lifecycle for instance types, from introduction to end-of-life status.

Which is a good thing.

Structure and interpretation of AWS service health dashboard messages

2014-06-13T00:00:00+00:00

For the past three months I’ve been busy and haven’t had much of a time to write new blog posts. If you’re expecting more EC2 spot instance analysis you have to wait some more, sorry. Instead I want to share some results from one of the things I’ve been up to for the last three or so months.

I’ve been analyzing AWS service health dashboard messages — a whole lot of them. Have you ever been to the AWS dashboard? In short it is a place where AWS publishes information about events that affect their services. This data is accessible via the web page itself, but also as multiple RSS feeds (there’s also JSON data, but it is internal API, subject to changes and doesn’t have as good incident history record as RSS versions).

This is what the AWS service health dashboard’s history section looks like. Most of the time it’s very boring reading, all green checkmarks.

TL;DR

It is interesting to look at what AWS publishes in the service dashboard. For ADHD and TL;DR and PPRT readers out there findings first:

There’s no knowledge of what AWS actually publishes in the dashboard. Are all outages reported?
Incident descriptions are written by humans and meant to be read by humans.

In this post I won’t go into any kind of analysis of outage events, instead I’ll just focus on what common patterns and features these AWS service health dashboard messages share. I’ll get to outage analysis later (I think).

The longer version

First of all, I haven’t found any definition about when an incident warrants publishing a message in the dashboard. It seems to be along the line of “large scale”, “affecting multiple customers” or “externally visible” but that is solely based on observation and not on any statement from AWS.

Simple transient single-point failures are not reported — a server failure is not covered neither are any other failures that are transparently handled by a high availability mechanism (making them mostly invisible to the customer). As an example a failure of an ELB host or a networking component with failover capability may appear as momentary connection terminations or decreased performance, but these are hard to detect over the background level of “normal” failures from gazillion other causes.

It might be that AWS dutifully publishes every incident they or their customers detect. Alternatively they might only publish incidents AWS thinks are actually public. There might be a threshold of “N or more customers” where a large failure could go unreported if it affects less than N customers. There might actually be no policy and it is entirely up to the current operations staff to decide whether to report or not (which might lead to biases between regions, too).

So there’s already a large possibility of both systematic and random errors there.

To summarize previous point: you don’t have any idea how complete the information in the dashboard actually is.

AWS doesn’t publish much information on how they run their datacenters, but from compliance information it is possible to infer that to meet SOC 1/2/3 and ISO 27001 requirements they must have mechanisms that track, record and assess incidents in more detail than is shown in the dashboard itself. Whether their incident management processes are based on ITIL or something else isn’t known, but for the purpose of this post it isn’t really relevant.

Secondly, let’s take a look at what actually is published. The published information consists of:

An identifier (as RSS GUID, based on service, region and publish time)
Region and service
Title
Message body

That’s it. Compared to Azure dashboard’s underlying JSON, for example, the data you get from AWS dashboard is very unstructured. It is essentially a pair of freetext fields. The title and body content also varies quite a lot. I’ll show a few sample messages. The first one is for Cloudwatch in the eu-west-1 region published on February 19th 8:15 AM PST (first line is title, rest is message body):

Service is operating normally: [RESOLVED] Delayed metrics in EU-WEST-1

Between 07:20 AM PST and 08:05 AM PST, customers may have experienced some delayed alarms in the EU-WEST-1 Region. We have resolved the issue. The service is operating normally.

and this one for RDS in the us-west-1 region from May 26th:

Informational message: Network Connectivity

We are continuing to bring the few remaining impacted instances back online in a single Availability Zone in the US-WEST-1 Region.

Going through a lot more of these messages you’ll notice there are some common features:

They mostly follow a common formula of a “we’re investigating” message followed by “we have identified the problem and are working on a fix” followed by a final “resolved: between then and now …” message.
They don’t follow the common formula rigidly. This means that although many events are ended by a message telling the exact time boundaries (“between …”) there are plenty of those that do not.
They are written by humans for humans. They contain typos (“EU-WEST-2” anyone?), contextual references easy for humans but not for computers, different representations for the same information (“Between 14:40 and 16:32 PST”, “Between 1:51 PM and 2:37 PM PST”, “Between 12/17 10:32PM and 12/18 2:12AM PST”, “Between 2:10 A.M. PST and 2:40 A.M. PST” and so on), …
There are no correlation identifiers available. This means that just by looking at two different messages you cannot determine whether they are part of the same event. There are overlapping events so just chaining messages in time sequence is not reliable.
They are retroactively edited. The simplest case is the inclusion of “[RESOLVED]” to the subject line for all messages for a resolved incident. There are more complex examples where the message body has been amended multiple times during the course of an incident.

Below is an example of one such message. The message itself was published at March 20th 2013 08:36 PM PDT. I have only two snapshots of the message so I can confirm only the addition of an 03:09 AM update (plus minor formatting changes), yet it is possible to infer that it has been edited multiple times at around 8:45 PM, 9:43 PM, 11:49 PM, 12:36 AM (next day), 02:33 AM, 03:09 AM and 04:03 AM.

Informational message: [RESOLVED] Back-end instance registration issue

Increased provisioning times 8:45 PM PDT.PDT We are investigating increased provisioning, scaling and back-end instance registration times for load balancers within the US-EAST-1 Region. 9:43 PM PDT.PDT We continue to investigate increased provisioning, scaling and back-end instance registration times for load balancers within the US-EAST-1 Region. We can confirm that request traffic to existing load balancers has not been impacted by this issue. 11:49 PM PDT.PDT We have identified the root cause of the increased provisioning times in the US-EAST-1 Region and are working to return the service to normal operation. We can confirm that request traffic to existing load balancers has not been impacted by this issue. Mar 21, 12:36 AM PDT.PDT Between 7:45 PM PDT on 3/20/14 and 12:14 AM PDT on 3/21/14 we experienced increased provisioning, scaling and back-end instance registration times for load balancers within the US-EAST-1 Region. Request traffic to existing load balancers was not impacted by this event. The issue has been resolved and the service is operating normally. Back-end instance registration issue 02:33 AM PDT.PDT We are investigating a back-end instance registration issue affecting a small number of load balancers within the US-EAST-1 Region. 03:09 AM PDT We have identified the root cause of the back-end instance registration issue affecting a small number of load balancers within the US-EAST-1 Region. We have made progress in resolving the issue for some load balancers and continue to work on remaining load balancers. 04:03 AM PDT We have corrected the back-end instance registration issue for the majority of the affected load balancers within the US-EAST-1 Region, and continue to work on the remaining load balancers.

Some messages have HTML formatting, but most are pure plain text. It seems that longer-running events with multiple updates are more likely to contain HTML formatting (primarily colors). The previous message originally contained HTML formatting, but I’ve stripped it out (it does not seem to contain any semantic meaning).
Severity of an event is almost never discussed in detail. What you get is “a subset of instances were affected”, “a small portion of”, “some” or similar. Sometimes as an added assurance the number of availability zones affected is included (which almost without fail is always “one”).
It seems that it is possible to differentiate between at least some people by their writing style, although this seems to apply more to older messages than more recent ones (internal standardization?).

Any of these are not big problems for humans. Most of the typos and mistakes are such that a human can easily infer the correct meaning from context. Humans are super-cool contextual inference engines, superb at piecing messages together into a cohesive understanding. What’s difficult (guess what I’ve been up to?) is trying to turn these automatically into quantitative information about outage events.

Now this isn’t a poke at AWS’s dashboard. Building trust by sharing outage information publicly is very important, all kudos to AWS for that. AWS has done also a great job in posting analyses of larger incidents (example). These are just things I’ve found out while doing in-depth analysis of AWS outages and digging deep into dashboard messages. I have not found any deficiencies or systematic errors that would devalue AWS service health dashboard as a very good source of current up-to-date incident and outage information.

(If your ops team is currently not monitoring AWS dashboard RSS feeds for the services and regions you are operating, well, do so.)

Minimum spot instance prices

2014-03-25T00:00:00+00:00

(You might want first see the introduction to this series of posts if you jumped in here randomly.)

Warning: This article is really about splitting hairs. If you think watching paint dry is boring then this post most probably isn’t for you.

In my previous post I stated that AWS has set minimum spot instance prices and incorrectly asserted that these minimums are can be seen in the price-too-low error when submitting low bids. This is wrong (I’ve updated the earlier blog post slightly to avoid spreading the wrong fact), as the “minimum” bid price given is actually the current spot price. Oh, how could I miss that.

How to find a minimum spot price

Thus there is no direct way to get the minimum spot price in any market. But it is possible to infer these indirectly from spot price history data? I looked at the minimum spot prices in all regions and instance types (picking the lowest of all zones) and plotted it getting the graph below (the dataset spans 2013-12-08 to 2014-03-09):

Minimum spot market prices by instance type and region. The minimum is calculated over all the zones in the region. Note that cc1.4xlarge is missing due to a limitation in the source data, not that it doesn’t have bids.

Take a look at the graph for a few seconds. Go ahead. Don’t skip ahead until you’ve taken a bit of time to look at the dots. Okay, let’s continue then. I think you have also noticed there are patterns. There is a distinct pattern for several instance types in the data — m3.xlarge, m3.2xlarge, t1.micro for example, but also continuing over the m1 and m2 classes. There may be another pattern with some c3 instance types and maybe yet another for m3 too, but let’s stick with the most obvious one for now. I’ll label the set of “similar” minimum price pattern the “suspect” group. Below is a plot with all of the “suspect” instance types (minimum relative prices) plotted on top of each other:

Minimum spot price for instance types m3.xlarge, m3.2xlarge, t1.micro, m1.large, m1.xlarge, m2.xlarge, m2.2xlarge, m2.4xlarge, m1.small, m1.medium, c1.medium and c1.xlarge. Different colors correspond to different instance types. Y-axis positions are slightly jittered.

This graph requires a bit of thought to understand so bear with me. These are relative minimum spot prices so although the absolute minimum spot price differs from region to region these should be comparable to each other. In the graph there are two things that you need to be considered:

Similarity between regions. It is quite clear that ap-southeast-1, ap-southeast-2 and us-west-1 are almost identical to each other. With eyes squinted us-west-2 has a cousin-style similarity to these three but all of the others are definitely dissimilar.
Levels of the relative prices. You can find the cheapest spot instances (for the instance types discussed) in us-east-1 whereas ap-northeast-1 and eu-west-1 are clearly more expensive (err… less cheap?). All of the others seem to have roughly similar average minimum level.

What I find very interesting is the identical levels and structure between the three topmost regions. Here’s the raw data from these three regions in a tabular format:

	us-west-2	ap-southeast-1	ap-southeast-2
c1.medium	0.028	0.028	0.028
c1.xlarge	0.112	0.110	0.110
m1.small	0.010	0.010	0.010
m1.medium	0.021	0.020	0.020
m1.large	0.042	0.040	0.040
m1.xlarge	0.083	0.080	0.080
m2.xlarge	0.056	0.059	0.059
m2.2xlarge	0.112	0.118	0.118
m2.4xlarge	0.224	0.236	0.236
m3.xlarge	0.092	0.088	0.088
m3.2xlarge	0.183	0.175	0.175
t1.micro	0.004	0.004	0.004

Lowest observed spot instance prices in dollars in the regions us-west-2, ap-southeast-1 and ap-southeast-2 between 2013-12-08 and 2014-03-09.

These values are in dollars e.g. they are not relative prices. First of all both ap-southeast regions have exactly the same observed minimum spot prices. The us-west-2 region has several instances where prices are identical to ap-southeast regions, some where prices are slightly higher and some slightly lower. However it should be noted (as can be seen in the earlier graph with relative prices) these differences are very small compared to differences to some other regions.

Regardless how you slice and dice these numbers I find it exceedingly unlikely that these very similar relative and absolute minimum spot prices in multiple regions would be result of pure chance. At least from practical point of view there are minimum spot prices.

How are minimum spot prices set?

If this really is so, it raises another question: How are these minimum spot prices determined? Have they been set by AWS itself, or are they an artifact of external bidders during the data period?

And if they are set by AWS, then by what policy does AWS set them?

One possibility is that they are calculated from region’s on-demand prices. Yet at least for the three regions considered above this does not hold. The ap-southeast regions have same on-demand prices but us-west-2 has substantially lower prices. Yet it has very similar minimum spot prices. For example the c1.medium instance type has an observed minimum spot price of $0.028, yet its price in ap-southeast regions is $0.183 compared to $0.145 in us-west-2.

Could the minimum spot prices be based on operational costs? This seems more plausible since there seems to be similarity in the minimum prices between regions for the same instance generation — that is, instances running on so-called “first generation” hardware (compared to “second generation” of m3 and “new generation” of c3 classes) such as m1, c1, m2 and t1 had somewhat similar prices between regions. For c3 class instance types this similarity is even more striking:

Minimum spot price for c3 class instance types. Different colors correspond to different instance types. Y-axis positions are slightly jittered. (The reason why you are not seeing many discs is because most of them are exactly on top of each other.) sa-east-1 is missing from this graph as it didn’t have c3 instance types during the period covered by the data set.

Note that apart two outliers (more on these below) the relative prices for c3 instances between all of the regions are almost exactly the same. Again although absolute minimum prices differ the relative minimums (percentage of the on-demand price) is almost exactly the same for almost all regions for almost all c3 instance types.

In plain English this means that for all c3 instance types the minimum bid price is a fixed percentage of that region’s on-demand instance price.

That’d be a mighty coincidence if there was no minimum and all of this similarity was result of different bidders all over the world?

Okay. There’s the question of the two outliers above: c3.large in ap-southeast-2 at $0.001 and c3.8xlarge in ap-northeast-1 at $0.060. Getting c3.8xlarge at 1/40 of the on-demand price seems like a great bargain. If AWS enforced minimum prices I’d expect them to be set in all regions. Was it left out accidentally for these two cases? If so, it was around for quite a while (the $0.060 price for c3.8xlarge was hit in 2013-12-10, 2013-12-12, 2014-01-03 and 2014-01-04 for a total of 338 minutes at that price and the $0.001 for c3.large on 16 distinct days between 2013-12-22 and 2014-02-03 for a total of 13.4 days.)

For the outliers I have two explanations:

It’s an accident. For some reason the minimum was set incorrectly for these two instance types.
There is actually no minimum spot price. Reconciling this view with the observed very similar minimum relative prices in multiple regions is difficult, though. One possibility that comes to mind is that maybe — maybe someone itself is bidding in all spot markets for spare capacity at a very low price and these low prices reflect those bids when there is very little demand by other customers.

Yet even this outlandish scenario raises new questions. Why would this hypothetical all-excess-capacity-sucking entity bid at different prices or even at different percentages of on-demand prices? Why not bid at absolute $0.001 or at relative 10% in all regions for all instance types? If someone was really doing this then it seems reasonable that more than one market would have seen such a lull in demand that such a consistent low bid would become visible. This hasn’t happened (at least not in the data I have).

What else could be gleaned from this data?

Minimum relative prices for 1st generation instances are more spread out than for c3 instance types. Actually, the c3 relative prices are very tightly packed.

I’m definitely not sure, but this might reflect the underlying hardware and its operational costs. I think AWS has learned a lesson or two from first generation instances and its hardware. It seems reasonable that they have better understanding on how to optimally pack instances into a single server and how to operate them.

The earlier generation instances run on multiple different types of hardware (it is know they have different CPU types, at least, see Is the Same Instance Type Created Equal? Exploiting Heterogeneity of Public Clouds by Ou, Zhuang et al, 2013) and it seems that the capacity progression in m1, m2 and c1 classes is not always a simple 2x step from instance type to another (potentially leaving “unfilled gaps”). These might explain why older instance types have more “spread” in their relative minimum prices than the newer ones.
Japan and Europe (ap-northeast-1 and eu-west-1) have consistently higher minimum relative spot prices than most other regions. OTOH, ap-southeast-1 is the odd one out since with old instance types it had identical relative prices with two other regions, but for c3 instances it stands out as substantially more expensive (“less cheap”).

This might be due to relatively higher operational costs in these regions and that the on-demand prices are not entirely based on operational costs. That is, they take into account competition, willingness of the market to accept the given price level as well as any desire by AWS to establish a (dominant?) market presence. All of these may provide a rationale to price on-demand instances at relatively lower gross margin level than in other, more established regions.
Also it is interesting that us-east-1 doesn’t stand out as “cheaper” for c3 instances in the same way it did for 1st generation instance types.

Summary

So as a conclusion to the question whether there are minimum spot prices and who sets them I think the answer is that from a practical point of view there are minimum spot prices. If you are bidding in these markets you have to understand that there appears to be a minimum bid price you have to use to have any chance of getting an instance, ever. Although not conclusive, I find the most plausible scenario for these minimum prices that AWS is setting minimum spot prices based on operational costs, but using different formulae for different instance generations.

Mechanics of the spot instance market

2014-03-20T00:00:00+00:00

(You might want first see the introduction to this series of posts if you jumped in here randomly.)

Deep dive

In previous posts I’ve discussed about what are spot instances and what is the spot market and what you can use spot instances for and how. In this post I’m going to write out my thoughts on what is the reason for spot market, its rationality and where actually do spot instances come from.

Purpose of the EC2 spot market

Why does spot market exist in the first place?

Spot instances were announced on December 14th, 2009. After that there has been several technical updates that brought spot instances to the same level as other instance types (such as EMR support, VPC support). There has been two major published changes on the spot market itself. First, the spot market price algorithm was changed on July 1st 2011 and secondly a default bid price cap was introduced late 2013. These are the visible changes that have the name “spot instance” on them.

What does this tell us about the purpose of the spot market?

Not yet much. But it is telling us something:

The spot market is meaningful to AWS.
AWS wants us to use spot instances.

But what about the purpose? Why did AWS go to the complication of providing spot instances (more code, more work, more bugs) and operating a spot market (apparent loss of pricing control) on top of that? Why didn’t it just say “spot instances at 50% price of regular ones” and leave it at that?

I have not seen that AWS would have directly stated the purpose of spot instances. All of the official information I’ve seen carefully skirts about the purpose of spot instances and spot market. The initial announcement tells that “[you can] bid on unused Amazon EC2 capacity” and the current spot instance landing page that “you simply bid on spare Amazon EC2 instances”. There are plenty of whys for the customer, but no why for AWS itself.

I believe the groupthink of the Internet is mostly in line with the following hypothetised (aka naive) purpose:

Spot market is for AWS to sell excess capacity to make at least a bit of more money out of resources that otherwise would remain unused (incurring both operational and capital costs).

This seems sensible and straightforward. Yet it does not tell about the purpose of a spot market. Dmitriy Samovskiy makes a good point about that — why are they “spot” instaces and not “discounted” instances? It is entirely possible that AWS would have priced “discounted instances” at -50% and left it at that (adding the “may-be-terminated-at-any-time” clause). Instead the spot market exist, with its high price volatility, spot price differences between regions and a potential to pay up to $999.99/hour per instance. All of this is bound to make a lot of people wary of spot instances.

Think about it. If prices were set at a fixed 50% (or 40%, or 30%) then the element of market variability would be removed. I think a lot of people would be more comfortable with fixed discounts over the variability of spot market prices.

There’s this thing called “efficient market hypothesis” in finance theory that posits that financial markets are “efficient” at setting prices on traded assets. That is, the public price reflects supply and demand in a true manner. So one possibility is that maybe — maybe AWS thinks that it can increase its income on spot instances by letting “the market” decide instaneous spot prices instead of a fixed discount?

I wouldn’t trust that. After all, the spot instance market is not a real market. Bids are not open. Supply is hidden. Even the pricing algorithm is unknown — assumptions about it being a true bidding market have been shown to be false in the past. (All of this this and more has been pointed out years back in blogosphere as well as in academia).

So what is then the purpose of the spot market?

I don’t know.

I am sure that part of its purpose matches the naive assumption — it is generating income for AWS that otherwise would have been lost. Later below I’ll talk about other partial purpose (that surprisingly ties spot market to reserved instances), but I’m not sure about that being the totality either.

In the end I don’t know what is the purpose of the spot market. I’m not saying that it wouldn’t be useful. After all, you can get substantial savings on operational cost using spot instances! You don’t have to theoretize about the purpose of rain to benefit from it, either (in case you’re a farmer).

I just don’t believe that the naive hypothesis is all there is to.

Is the market rational?

The answer is absolutely clear and simple for this one: yes and no.

See both above and earlier post. The spot market pricing algorithm is not known. I’m not going to call any market rational whose price is potentially set by a random number generator and the market players are finding causality in places where there is none.

Yet if you make the assumption that the spot market price is at least mostly a market-driven proxy of supply and demand (and leave the algorithm in the hands of the benevolent AWS) and ask questions about the behavior of the bidders, then the answer is yes. Yes, at least most of the bidders are making rational choices.

The question of AWS spot market’s rationality is a common question (see here, here and here). Although the famous $999.99 spike was probably a genuine human mistake (e.g. not rational), it is still useful to ask why anyone would bid over the price of an on-demand instance. A lot of people think it is not logical. Yet this does really occur all the time. Is the net full of loonies or not?

(Yes it is. But let’s consider the spot market only, shall we?)

Earlier I’ve pointed out that the total costs of running a spot instance can easily be less than the cost of using equivalent on-demand instance even when you bid at 10x the on-demand price. Thus at least for those cases where you can expect (based on past history, which of course is not indicator of future blah blah blah) likely savings with high bids then it is entirely rational to bid at >1x prices.

But then what about c3.2xlarge in us-east-1? See the graph below:

Daily average and maximum prices for c3.2xlarge spot market prices in us-east-1. Solid line is the weighted daily average price, lineless blocks are the maximum daily bid price and colors represent different zones.

Although not we have the benefit of hindsight, I think anyone bidding for c3.2xlarge during January 2014 would have quickly realized that they are not getting an instance at on-demand prices. Why then?

The c3 class of instances was announced in November 2013 and from the very beginning demand for them was high. In fact, demand was higher than supply. For anyone familiar with economy 101 this is a case of supply vs. demand where the price of a good should rise when demand is higher than its supply. Yet the on-demand instance pricing is [not elastic](http://en.wikipedia.org/wiki/Elasticity_(economics) and cannot be changed rapidly by AWS to match the unexpected demand (AWS can change the price, but I don’t think they want to increase the price for PR reasons).

You can see where this is going, right? Spot market price is elastic, and in this case it clearly shows that when demand outstrips supply, the per-unit price increases. In the graph above you can see that c3.2xlarge prices have started to fluctuate and on average, have gone down since late February. This is most likely due to AWS being able to introduce capacity faster than the demand has increased. (An alternative interpretation is that a lot of those interested in c3.2xlarge have become disillusioned at its (un)availability and gone elsewhere.)

But why would anyone pay >1x cost for an instance? There are, after all, plenty of other instance types (even in the c3 class) that are available at on-demand prices from either on-demand or spot markets. Why?

I have no idea what goes in bidder’s heads. But there are a few possibilities that are entirely rational that come to my mind:

Someone values uninterrupted service over savings. See BrowserMob’s bidding strategy at 4:00 in this video. They clearly put a large weight into getting the resources now even at higher price than later and cheaper.
Someone tests how their application runs on c3 instances. They might be contemplating moving a production environment over (c3 is cheaper than m1… at regular prices). Doing a time-limited performance evaluation even at the overpriced spot market prices isn’t going to break your bank and would provide you with valuable information for the future (e.g. will you buy c3 reserved instances or not).

I’m sure there are others, but eventually they all boil to the same conclusion: buying at high cost in the spot market is rational if doing so offers larger potential benefits than waiting to buy at regular prices later.

Spot market price drivers

Although we don’t know the actual spot price algorithm, it is possible to observe it and see whether its behavior correlates with other, known events.

When talking about the spot market algorithm the first stop most definitely has to be a paper called Deconstructing Amazon EC2 Spot Instance Pricing by Beh-Yehuda et al (2011). The researchers did a very thorough analysis of AWS spot instance market and the spot price behavior. Even though most of the analysis is using data prior to the 2011 pricing mechanism change and thus is not valid today it is still a good read. Especially the bit in epilogue where the researches state that

“While these radical qualitative changes [June 2011 pricing mechanism change] are further evidence of the former prices being artificially set, the October prices are consistent with a constant minimal price auction, and are no longer consistent with an AR(1) hidden reserve price.”

So … AWS didn’t use a “market” algorithm before, but they seem to be using one today. As a working hypothesis I’ll take it that there is some market-based price algorithm that takes some inputs and outputs a spot instance price. What are the inputs?

One thing that we know is that “the Spot price will raise when our [AWS] capacity lowers.” and “the increase in the m2.2xlarge Spot price today [the $999.99 price spike event] was related to an sudden increase in demand for On-Demand m2.2xlarge instances which significantly depleted the unused capacity.” (source).

Available EC2 instance capacity affects spot price.
From looking at the c3.2xlarge spot price graph it should be also obvious that demand has an effect. When considering also quotes above it is possible to infer that:

Demand for EC2 instances affects spot price.
There is also a *minimum bid price* set by AWS. If you try to bid below this you'll get a `price-too-low` error with a message *"Your Spot request price of 0.02 is lower than the minimum required Spot request fulfillment price of 0.403."* (numbers naturally vary). **There is a minimum spot price**, which varies by instance type and region. (I am not sure about zones.)
(Updated 2014-03-25) Wrong wrong wrong! The price-too-low message is really only talking about current spot price. My bad. However there still appears to be minimum spot prices which I go through in the next post in this series.
AWS has a default maximum bid limit of 4x on-demand price but this is a soft limit and can be raised or removed. The maximum relative bid price varies, but in the data set I have there are several instane types with >50x spot prices. This implies that there are bids at that level and potentially higher.

It is not known whether there is any spot price upper limit.

(If anyone is brave enough to do a short bid at 10000x price level I’m interested in hearing about the results.)

In the next section I’ll talk about my hypothesis about where spot instance capacity comes from, but from Dave@AWS’s quotes and other observations it should be clear that spot market price is affected by demand on all instance types, including on-demand and reserved instances. That is, an increased demand for on-demand instances may affect spot market price even when no changes occur in the spot market bid pool.

It seems reasonable to assume all of the above affect prices. But this doesn’t remove the possibility of other price drivers. It is entirely possible that AWS would artificially push the spot price up (to maximize their profits — 50x0.99 is more than 100x0.20) or depress the price (to make spot instances more appealing?). Or an increase in the capacity is fed to the pricing engine slowly to prevent rapid price fluctuations. Or a decrease in capacity is pre-factored so that it is removed in steps instead of a large drop (and matching rapid price increase). Or …

Where do spot instances come from?

Note: Most of this section is pure speculation. I am presenting a hypothesis about AWS’s division of instance resources which may be completely wrong. However as far as I’m concerned it is a hypothesis that is in line with actual observations.

The official statement from AWS is that the capacity for spot instances is “spare Amazon EC2 instances” (source). A bit more verbose is Dave@AWS’s commentary in the AWS forums:

“To answer your second question, you asked what other capacity pools could be a part of Spot. Behind the scenes, our goal is to have all of Amazon EC2’s unused capacity integrated into Spot. By optimizing the use of these instances, we hope to be able to pass along more savings over time to our customers. Selling our unused capacity means we may leverage unused capacity from other pools like On-Demand or other parts of our capacity that can be temporarily sold but may need to be reclaimed at a later time. It would take precedence over On-Demand, because we do not have the ability to reclaim On-Demand instances, so they cannot be sold there.” (Emphasis is mine.)

Let me go through behaviors associated with the main instance types:

Instance type	Guaranteed availability	Arbitrary termination
On-demand instance	No	No
Reserved instance	Yes	No
Spot instance	No	Yes

(Notice how reserved instances and spot instances are complementary to each other.) Although AWS may have other (internal?) capacity pools with other access constraints, I think that “unused” capacity at any moment can be divided into two sets: one that can be used for on-demand instances and one that cannot be used for on-demand instances. This is because of the semantics of reserved instances.

“Reserved Instances provide a capacity reservation so that you can have confidence in your ability to launch the number of instances you have reserved when you need them.” (source)

When you purchase a reserved instance you have no obligation to run it, but AWS has an obligation to provide you with a reserved instance any time you want to run it. This means that any reserved instance that is not running could be sold, but not as an on-demand instance since AWS cannot evict an on-demand instance at will. See the figure below (which doesn’t have too many non-negations):

At any moment in time AWS’s total capacity is split into running instances and unused capacity. Running instances are further divided by type into reserved instances, on-demand instances and spot intances. The pool of unused capacity has a portion which cannot be sold as on-demand instances (because if it was sold and a lot of powered-off reserved instances were started it might not be able to provision resources for all those reserved instances). Thus there is unused capacity that can only be sold as spot instances.

This also means that there can be unused spot instance capacity even when on-demand instances cannot be provisioned. So finally we can explain why c3.2xlarge instances could be purchased from spot market even when you couldn’t buy on-demand instances: there was a pool of c3.2xlarge reserved instances already sold that were not powered on.

When reserved instances are powered on I think this is what happens:

If there is unused capacity, it is used to provision the reserved instance. End of story.
If no unused capacity was available, the spot market is notified that it needs to release capacity from the spot pool.
Spot market algorithm recalculates the spot price based on the new (reduced) total capacity. If this changes the spot market price then it’ll terminate those spot instances whose bid price fell below the spot market price. The released instance capacity is allocated back to the reserved instance.

What happens if the change of the capacity does not change the spot price? I’m not sure. It might be that the spot market algorithm will forcefully increase the spot price. As well it might not. The exact wording from AWS is “If the Spot price exceeds your max bid or there is no longer spare EC2 capacity in a given Spot pool, your instances will be terminated.” which I think leaves open the possibility that a spot instance is terminated also on capacity decrease even when the bid price doesn’t change.

If anyone has had their spot instance terminated even when the bid price equals spot price I’d be delighted to hear about your experiences.

(It is possible that there are also other pools of resources that are available for spot market use. Maybe new servers are first assigned to a “burn-in pool” which is sold only via the spot market. Maybe AWS has internal testing pools that are available for customers when not needed. I have not seen anything that would suggest so, though.)

Market efficiency and reserved instances

If my hypothesis about that powered-off reserved instance capacity is sold in the spot market then (I claim) that spot market is essential for AWS to maximize its income from reserved instances.

You could say that a working spot market is a requirement for reserved instances.

Think about it. If AWS was not able to resell unpowered reserved instances then it would be making loss with reserved instances. The c3.8xlarge light usage reserved instance upfront cost is $2666. It has 32 (virtual) cores and 60 GiBs of memory so I think that c3.8xlarge represents almost a physical single server (each E5-2680v2 has 20 threads, which I guess maps to an EC2 core) and I’m pretty sure it’ll cost more than $2666.

If AWS was not reselling unpowered reserved instance capacity then anyone buying a light utilization reserved instance would most likely end up costing AWS concrete and real money. At the minimum it would make the gross margin on those physical servers very low.

Open questions

I have a hypothesis. Good hypotheses can be tested with tests that either falsify the hypothesis or give results that are in line with earlier predictions.

Actually, to be accurate, I have two hypothesis. The first one is that spot market price is affected by supply and demand for all types of EC2 instances (also that there is a minimum spot price and there is no maximum spot price but we know the first for a fact and I’m not sure the second one is meaningful to explore at all).

The second one is that unused but purchased reserved instance capacity is re-sold as spot instances.

I’ll infer from these that the spot market should behave in the following manner (all of these apply to each region, instance type and availability zone separately):

Unpowered reserved instances and stopped on-demand instances should not affect spot price.
Purchasing reserved instances (without powering them on) …
- … should not affect spot price.
- … should decrease unused on-demand instance capacity. (This of course may not be visible in any way.)
- … should increase unused spot instance capacity.
Powering reserved instances on …
- … may increase spot price.
- … may cause spot instances to be terminated (even when spot price remains unaffected).
- … should not affect availability of on-demand instances.
Powering reserved instances off …
- may decrease spot price.
Provisioning new on-demand instances or starting stopped on-demand instances …
- … may increase spot price. (We already know via AWS forum comments about the $999.99 spot price spike that demand for on-demand instances can affect spot market prices. It is not clear what the mechanism here is though — does AWS preferentially give capacity to on-demand requests?)
- … may cause spot instances to be terminated. (See previous.)
Terminating on-demand instances or stopping on-demand instances may decrease spot price.

That’s a lot. How could these be tested? First and foremost, testing any of these is potentially expensive as you need to provision instances and put in bids for spot instances and you’ll need to pay up for all of that. (These might also violate AWS’s Acceptable Use Policy.) It might be possible to infer some of this data from actual spot market logs and/or other monitoring data, though how I don’t know.

Buy one or more reserved instances. Start reserved instances. Based on the hypothetised behavior this should cause the spot market price to either increase or remain the same.

(Given that there are other people powering instances on and off this would show up only as a statistical result from many iterations. This applies to all other tests too.)
Power off reserved instances. This should cause the spot market price to decrease or remain the same.
Purchase spot instances at spot market price or slightly above. See how often their termination is associated with spot price increases. (Some of them should not be.)
Purchase spot instances at spot market price. Power on reserved instances. There should be a correlation between starting your reserved instances and termination of your spot instances.
Start on-demand instances. This may be correlated with spot price increase.
Stop on-demand instances. This may be correlated with spot price decrease.

Of course testing any of these is fraught with difficulty. Starting and stopping one instance is unlikely to affect the spot market price (the system would have to be near a transition point) and any result could be swamped badly by random effects (other users). You could reduce environmental noise by choosing relatively unused region, zone and instance type but in that case you’d probably have to purchase a significant number of instances to see any effect.

How does this affect you?

Not really much. You shouldn’t try to second-guess future behavior of spot prices.

If my hypothesis is correct, then you might want to keep in mind that the spot market price is affected by events that occur outside the spot market. That is even an apparently stable market can change suddenly without any change in the bidding pool.

But you already knew that spot market is volatile, didn’t you? No new news, then.

Here’s the next post in the series.

Using spot instances

2014-03-19T00:00:00+00:00

(You might want first see the introduction to this series of posts if you jumped in here randomly.)

How to use spot instances

I’m going through a couple of topics related on how to use spot instances:

Suitable applications and workloads for spot instances
Bidding automation and bidding strategies
Minimizing effects of price spikes

Determining whether your application can benefit from cost savings using spot instance is quite straightforward to analyze — it’s a cost vs. cost analysis. There is quite a lot of information on bidding strategies and cost spike mitigation, but information in bidding automation is sparse as companies using spot instances generally do not publish their bidding engines or its parameters.

Suitable applications and workloads

Here’s a list of applications suitable for spot instances according to the source itself:

Batch processing
Hadoop
Scientific computing
Video and image processing and rendering
Web / Data crawling
Financial (analytics)
HPC
Cheap compute (“backend servers for facebook games”)
Testing

The common theme in all of these is loss of an instance is not a catastrophe. You can influence the likelihood of an instance loss through the bid price (see instance availability in previous post), but unless you are willing to face potentially absurd costs to guarantee 100% spot instance availability you’ll have to come to face with the fact that:

You have to be able to recover from sudden spot instance termination.

Whether you would want to use spot instances and whether you can use spot instances is determined by three factors:

Potential savings gained by using spot instances.
Costs of a spot instance failure. For example loss of profit and money and work required to recover.
Costs required to either completely avoid failure in face of spot instance failures, or to mitigate the risk to acceptable levels.

The firsts two are recurring (you get savings continuously, but spot instance failures also occur continuously) whereas the third one is mostly one-off cost.

And face it, if you are using spot instances you have to be prepared that many of them fail at the same time. You can have some influence over the number of lost instances by using multiple availability zones and tiered bidding (see moz.com developer blog for excellent insights) but however you slice and dice you still come to the fact that:

You have to be able to recover from sudden spot instance termination.

How you deal with instance termination is affected by what are your costs to fail and costs to prevent failure. Consider a few cases:

Spot instances as build slaves. Your CI automatically provisions build slaves from spot market as needed (and tears them down when demand goes down). So now suddenly all your build slaves went away — so what? Jobs failed, builds lost, but it’s not going to kill your devtest.

The recovery method in this case would be simple: first of all, the CI instance launcher might already have built-in balancing from many zones (meaning it’ll bid in multiple availability zones). Even if that wasn’t the case you could go and manually change the bidding parameters to a higher bid price (maybe you accept a bit higher costs while thinking of some other solution), use another zone or use another instance type. You might equally well just wait a while to see if the price just spiked and would go down soon.

In this case it is likely that the cost to prevent interruption of CI jobs would be higher than productivity losses so it is reasonable just to wait it out and handle any aftermath manually.

(Just do not run your build master in a spot instance.)
Hadoop cluster. Assuming you are using your Hadoop cluster semi-continuously (ground radar signal processing, mobile game user analysis etc.) there are a few possible scenarios. For the most part Hadoop will automatically re-assign map-reduce jobs from failed nodes, so loss of some nodes isn’t a biggie for Hadoop at all. Mahujar’s post Riding the Spotted Elephant is an excellent article discussing various pros and cons on different ways to use spot instances with Hadoop. Essentially this boils down to:

It is possible to run a Hadoop cluster using spot instances where sudden price peaks will have only a limited effect (delaying completion of some jobs) sans force majeure situations.

In this case you could be hedging your bets by using a hybrid cluster, some on-demand instances and some spot instances, potentially with tiered bidding. This will increase the running cost but will be highly likely to prevent massive failures.
Financial analysis. (I’m not a financial market wiz, so bear with my unbelievable scenario here, please.) You’re running a financial modeling job nightly using spot instances. The job will take 4 hours to complete and the time window to run it is six hours. It must not fail.

Okay, if it must not fail then you should not be running it using spot instances in the first place. So let’s reword the requirement. “Must not fail with nightly operating costs less than X.” That is, if the cost of not failing would be over X you can fail.

You’ll need three things: bidding automation, provisioning automation and checkpointing. The first one is to try to keep your instances alive as much as feasible. The second one is to try to acquire complementary resources (on-demand instances, other types of spot instances, another region — whatever it takes) in case you start losing spot instances and the third one is to ensure that when you get replacement instances you can quickly continue from where the analysis stopped without having to re-do everything from scratch.

In this case the cost of prevention is large — setting up the required automation and testing it to death will itself require a large effort, not to speak about the costs that will come after the automation kicks in. But then again, the failure to run to completion would be expensive too.

The less time-critical and more resilient your computing requirements are the easier it is to move them over to use spot instances.

If you are using AWS in a large scale then you should already have disaster recovery plans for situations that would affect your service such as a whole availability zone going out (or a whole region in case you are Netflix). When using spot instances you’ll need to factor in plans for persistent spot price increases. If spot prices go up, for how long are you willing to “wait it out” to see if they drop back down? What will you then do when you decide they’re not coming down?

To recap:

Don’t use spot instances if your requirements include “must not fail”
Do a cost-benefit analysis:
- Estimate savings
- Estimate cost of failure
- Estimate cost of avoiding failure
- Compare

Bidding automation and bidding strategies

If you are using spot instances now and then for one-off tests you should do bidding manually. In this case you should bid higher than the current market price (see what I wrote about instance availability in previous post) to prevent small price fluctuations from terminating your instance. Just remember — don’t bid higher than you are willing to pay!

For simple use cases a using auto scaling for provisioning automation and setting the spot instance bid price (in auto scale launch configuration) is sufficient. This can’t alone guarantee availability of a service, but it will be enough for less than 24/7 operations.

If you had provisioning automation (automatic scale-up and scale-down) before then adding spot instances brings in a few complications:

Launching spot instances takes a longer time than on-demand instances (bidding process itself takes extra time).
Spot market price ~~can~~ will vary over time, including potentially large spikes. You have to decide how to deal with spikes.
Your spot instances can all just vanish with a sudden spot price spike.

Writing a spot market bidding and provisioning engine is thus more complicated than for scaling up and down with on-demand instances. Do make sure that you put in hard limits to your bid prices. Remember the poor sod who paid $999.99/hour for his/her spot instances.

Strategies

Depending on your application requirements you can apply several different provisioning and bidding strategies. Here’s a video that discusses various strategies AWS has detected its customers using:

Optimizing costs. These customers bid at reserved instance pricing level with the goal of gaining RI-level costs without their up-front costs. Needless to say, bidding at this low level you are facing loss of all spot instances during a price hike.
Optimizing costs and availability. These bid at a level between reserved instance price and on-demand instance price. This will not protect from sudden price hikes, but will prevent smaller fluctuations from terminating instances.
Capable of switching to on-demand instances. These customers have provisioning automation that can automatically shift from bidding for spot instances to provisioning on-demand instances when it detects that spot prices have increased >1x price level. These typically bid at on-demand price level or a little higher.
High bidders for availability. For these they are interested in getting average savings from spot instances, but put a large value on availability of their spot instances. These will bid significantly higher than on-demand price.

I think this is a reasonable strategy to deploy interrupt-sensitive application using spot instances with the caveat that you must be able to later move to cheaper resources (on-demand instances, reserved instances, other zones, other instance types, other regions) without service interruption. If you cannot move over, then permanently bidding high in hope of getting both savings and availability is gambling, not a strategy.
High bidders for resources. There’s another reason to bid high. At 4:00 in the video there’s a description about BrowserMob’s provisioning strategy where they put a very high value in getting the resources they need. When BrowserMob’s system determines it needs more capacity, it’ll first bid at spot market (the video doesn’t say but I’d guess at on-demand price). If it can’t get resources from the spot market, it’ll try to acquire an on-demand instance. If that fails, it’ll start bidding in the spot market at a high level.

Note that bidding high is a workable strategy only as long as most don’t bid high.

Bidding over the on-demand price

I want to emphasize the following:

Contrary to a lot of comments in the Internet bidding over on-demand price is an entirely rational bidding strategy in certain cases. Consider the two graphs below:

c1.xlarge in us-west-1. Left graph shows market price where solid line is daily average and lightly colored boxes are the daily maximum price. Right graph shows what would have been the total cost to achieve certain availability target. The light vertical bar is 1/4x on-demand price.

The table below shows what you would have had to bid (again, this is post hoc analysis, you would not have been able to know these values beforehand) to gain 100% availability and what it would have cost you had you bid at the given level.

Zone	Relative Bid Price	Relative Cost	Availability
Zone 1	1.293	0.201	100%
Zone 2	17.241	0.267	100%
Zone 3	17.241	0.193	100%

Bid prices and total costs relative to on-demand prices for c1.xlarge instances in us-west-1 over the same time period as with earlier graphs. The cheapest zone ended being zone 3 with the required bid being >17× on-demand instance price. Yet the zone with the lowest maximum bid price (zone 1) ended up being more 4% more expensive than zone 3.

I think this make it clear that bidding over the on-demand price can be entirely sensible strategy in some cases. It just isn’t a strategy you should be doing blindly. If you can’t handle interruptions nor you can move your workload to other zones, other spot instance types, or on-demand instances and are bidding high, then you are in a very, very bad place when the price goes up for an extended period of time.

To summarize the last point: unless you have good automation that can shift your workload seamlessly from high-priced spot instances then you should stick to one of the three first bidding strategies. They at least have a known failure model (e.g. you lose instances).

Minimizing effects of price volatility

Since spot price volatility is a given, is it then possible to somehow control the effects of that volatility? The basic approach is to reduce the probability of that volatility causing problems and secondarily to limit the impact of any problems encountered.

Tiered bidding and multiple zones

There are few other tricks noted elsewhere that you can use to restrict the severity of price hikes:

Bid in multiple tiers. Add some randomness to your spot bids. If you determine that you should bid your resources at X, then bid at X, X + 5%, X + 10% and X + 15%. This means that if the spot price peaks at X + 4% then you’d lose only 3/4 of your spot instances. (You can elaborate this further and match the bidding structure to some “reasonable” estimates of price volatility based on history etc. etc.)
Bid in multiple availability zones, but in different bids. Don’t blindly use the AWS’s behavior of picking the cheapest zone when you specify multiple zones in a bid. If you have bid automation, don’t blindly always bid in the “cheapest” zone either.
If your application can automatically handle new instances (self-registration, autodiscovery etc.), you can live short price spikes through with persistent bids. Persistent bids stay in the bidding pool and will be filled at any time the spot price is below the bid price — even if the bid “lost” its instances due to a price spike.

AWS allows you to specify multiple availability zones in a single bid. In this case AWS will pick the cheapest (lowest spot price) zone at that moment where the spot request can be fulfilled.

If you continuously put your instances into the cheapest zone the majority of your instances are likely to end up in a single availability zone. Take a loot at the graph below showing m1.small spot prices in multiple availability zones. There is always a possibility that a single zone has a long stretch of relative tranquility and low prices.

m1.small in us-east-1. Notice how zones 1, 2 and 4 have a long history of low prices and low volatility, yet zones 2 and 4 have sudden spot price level changes.

Yet that tranquility can always end suddenly. I haven’t looked at time correlations between zone prices, but from a look at the graphs I think there is sometimes correlation (e.g. if spot price raises in a zone for an instance type it is likely to go up in another zone), but similarly sometimes there is no such correlation.

So you should ensure that your spot instance bids are distributed over multiple availability zones if that is feasible for your application. See Bryce Howard’s commentary on moz.com crawler outage and how it was primarily caused by placing spot instances in a single availability zone.

Hybrid

A very common advice is to not run all your infrastructure on spot instances. This is a very good advice. It is not always sensible to go after the highest savings. A good strategy is to use a healthy mix of reserved instances, on-demand instances and spot instances.

Keeping state, checkpointing, job subdivision

This is a topic I’m not going to go deeply, but the core idea is simple:

Periodically save the state of whatever your spot instance is doing (checkpointing) so that if it is terminated, another instance can continue from the last saved checkpoint.

(Edit 2015-01-06: AWS announced a two-minute termination notice available via instance metadata. You still can’t prevent termination, but you do not get a short notice before it occurs.)

Extension of this is to store the state continuously, but there are tons of tradeoff and what’s a good choice depends on your goals and your applications. Computation tasks that split naturally into iterations or queueable jobs are easy, those that have gigabytes of state or require a lot of I/O to store temporary results are more difficult.

Keep in mind that AWS will not charge for a partial hour on spot instances it terminates. This means that you should consider checkpointing only for long-running jobs and those where job completion time is an important factor. If your jobs take less than an hour then a loss of a spot instance will only delay the job, but that delay won’t cost you anything in instance charges either.

(You can play chicken with spot instances where after you’re done with the instance you won’t actually terminate it immediately, but wait to see if AWS does it before the full hour. Sometimes this gives you the instance-hour for free…)

There is some research into checkpointing and spot instances. See for example Monetary Cost-Aware Checkpointing and Migration on Amazon Cloud Spot Instances (Yi, Andrzejak and Kondo, 2012) and Reliable Provisioning of Spot Instances for Compute-intensive Applications (Voorsluys and Buyya, 2012). I’m not myself aware of systems that use heavy-handed state checkpointing. There are quite a few that use spot instances as worker nodes (with <1hour jobs) where the real difficulty boils more into detecting failures and tuning retry timeouts than bothering with any form of checkpointing.

Commentary

It is relatively easy to understand the behavior of spot instances in itself — Bid < Price ⇒ Terminate. The difficulty of using spot instances lies in the fact that it is a market (at least that’s what we’re led to believe) driven by supply and demand and a lot of mostly rational bidders.

We can know how our spot instances behave when the spot market price changes. But we cannot predict the spot market itself.

This means that although you can influence the likelihood of spot instance termination through bidding strategy, you still have to be able to recover from sudden (and massive) spot instance termination.

Did I get that through?

Spot instances and price behavior

2014-03-12T14:00:00+00:00

(You might want first see the introduction to this series of posts if you jumped in here randomly.)

Spot instances and the spot instance market

I’m covering the basics of what spot instances are (e.g. how they differ from on-demand and reserved instances), what is the spot price, what are its characteristics and how the spot price and the bid price affect availability of spot instances (from bidder’s point of view). Finally I’m discussing a famous $999.99/hour instance pricing event.

A lot of the information in this section can be found in AWS’s own spot instance documentation. Most of the graphs have been generated by me using 90 days of spot pricing data from December 9th 2013 to March 9th 2014.

What are spot instances?

For the purpose of computation, spot instances are like any other instance type AWS offers. Where they differ is that you do not have a complete control on the lifecycle of spot instances.

Spot instances can be terminated by AWS at any time.

(Edit 2015-01-06: AWS announced a two-minute termination notice available via instance metadata. You still can’t prevent termination, but you do not get a short notice before it occurs.)

With on-demand instances (the regular variety) and reserved instances you get to choose the lifetime of the instance. With spot instances it is you and AWS who get to terminate the instance. AWS of course plays by the market rules so any loss of spot instances is not arbitrary although it may sometimes seem like so (because not all variables that affect the market are visible).

Why would anyone use spot instances then? Simple: cost. Spot instance prices are variable but on average they offer significantly lower prices than with on-demand prices. With spot instances it is possible to get same savings as with 3 year heavy usage reserved instances offer without the up-front costs.

If you can structure your computing needs around the potential arbitrary instance loss then you can gain substantial benefits from using spot instances. AWS’s own marketing material references to customer cases with 50-60% savings on instance costs.

Spot instance prices cover only EC2 instances. Other instance-related resources such as network traffic and EBS usage by the instance is billed at regular rates.

To recap:

Spot instances are functionally equivalent to other types of instances.
Spot instances may be terminated at any time by AWS.

What is the spot instance price?

First of all, spot instances are priced by instance type, by region and by availability zone in that region. This means that spot market price for m1.small differs from zone to zone even within a single region, not alone between regions.

Neither is spot instance pricing fixed. It varies over time and is determined by the AWS spot market. The market is essentially an auction where buyers (spot instance users) submit bids. AWS determines the spot instance price based on these bids and then

Everyone with a bid higher or equal to the resulting spot instance price “wins” and gets the instances they requested (or keeps them, in case they already exist), and
Winners pay for the spot instance price and not their own bid (e.g. everyone pays the same value which may be lower than their bid).

Note that anyone losing their bid either will not get their instances or will get their existing spot instances terminated.

Spot market is a continuous auction where the spot instance price is continuously updated. The update interval may be anything from minutes to days, depending on the supply of instance capacity and demand for spot instances.

You can see spot market price history in the AWS management console. Here’s a typical graph you can get:

You can twiddle the settings in the UI, but you are limited to 90 days of pricing history.

Finally for the sake of completeness (but don’t worry, this won’t part of the quiz) understand that the actual bidding process is a bit more complicated than saying “I bid for c1.medium at $0.050”. You can bid for multiple instances, specify validity time for the bid, zones to bid in, enable persistent bid requests — and of course you’ll also need to specify all the other parameters needed to launch an instance (instance type, AMI, disks and so on). Finally you can put in as many separate bids as you like.

To recap:

Spot instance price is variable and is determined continuously by AWS based on how customers bid for spot instances.
Each region, availability zone and instance type is a separate market for the purpose of pricing.
Whether you get or lose spot instances is determined whether your bid is equal to or larger than the current spot price.
You pay only the current spot price regardless of your bid price.

How do I actually buy spot instances?

RTFM or watch this video.

Price volatility

Spot prices are volatile — they go up, they go down, they go sideways and all at the same time. I’m no economist and can’t give you an exact definition of volatility, but please take a look at the graphs shown below:

Examples of price volatility difference between instance types (on left) and between different availability zones (on right). Vertical axis is logarithmic, in units of on-demand instance pricing. Solid lines are daily averages and the translucent blocks show the daily maximum price. Different colors represent different availability zones. Faint gray horizontal lines correspond to 4x, 1x and 1/4x price compared to on-demand instance pricing. Click images for larger versions.

From these images it should be clear that there are differences in volatility, minimum and maximum prices, average prices etc. between instance types (left image) where the overall volatility for c1.medium is high over the whole data period, but for cc2.8xlarge there is a clear and persistent volatility drop on January 8th.

There can be significant volatility between different availability zones in the same region (right image) where the price for c1.medium has been pretty stable and low in two zones (zones 1 and 2). This isn’t the case with all of the other zones (3 to 5) where both daily averages (solid line) and the maximum daily price (lightly colored blocks in the background) vary massively from day to day.

Yes, in the graphs above the daily average prices are over 10x the on-demand instance pricing on several days with spikes even higher. In the above graphs the weighted average price for c1.medium instance in zone 2 is $0.0184 and for zone 3 $0.3174. The regular on-demand instance price for c1.medium in us-east-1 is $0.145 per hour. This may give you a WTF moment but see below. I’m going later to discuss one situation where bidding over the on-demand price would have been a reasonable strategy post hoc.

AWS assigns a random permutation of availability zones for each customer account. In plain English this means that my us-east-1a might be your us-east-1d. It’s a common tripping point when comparing metrics related to zones between different accounts. This is also why I omit zone labels from the graphs.

To recap:

Prices can both vary widely, up to multiple times the price of equivalent on-demand instance.
Price volatility can vary massively between instance types in the same region, and between availability zones for the same instance type in the same region.

Instance availability is determined by bid prices

IMPORTANT: All of the graphs below use post hoc analysis. The theoretical maximums on availability and price savings would be possible to achieve only if you can predict the future!

So far I’ve said that spot instances may be terminated at any time the spot price goes over your instance bid price. This doesn’t yet tell us what is the typical expected lifetime (which in turn determines availability) of an instance based on a particular bid.

There is research into algorithms to optimize availability vs. cost. See Mazucco and Dumas (2011), Andrzejak et al (2010), Wee (2011) and Ben-Yehuda et al (2013). The last one (Ben-Yehuda et al) is probably the most thorough in considering price vs. availability tradeoffs. Be careful when interpreting conclusions from these and other papers as most of them use data prior to the July 1st 2011 change of spot market pricing mechanism.

The figure below shows how achievable availability varies with the normalized bid price for two types of instances and availability zones. (I’m calculating availability instead of expected lifetimes just because it’s easier. A value for expected lifetime as well as number of interruptions versus bid price would be interesting, though.)

Example showing theoretically achievable availability versus normalized bid price with c1.medium and cc2.8xlarge in the us-east-1 region. Vertical lines correspond to 1/4x, 1x and 4x on-demand instance price.

This shows that sometimes it is possible to get 100% availability at less than on-demand instance bid price — look at the purple line for c1.medium which hits 100% at bid price of 98% × on-demand price and 99.9% availability at bid price of 32% × on-demand price. But wait, there’s more! Remember that you don’t pay the bid price but the market price!

The figure below is otherwise identical to the one above with the exception that horizontal axis is the relative total cost (… over the whole data period analyzed — the result over any other random range of dates will be different).

Example showing theoretically achievable availability versus normalized cost. Note that even when the bid price might have to be substantially higher than on-demand price to gain 100% availability the total cost can still be less.

This shows that even when you’d have to bid for cc2.8xlarge at about 4× on-demand price to achieve 100% availability, that availability would have cost you less than 73% of the total on-demand instance cost (that is, over the 90 days of the sample data).

Finally, as a bad cost case example see the figure below:

Availability vs. total cost for c3.2xlarge in us-east-1 over the data period. It is not possible to achieve even 50% availability without paying substantially more than for equivalent on-demand instance.

During the time period this data set covers it was not possible to achieve any level of reliability for c3.2xlarge instances without paying substantially more than the equivalent cost for an on-demand instance.

Why would anyone pay >1× rates? During this particular time period there was a shortage of on-demand and reserved c3.2xlarge capacity, so the only way to get such an instance was to bid high in the spot market. This is classic supply and demand equation — at the moment there is very limited supply of this instance type, yet there is demand and people are willing to pay. (Why there are instances available in the spot market while not in the on-demand market is a topic I’ll cover in a later post.)

To recap:

Your bid price determines not only whether you get a spot instance in the first place, but also how long acquired spot instances stay alive.
You control your spot instance’s availability through bid prices.

Instances at $999.99/hour

On September 2011 there was a huge price increase in one zone of the us-east-1 region for m2.2xlarge instances where the spot market price jumped from about $0.44 to $999.99 per hour.

What happened?

Someone had put in $999.99 bid
Spot instance capacity / demand changed rapidly (Dave@AWS: “an sudden increase in demand for On-Demand m2.2xlarge instances”)
Some poor sod ended up paying $999.99 per hour for their spot instances.

To understand why this could happen, let’s try to imagine what the situation might have been in the “bidding pool” (the set of bids on the m2.2xlarge spot capacity) before the price hike in a quite artificial setup with only a few bidders and total supply of just five spot instances:

One possible bidding scheme is to allocate capacity to bids in highest-bidder-first with the final spot price being determined by the lowest winning bid. Thus in this situation the person with $999.99 bid will still pay $0.200 per hour.

If this person now needed three more instances at the same bid price, then the bidding scheme would work like this:

BOOM!

In reality we don’t know why the bid price rose as AWS does tell us how the spot price is determined. It is supposed to be based on some form of auction model, but it might not be. See Achieving Performance and Availability Guarantees with Spot Instances by Mazucco et al which discusses bidding schemes and server allocation policies that maximize the seller’s profit.

The best description on how the spot price is determined is “The Spot Price is set by Amazon EC2, which fluctuates in real-time according to Spot Instances supply and demand” (source). Without AWS disclosing the actual algorithm it is entirely possible that it is not even remotely following the simple auction model described above. It could be really out to maximize AWS’s revenue — in the previous case the algorithm could have realized that at that particular moment profit would be maximized with the absurd $999.99/hour spot price! (Though I do think that this behavior took AWS by surprise too. I think they fell into the theoretical trap of assuming that people participating in market are all rational players, where in reality they often are not.)

This is however pure speculation. From a buyer’s perspective the spot market does work as it statistically does provide cheaper resources than the on-demand market.

AWS has since added a cap on the bid price (see also here), limiting potential accidents like this. The default cap limit is 4x the equivalent on-demand instance price, but it can be increased and clearly has been increased by some bidders (see this graph and note how the maximum and average daily prices have been over 4x several times).

Regardless of the cap you should bid only what you are willing to pay.

For more information on the actual event, please see brandon’s early report in devblog.moz.com, a later analysis by Jonathan Boutelle from Slideshare and Dave@AWS’s responses on the event in the AWS discussion forum.

To recap:

Bid only what you can bear.

Series on AWS spot instances

2014-03-12T13:00:00+00:00

While pondering about whether and why AWS would retire instance types (see here) I started looking more deeply into the spot instance market to see if and what it could potentially tell about instance type retirement.

I started writing one post about the topic. Then I realized it had to be split into two. And into three. So now I have a series of blog posts about AWS spot instances and the spot instance market.

I’ve split these posts into the following topics:

Overview of spot instances and the spot instance market
How to actually use spot instances
Why spot market exists, where do spot instances come from and what drives spot instance prices
Interlude into discussion of spot price minimums
What can the spot market tell about AWS’s capacity changes (in progress)

Most of the information I’m going to present is collected from several sources which themselves have done excellent research and provide excellent advise for spot instance users. I’ll try to give credit to those sources as far as possible without sacrificing readability of this piece.

Now, onwards to the first chapter.

m1 marching into obsolescence?

2014-03-07T00:00:00+00:00

I’m revisiting the topic of my earlier post retiring instance types from a couple months back. You might want to check it out first. It has more pictures than this post.

Ars technica wrote about Intel’s “Powered by Intel Cloud Technology” just two days after my previous AWS post. I couldn’t find a date when this branding program was launched, but the Ars post was the earliest I could find (Intel’s blog has a post from January 15th) so I’m assuming this really was announced in January.

It is clear that AWS and Intel have been working together on this programme for a longer time. It is telling that no AWS announcement on year 2012 is more explicit about processor type than “Intel Xeon”, no announcement from year 2011 mentions processor type at all but year 2013 starts right off the bat with the announcement of the cr1 instance class giving a full lowdown on its processor specs.

There was an Intel PR announcement on September 10th 2013 about AWS’s use of Intel processors but that story does not contain reference to the “Powered by Intel Cloud Technology” program. So something was brewing already in September 2013 but it wasn’t yet given a name …

So it seems that the reason behind AWS becoming more explicit about the underlying processor hardware is due to its relationships with Intel and the “Powered by Intel Cloud Technology” program. I just wonder what kind of benefits this program gives AWS — and as Ars points out, why neither Compute Engine or Windows Azure partake in the program?

If you trawl the Internet archives you’ll also find that AWS did not specify m3’s processor type when they were first announced. The exact processor type was added to EC2 instance description sometime between September 1st 2013 and September 9th 2013.

Okay but how does this buhaha about “Intel Inside” and m3 have to do with m1?

AWS has given a lot of screen estate telling its customers how m3 instance types are cheaper and better and shinier than the old first-generation m1 instances. For example, see the announcement on m3.medium and m3.large types and availability of m3 RDS instances for a few choice words. Alternatively hear what AWS’s chief evangelist, Jeff Barr says: “You get significantly higher and more consistent compute power at a lower price when you use these instances”. Or “compared to M1 instances, M3 instances provide better, more consistent perfromance at lower prices” on the EC2 instance description page itself.

For me this seems like less-than-subtle prodding for AWS customers to move away from m1 EC2 and RDS instance types. But why? Moving your customers to a cheaper platform makes no business sense unless it generates more revenue than is lost due to lower pricing. How could this be true? A couple of possibilities exist:

New instance classes are cheaper to purchase and/or operate (cheap enough to give better operating profit than old instances). Note that the newer instance types fall under Intel’s cloud technology program whereas old ones do not.
There is a desire to obsolete old instance class for some other reason than operating profit alone. (Maybe AWS wants those racks freed for other uses?)

Following the business logic of the first case will still eventually lead to obsoletion of old instance class hardware. Whether it will lead to obsoletion of the instance class is another thing entirely. Yet it is hard to see how the “old” m1 instance class could be kept interesting to customers without reducing its price. But why do that? The only reason would be to squeeze the last cents out of EOL’ed class.

(Of course AWS has the spot market to peddle those less desirable instance types at so-called “market rates” … More on the spot market later.)

Now for the practical advise section! Now after m3.medium was announced it is clear that you should:

Use m3.medium instead of m1.medium.
Use m3.large instead of m1.large.
Use m3.xlarge instead of m1.xlarge.

If you are using any other m1 type than m1.small you really really should go and evaluate m3 class instances instead. They offer better performance at lower cost. (Just don’t do it blindly. Test first. Never assume anything with instances. Trust your own numbers, not others’.)

Which makes me ponder, what of m1.small? I earlier argued that m1.small fills an important sweet price spot between t1.micro and the next type up the line (at that time either m1.large or c1.medium, now to m3.medium). This still applies. There is no m3.small.

At least not yet.

I wonder.

Sockets and concurrency the buggy way

2014-03-06T00:00:00+00:00

Updated 2016-02-16: I have added more details pointing out the exact conditions under which the race condition I describe can be triggered. See below.

I’ll once again share a small gotcha moment from recent programming experiences. This comes from my jab at Erlang programming and concerns about a very subtle bug I introduced into the hypercube node code I was writing.

With subtle I do mean subtle. It took a specific set of conditions to manifest the bug. It had a tiny time window at system startup where it could be triggered and never again after that. I finally could reproduce the bug somewhat reliably by starting a total of 1024 node processes in less than 1/4 second, in parallel, in multiple 16-core physical servers— and even then it showed up in only for one or two network connections out of 10240 connections that were created during the system initialization.

As most bugs go, this is obvious once you realize the underlying problem. For long-time Erlang programmers this might be a known problem and avoided without a second thought.

What I tried to do

So I’m writing this post hopefully help anyone who might run the same problem. But before delving into the actual bug let me first tell what I was trying to do:

I wanted to have a server listening on a port, where
Each new connection would be handled by a spawned Erlang process (e.g. in a separate thread)

There are two ways to process incoming traffic on a socket in Erlang:

Use gen_tcp:recv in a loop to receive input, then process it. This is the typical approach taken to network programming in ~~most~~
all languages.
Use Erlang’s (unique?) method of active sockets where the Erlang runtime will send incoming network traffic to as messages to the socket’s controlling process.

I decided to use the latter method. It fits nicely into Erlang’s view of the world where asynchronous interactions occur via messaging. It also allows nice integration with other processes since you can handle both Erlang-world messages and non-Erlang-world interactions in the same receive loop.

Using an active socket in a network client

Here’s an example Erlang program to connect to port 12345 on localhost, reading data from the socket and printing it out:

main(_) ->
    {ok,_} = gen_tcp:connect("localhost", 12345,
			     [{active, true}, {packet, line}]),
    loop().

loop() ->
    receive
	{tcp,_,Data} ->
	    io:format("Received: ~ts", [Data]),
	    loop();
	_ ->
	    io:format("Error or socket closed, exiting.~n"),
	    halt(0)
    end.

To try this out, put this into a file and, run echo hello | nc -l 12345 in another terminal and use escript to run the script. Of course you need an Erlang installation in the first place.

The program opens a connection with {active, true} socket option. This sets the connected socket into active mode. Incoming data is then processed by loop which keeps calling receive in a loop until the socket is closed (or an error occurs).

Using active sockets in a network server

A socket server with active sockets is also straightforward (except don’t use this code, see below):

%% WARNING: Don't use this code, it contains a race condition. See below.
main(_) ->
    {ok,S} = gen_tcp:listen(12345, [{active, false}, {packet, line}]),
    server_loop(S).

server_loop(S) ->
    {ok,C} = gen_tcp:accept(S),
    Pid = spawn(fun () -> connection_loop(C) end),
    gen_tcp:controlling_process(C, Pid),
    server_loop(S).

connection_loop(C) ->
    inet:setopts(C, [{active, once}]),
    receive
	{tcp,_,Data} ->
	    gen_tcp:send(C, Data),
	    io:format("~w Received: ~ts", [self(), Data]),
	    connection_loop(C);
	_ ->
	    io:format("~w Error or socket closed, closing.~n", [self()]),
	    gen_tcp:close(C)
    end.

This program will bind to port 12345, accept connections on the port, spawn an Erlang process for each connection which in turn will echo all traffic back to the originating socket. Test it out with echo hello | nc 12345.

You might be wondering about gen_tcp:controlling_process, {active, false} and {active, once} in the code:

When a socket is in active mode it will send packets to the controlling process which is initially the process that created the socket. Thus server_loop must explicitly give control of the socket to the connection_loop process.
Similarly we don’t want the server process to receive any packets, which is why the listen socket is defined as {active, false} — this setting is inherited to the accepted socket so it will also start in inactive mode.
Finally, the connection handler sets the socket {active, once} which is mostly similar to {active, true} except it adds flow control to the mix. Which is a good thing before trying to drink from a fire hose …

But it has a race condition!

If you were not dozing off you’ve realized that this version has the race condition I mentioned earlier. The race occurs when code is executed in a particular sequence and the client is sending data at just the right moment.

Below is a figure showing two possible sequences of events within the code, on the left is a sequence with the desired (working) outcome and on the right side is another but possible sequence which doesn’t work (there are a few more “bad” execution sequences, but I’ll use just one as an example):

In the figure red is server_loop code, green is connection_loop code and blue is Erlang’s internal-ish network-ish code handling incoming data for active sockets.

What we want is that the connection handler (connection_loop) will receive all data that is sent to the connected socket. Just like happens in the left sequence — data is received on the socket after socket’s ownership has changed and the handler code is ready to receive data.

On the “bad” sequence the child process will set the socket to active state before the parent process has changed the socket’s ownership. This means that any data received on the socket before ownership change is sent to the wrong process. The recipient will be the listener process and not process running connection_loop code. Oh boy, the data is now lost. (Technically it’s not lost. It is just unread in the message queue of the wrong process. Regardless, it is never read.)

I wrote in the previous post “[Erlang’s] shared-nothing process model removes most problems with shared resources.” Yep. Erlang removes most race conditions on shared resources by eliminating most shared resources. When resources are shared such as on-disk files or network sockets there can still be concurrency problems.

If this would be a real server process with request-response protocol and client-initiated handshake, then the connection would also be stuck permanently (server never sees the handshake, yet client has successfully sent it and is expecting a reply).

I want to emphasize how difficult this bug is to trigger. The remote client will not be sending data until the TCP handshake completes. When listen returns, the TCP stack has already sent a SYN-ACK packet to the client. After it reaches the client it can start sending data, but this will take with any Internet connection anything from a few milliseconds to hundreds of milliseconds.

I instrumented the code, showing that server_loop took (on a MacBook) on average 103 µs (99.9% percentile was 170 µs — these are microseconds e.g. 1/1000th of a millisecond) to spawn a process and hand the controlling socket over. Thus unless the server is massively overloaded it is nigh-on impossible to trigger the bug over the Internet and difficult even on a local network (the network I used has ~150 µs average packet latency).

The next few paragraphs were added on 2016-02-16. My thanks to Robert Gionea for pointing out the distinction between {active, true} and {active, once} in how parent process queue is handled.

Robert’s email got me looking much more closely on the bug and digging deep into Erlang runtime’s internals which means I can now give out the exact conditions what cause the bug to occur — there are two conditions that need to hold so that the code above may lose a packet due to a race condition:

SMP is enabled (enabled automatically on multicore/multiprocessor systems).
Child process modifies socket’s active state while socket’s ownership is being transferred (parent calls gen_tcp:controlling_process.)

See ERL-90 bug report for much, much more in-depth description of the actual underlying problem - it has surprisingly old roots, probably being the side effect of introduction of SMP capability introducing a new failure mode to gen_tcp:controlling_process that was not fully appreciated at that time. The fix discussed below prevents the second condition from happening and thus also prevents the race condition from occurring.

(End of 2016-02-16 edit.)

De-bugged versions of the server code

Fixing this is easy once the problem is identified: just add a synchronization barrier to ensure that connection_loop won’t be called until the parent process has relinquished its control on the socket:

%% Version spawning off a process to handle the connection.
server_loop(S) ->
    {ok,C} = gen_tcp:accept(S),
    Pid = spawn(fun () -> receive start -> ok end, connection_loop(C) end),
    gen_tcp:controlling_process(C, Pid),
    Pid ! start,
    server_loop(S).

Since this race condition occurs only when not using recv and switching controlling process there are also two other ways to write the code so the race condition never occurs. First one is to eliminate the need to use controlling_process by spawning a new process for the listener instead:

%% Version using the current process to handle the connection, passing socket
%% listening to a spawned process instead.
server_loop(S) ->
    {ok,C} = gen_tcp:accept(S),
    spawn(fun () -> server_loop(S) end),
    connection_loop(C).

and the other is to not use active sockets at all:

%% Version eliminating active sockets completely using gen_tcp:recv only.
server_loop(S) ->
    {ok,C} = gen_tcp:accept(S),
    spawn(fun () -> connection_loop(C) end),
    server_loop(S).

connection_loop(C) ->
    case gen_tcp:recv(C,0) of
	{ok,Data} ->
            ...;
	_ ->
            ...
    end.

Concurrency …

This should be a reminder that concurrency is hard. (If you don’t believe me, check what Simon Peyton Jones says about what’s wrong with locks.)

I have programmed in concurrent environments for decades and I do consider myself to be highly skilled in concurrent and parallel programming (multi-thread, multi-process and multi-machine all alike). (And yet I still fail.) Over my programming history I’ve seen that almost all novice programmers and even most senior programmers 1) try to avoid using concurrency in the first place, 2) not realizing when they’ve accidentally created concurrent systems and finally 3) when having to face concurrency they often get synchronization and sequencing wrong (leading to hard-to-find bugs).

Parallel tracks. Get it? Parallel - parallelism? I know, I know … (Image source: Daniel Zimmermann)

I think this makes for a very good case to prefer systems which provide better and safer concurrency programming models. This way at least the most common concurrency problems get eliminated entirely by design.

In modern hyperthreaded multi-core computer architectures the ability to use multiple cores efficiently is a key to high-performance and/or responsive applications and services. In scalable architectures concurrency also is an important tool and similarly a problem (though it comes in different guise through Brewer’s theorem).

Yet performance or parallelism should not be gained at the cost of correctness. For this reason I think that the approach to concurrency and parallelism taken by most languages is unproductive — where the programmer is given low-level primitives (threading, mutexes) and then left to sort the rest by themselves. There should be much better support.

To see some examples of how concurrency and parallelism can be made simpler for programmers see a presentation on multicore Haskell, Learn you some Erlang’s section on concurrency or introduction to Clojure’s concurrency and STM mechanisms (slides).

I don’t think concurrency is never going to be easy, but let’s at least try to figuratively default to giving new programmers a bicycle instead of an unicycle?

Experiences in Erlang

2014-03-03T00:00:00+00:00

I got an programming assignment in a course I was taking. The task was to create an overlay network topology and implement a routing protocol for it with some given constraints — where I quickly realized a hypercube mesh would meet the rating criteria. (This shows my age — hypercube networks were a hot topic in the 90’s. They were used in supercomputers such as the CM-5.)

Unfortunately a binary N-cube routing algorithm is pretty much trivial. Here’s the whole routing algorithm written in Erlang:

find_route(_,_,[],_) ->
    {error,noroute};

find_route(Id,Dst,[Route|Routes]) ->
    if Id band 1 /= Dst band 1 ->
            {ok,Route};
       true ->
            find_route(Id bsr 1, Dst bsr 1, Routes)
    end.

That’s it. 8 lines of code. The first function could be omitted (deducting two lines) as it guaranteed to be never called. (Id is this node’s address and Dst is destination address, each element in the Routes list is a neighbor in the matching dimension.)

Since the actual network problem became trivial you can see why I picked up Erlang. It was for the sole purpose of making the assignment more interesting. I had not previously used Erlang — I was familiar with the syntax and could read Erlang programs — but all the libraries, conventions etc. were new to me. I knew of Erlang’s approach to distributed computing and parallelism and wanted to give it a spin.

So, what I learned? I’ll first summarize its pros and cons from my viewpoint and later elaborate on these:

Pros	Cons
Language and core libraries are compact, consistent and mature Built-in concurrency and messaging Pattern matching Symbols are always sexy Registered processes	Cryptic compile-time and runtime errors Package management Structured data ~ painful syntax No hierarchical namespaces Registered processes

The overall result for me is:

Erlang is a very nice language, it has great features and I’d love to use it again.
… but it won’t become my default go-to language.

Please note that I’m basing this post on my experiences. I might have missed or misinterpreted things that are obvious to other people, so don’t take this post as any kind of gospel truth of Erlang.

Now the long version. Erlang is really nice in several aspects:

The language is compact and consistent and the standard libraries are mature (e.g. well documented and debugged). There’s also a good variety of non-core libraries available which I didn’t have any trouble of using.
Its built-in support for massive concurrency and distributed messaging are just manna from heavens.

Erlang’s lightweight process model just kicks ass. I’ve spawned 15k Erlang processes (e.g. threads) without problems whereas in Python 1000 threads? Forget it (you’ll hit the maxproc limit). 99.99% of the time parallelism is a tool to achieve asynchronous behavior so that case should be as least limiting as possible. Like Erlang does (it runs green threads on multiple native threads, getting the best of both worlds).

Also the shared-nothing process model removes most problems with shared resources. It does make some things more cumbersome and less efficient, but hey, I’m quite willing to trade a little inefficiency to programming with a massively less error-prone concurrency model.
Pattern matching in functions, assignment and conditionals is sinfully easy. Don’t care to handle errors, but still want the process to fail when they occur? {ok,Result} = maybe_failing_function() — if the function does not return {ok,_} the runtime will signal that as an error. And of course guards.
Symbols are always a good thing. Scheme, Ruby and Erlang (among others) do this right. Oh Python, when will you realize symbols are a very useful first-class citizen?
Registered processes. They are very useful when they fit the need, but see below when they don’t.

On the minus side there are a few things that will mean that Erlang won’t be my choice as a default go-to language in the future:

Compiler and runtime errors. So I forgot to make the variable uppercase and now it thinks I want to do pattern matching with a symbol? My bad. But you just should give a less cryptic error message about it.
Package management. Rebar can pull dependencies automatically, but there’s still a world of difference between writing PyYAML into setup.py vs. writing {jiffy, "0\.8\.5", {git, "https://github.com/davisp/jiffy", {tag, "0.8.5"}}} into rebar.config over and over again. This is not a problem for large projects where dependency setup is one-time-only affair, but when doing smaller or one-off programming jobs it would add up quickly to the workload.
Horrendous syntax for structured data. Changing a field? NewStruct = Struct#struct{value=Struct#struct.value + 1} A little syntatic sugar here would do miracles. Yes, you can use parse transformations to help. But that then gets you into another problem of having to first get those parse transforms (see previous) and second to apply them.
Registered processes. These are nice, I would very much want to use them but can’t. The idea is that you can register processes by name, say identify the router process as router and use it directly in messaging like router ! {route,Packet}. Add a supervisor which will re-spawn a failed router thread you’ve got a model where you always know how to reach a working “router” process.

Except registered processes are global. I needed to run multiple hypercube nodes in a single server process, with each having a separate router and separate local message handler and separate remote connectors and separate state manager.

I think this is due to the history of Erlang. It was designed to run in a loosely coupled but purpose-designed system (telco exchanges). In that context it made perfect sense to have a globally identified process. After all, there was only a single system, so why there would be any need for multiples of any single process?

Which does not work in a multitenancy scenario where you have multiple “domains” of processes, where intra-domain visibility is a good thing but inter-domain visibility is verboten.

So I was stuck to using the process dictionary and lugging process identifiers around in function arguments.
Lack of hierarchical namespaces. It is two levels and only two levels. Module and function name and that’s it. So when your frobnizer app needs to have internal module, it is frobnizer_internal_module and not frobnizer.internal.module. Module hierarchy and scoping isn’t only syntatic sugar regardless what hard-core erlangistas say. I personally have found module hierarchy and its close ally, scoping rules a useful feature in other languages. So why not here? I don’t understand the opposition for such a simple and non-intrusive change.

I love Erlang’s pragmatic approach to functional programming, contrasted with Haskell for example, which as a language I simply admire but Always. Find. It. Painful. To. Do. Anything. Useful. Using. It. We don’t write programs (at least mostly) for the pleasure of seeing beautiful and pure programs. We write programs to get things done in a real-world environment where interactions with that non-functional world is the primus motor. So why make that painful?

Erlang is a functional language which understands its purpose of interacting with the non-functional real world. Functional but does not try to whack you with the +4 Mace of Lambda the Pure every time you interact with the world.

Interestingly I see a pattern in my choice of programming languages. The languages I use the most have the following traits:

Good package management with a centralized package directory.
Ability to write quick one-off programs easily (scripting).
…

I could add “nice syntax” etc., but that’s beside the point. I don’t do non-nice languages. I want to retain my sanity. (So goodbye the lucrative MUMPS jobs there.)

In a world where you do not write your own JSON parser, networking library, UI framework, HTTP request processor etc. etc. the ability to easily discover, pull and manage external dependencies is important. My life is too short to waste on libraries and languages which start with “to install, first … then … then” instead of pip install thispackage or even “download, unpack, ./configure && make install”.

Somehow Erlang falls short of my definition of “good package management” and “good scripting”. Not by much, but still.

Retiring instance types?

2014-01-13T00:00:00+00:00

TL;DR: AWS is building an interstellar spaceship.

Amazon Web Services is the canonical infrastructure cloud provider. EC2 beta was announced in 2006 and started with just one instance type: m1.small.

This day there are … a lot more instance types. From the simplified EC2 instance type & pricing page I can now count 27 different instance types: c1.medium, c1.xlarge, c3.2xlarge, c3.4xlarge, c3.8xlarge, c3.large, c3.xlarge, cc2.8xlarge, cg1.4xlarge, cr1.8xlarge, g2.2xlarge, hi1.4xlarge, hs1.8xlarge, i2.2xlarge, i2.4xlarge, i2.8xlarge, i2.xlarge, m1.large, m1.medium, m1.small, m1.xlarge, m2.2xlarge, m2.4xlarge, m2.xlarge, m3.2xlarge, m3.xlarge and t1.micro. Just try to say those aloud in one go!

This profileration is due to (I believe) three drivers: customer demand, enterprise adoption and advances in hardware. This is great, I have no gripes about the usefulness of the new instance types. I’ve had customer cases where “hi1.4xlarge” would have been the perfect solution but just was not yet available. Similarly the introduction of PIOPS and SSDs was a godsend for database-type workloads.

Death of Instances by … errr, actually it’s Takiyasha the Witch and the Skeleton Spectre by Utagawa Kuniyoshi (Image source: Wikimedia commons)

Hardware generations

But what happens to old stuff? What about the old hardware? What about m1.small which has been around for 7+ years?

Currently the AWS instance types can be grouped to roughly three categories:

Shared core instance types (t1.micro and m1.small). Here vCPUs are not dedicated to an instance, but shared between multiple instances (50% for m1.small, no information on t1.micro but I’d expect its CPU allocation to be be both smaller and dynamic).
Generic instances which have 1 vCPU = 1 dedicated core, but otherwise don’t have any particular hardware affiliation — they are primarily defined by (vCPUs, memory, disk capacity) tuple. This includes all m1 and c1 class instance types (AWS specifies m1, m2 and c1 class instances to have “Intel Xeon Family” processor and t1’s as “Variable”.)
Hardware specified instances, e.g. instance types which are defined by particular hardware. This includes g2.2xlarge (“G2 instances provide access to NVIDIA GRID GPUs (“Kepler” GK104)”, from AWS) and c3 class (“Each virtual CPU (vCPU) on C3 instances is a hardware hyper-thread from a 2.8 GHz Intel Xeon E5-2680v2 (Ivy Bridge) processor”) among others. These also have 1 vCPU = 1 dedicated core.

GPU generation gaps

It is easy to see that the last category will pose difficulties in the future. The GK104 GPU is already a previous generation GPU with its successor (GK110) having been in production since May 2013 (both are based on the same GPU family architecture, e.g. Kepler). What happens when GK104 becomes unavailable?

AWS is not going to throw the g2.2xlarge machines to junk heap — after all, they’re going to be deprecated over 5 years. So when GK104 GPUs become unavailable, AWS is likely to keep g2.2xlarge around but crucially it is no longer able to increase g2.2xlarge capacity even if demand increases using GK104 alone.

(With one caveat, see end of this section.)
GK110-based machines can then be introduced as g2.4xlarge or other. Eventually, the successor to GK110 comes around and AWS faces the same situation as above.

(It makes no sense for AWS to roll GK110 GPUs into g2.2xlarge as even the lowest-specced GK110 die has 800 CUDA cores more. Why would they give those away at the same price?)

This leads to interesting economical dynamics. Let’s assume that year-to-year AWS purchases enough g2 class hardware to increase each instance type’s capacity by 100% and each hardware generation is on sale for 2 years, and that each new hardware generation will be initially purchased at same capacity as the previous generation. Looking at the graph below you’ll see that at year 3 the first-generation g2.2xlarge will represent one fifth of the total available capacity. (Caveat emptor: These values are completely arbitrary, so don’t rely on them even if they would seem sensible.)

Note that I’m counting instances, not CUDA cores in the above graph! Also, the choice of 4xlarge and 8xlarge is arbitrary, they could be 3xlarge and 4xlarge equally well. My point is in hardware generations, not per-instance computing horsepower.

Assuming that in year 3 you are need some GPU horsepower, but you would be satisfied with g2.2xlarge. There are two potential outcomes:

There is spare capacity and you’ll get the g2.2xlarge on-demand instance.

Why would there be excess capacity available, assuming that the demand has grown from year to year? A possible scenario is that most of the demand has been satisfied with newer g2.4xlarge and g2.8xlarge instance types leaving the g2.2xlarge capacity underused.

That, of course, is a problem for AWS since they have hardware but no-one is paying for its use. (Opportunistic use of GPUs via spot market can provide some help, but in this case spot prices are going to be substantially less than on-demand prices thus not offsetting the loss of demand completely.)
Demand for GPU instances is much more evenly distributed, most likely due to people trying to pick up most cost-effective instance types. This means that demand for g2.2xlarge is substantial, yet since its total capacity (in this model) is just 1/5 of the total g2 class capacity your request is often blocked since no capacity is available.

This is problematic for you, of course, but it might be a publicity problem for AWS, too (“AWS unable to meet up to customer demand!” would the ~~tabloids~~ blogosphere scream.)

AWS is going to handle this situation somehow. It is possible to control demand via pricing. Future GPUs may support GPU virtualization (Kepler already has some hardware virtualization support) in a way that allows more flexible partitioning of GPU resources between multiple instances. They might do something else. I don’t know.

Anyway. If the first case happens and g2.2xlarge instances go fallow and there’s no demand for them after 1-2 years then how long is AWS going to keep them taking space in racks and costing maintenance effort? In this case there is bound to be a write-off at some point and the whole instance type would be nixed from inventory.

However even if they manage to keep g2.2xlarge instance demand up at some point hardware maintenance is going to exceed the marginal profit gained from keeping g2.2xlarge instances running instead of using the same space, electricity and personnel for something other. Since it is not possible to replace failed machines with identically speced machines (remember, GK104 no longer in production at that point), hardware failures are also going to slowly drain the instance type capacity down too.

So my final point is this: for instance types specified in terms of hardware, it is likely that they have a limited lifetime as that particular instance type (the hardware may live on, repurposed to serve another instance class).

When the underlying hardware becomes unavailable and that instance type’s capacity cannot be increased anymore, its fate is set. The maximum lifetime is the useful lifetime of the hardware (about 5 years), but due to economic reasons it may be also less.

CPU generations

Note that the above reasoning applies also to c3 and other classes that are specified by their CPU type. Yet, for CPUs the situation might be a little different. The t1, m1, m2 and c1 classes already run on multiple CPU generations. As Ou et al. show in their paper Exploiting hardware heterogeneity within the same instance type of amazon EC2 there are several CPU generations with different performance already deployed in AWS.

So for those instance classes which are not bound to a specific set of CPU or disk configuration AWS can just keep adding capacity using the current hardware generation. Yes, some customers get more recent (e.g. more powerful) hardware, but if you are really interested in raw performance these aren’t really your choice anyway.

Eventually c3 class with Intel Xeon E5-2680 will suffer the same fate as g2.2xlarge with Kepler GK104 — that specific CPU won’t be available indefinitely. Will AWS at this point introduce c4, and let c3 keep running as long as it is economically sensible?

Alternatively AWS may choose to re-define c3 to have a physical processor as “Intel Xeon E5-2680 or <whatever is the next generation>” and keep it running with the same caveats about hardware heterogeneity as t1/m1/m2/c1 classes.

One more possibility is that if AWS introduced c4, what would they do with c3 capacity in case its demand goes down? Since the hardware is completely capable of being serving the non-hardware specific instance types (t1/m1/m2/c1) it is possible that AWS decides to move any machines no longer in demand into “graveyard” instance types where the specific CPU classification is not relevant.

What lies in the future?

I have no idea how AWS plans to handle changes in hardware in the long run. Maybe they’ll keep adding new instance classes and types. Maybe they’ll re-define instance class specifications. Maybe something else happens.

While writing this post I came up with the following insights:

m1/m2/c1 generation-to-generation performance gap keeps growing. Eventually that gap between the first and latest generation may become too large so that it will affect their users detrimentally (“What? 2x difference between execution times on same instance type?”) if left unchecked.

The 1 vCPU = 1/2 core (m1.small) and 1 vCPU = 1 core guarantee (others) prevents more fine-grained core sharing in these (unless re-defined). It might be possible that AWS will move older generation machines into serving t1 class instances. They might even introduce t1.small, t1.medium or other t1 classes to supplement the t1.micro instance type (these wouldn’t get any vCPU-to-core matching guarantee, meaning less predictable performance profile) as well as to act as “graveyard” for servers from retired classes or from classes with substantially decreased demand.
c1 class feels the odd ball out. c3.large is better than c1.medium (being only $0.005 more expensive) and c3.2xlarge beats c1.xlarge ($0.040 increase). I don’t really see any reason to use c1 instances over c3 instances.

(Apart from the anecdotal information about low c3 instance availability, which I believe will eventually be sorted out.)

BTW, the same applies between m2.xlarge ↔ m3.xlarge, m2.2xlarge ↔ i2.xlarge and m2.4xlarge ↔ i2.2xlarge instance types. m2 instances have slightly, but not substantially more memory whereas m3 and i2 instances have either more or same number of vCPUs and way faster SSD disks.
I don’t believe low-end instances will be retired any time. AWS needs a broad range of instance types to cover different needs and the t1.micro and m1.small especially fill a need of as-cheap-as-possible instance types for situations with low performance requirements. It might become impossible to keep performance divide between m1.small hardware generations, in which case AWS might redefine m1.small’s performance characteristic upwards and move oldest hardware generations to serving t1 class instances (it is no coincidence that t1.micro’s maximum CPU performance is 2× m1.small’s).

If you want my guess on which instance types will be retired first, my guess is something from c1 or m2 classes. At least unless their prices get substantially cut to make them cost-competetive with m3 and i2 classes to keep demand (and cash flow) up.

How does this affect you?

First of all,

There is no way that the profileration of instance types won’t be followed by some sort of change. Whether it is a cull of instance types, redefinition of their specifications or something completely different, I don’t know. But I know that there is no sensible future where AWS can have gazillion instance types and still keep profitable and themselves and customers sane.
Don’t specify instance types in code. The instance type used for a particular purpose is configuration data (in launch configuration, in configuration file etc.). If c1.medium is going away then you’ll just need to grep config data and not the code (which may construct "c1.medium" as `“c1” + “.” + "medium"` which you won't find with simple grep at all).
Have a policy where production instances must be attributable. If after all configuration references have been changed but you are still seeing c1.medium instances it is super-useful that you can determine what they are for and find the group / person responsible. For this you can use tags like the built-in Name or introduce your own like Unit, Product or Responsible.

If you are concerned about instance availability,

Do not rely on availability of an instance type. So you need more capacity but fail launching the super-price-optimal c3.2xlarge? Fall back and try launching two c3.xlarge instead, or c3.4xlarge or even c3.8xlarge. Or switch to g2.2xlarge or m2.2xlarge. The more heterogenous AWS’s instance lineup becomes the less likely I think there is going to be capacity in all instance types available all the time. (This applies only if you have your own instance management system, since AFAIK this is not possible with AWS auto scaling.)

And finally. My head hurts. Picking an “optimal” instance type is becoming more and more difficult. Yes, it is now possible to pick “more optimal” instance type than before, but finding that “more optimal” is taking more and more time than when there was a smaller number of instance types. (Will AWS abandon instance types at some point completely, allowing you to tune all the CPU/memory/disk parameters freely?)

This is starting to feel like the cereal aisle at the grocery store. Are you going to pick up the müsli with “berries and nuts” or “berries and plenty nuts” or “fresh nuts and berries” or “nut and berry extragavanza” or just “plain” or …

Replay

2014-01-07T00:00:00+00:00

Lately I’ve been reading a book called Replay - The history of video games by Tristan Donovan. As the title suggests, this is a book about the history of video and computer games up to the year it was written, 2010. But it is much, much more than just a record of games of the past!

I’ve read quite a bit about computer, computing and gaming history and I have to say this is one of the best books in that lot. If not the best so far. This is not only because it is well written, nor only because it feels like a proper global history of video games (e.g. not being US-myopic), nor only because it digs and describes trends and social causes for changes in games and markets. It is all that, but most importantly it captures the excitement of the generations of young and focused game developers over the three decades of time covered.

I am not a game developer and probably at my age never will be. Yet I feel connected — as much as the book is a trip down the memory lane (I probably have played 50-70% of the 80’s games listed), it also brings up strong memories of my personal history of late 90’s when I was part of the “new media” internet startup bubble. It brings up memories of the excitement of venturing on completely new and unresearched areas of computing, business and entertainment. About how you can be passionate about what you are doing. And sometimes, what kind of small effect your work can have.

Whether you or someone else will agree with my view on the book is likely to depend on how your own history connects with the themes of the book. For myself the book gave me a whang that I still feel resonating in me! When thinking about all of the pioneers and breakthroughs in games covered in the book I cannot help but feel the same optimistic giddiness right now too. It feels … good.

Freezing Travis

2013-12-18T00:00:00+00:00

This is just a quickie. While working on freezr I decided to take a look at Travis CI, which is a “hosted continuous integration service for the open source community” (as they say).

And wow, is it easy. It is.

In just a few lines of .travis.yml and some clickety-clackety of enabling github hooks to travis made all of new code to be automatically tested in travis. Being free of charge for open source projects just makes it doubly good!

(Which reminds me, I’ll have to attach the OSI-approved license to freezr. It is open source, but I haven’t just gotten around to writing the boring licensing stuff…)

There was … well. Why am I always getting a gotcha moment? Am I just somehow abnormally suspectible to finding corner cases?

Anyway, here’s the .travis.yml file:

language: python
python:
  - "2.7"
env:
  global:
    - PATH=$PATH:$TRAVIS_BUILD_DIR/node_modules/.bin
install:
  - npm install less coffee-script
  - pip install .
services:
  - rabbitmq
script:
  - make actual-test

The gotcha is getting node’s local install bin directory into PATH environmental variable. Travis by default does have ./node_modules/.bin in PATH so unless you change the current working directory you have no problems in running npm-installed programs.

(Of course I did change the working directory during tests.)

So if you do npm install in Travis, keep in mind that by default the non-global NPM install bin directory is not necessarily found via PATH. That it works by default is a happy coincidence, not a guarantee.

(I could do sudo npm install -g, but I try to avoid changing global system state unless absolutely necessary.)

Ember - (A)bort, (R)etry, (F)ail? R

2013-12-11T00:00:00+00:00

Earlier I wrote about problems I had while trying to develop an Ember.js application with a Django REST framework-based backend. I did some research (I’ll get back to other results from that later) and tried using AngularJS for browser-side development, but it didn’t work out too well. I checked some other client-side frameworks but I really, really wanted to have a good model representation in the browser side code including relations between models and I couldn’t find one that felt right.

Eventually I decided to give it another go with Ember. I had an earlier semi-static UI mock that I extended using Ember and static fixtures. Which despite the steep learning curve eventually worked

~~great~~

well enough.

Though I could not postpone the dealing with the backend indefinitely.

I decided to ditch ember-data-django-rest-adapter completely, the main reason that I didn’t understand how I should format the backend response just by looking at the code (and no docs on that, unfortunately). It might be just the greatest thing since pre-buttered bread slices, but I just couldn’t understand how to get it working with the backend framework I was using even when it is by name supposed to work with it. Doh.

This is an after-the-fact reconstruction from memory on how I progressed:

Attach a custom adapter based on DS.JSONAdapter (e.g. set application’s ApplicationAdapter value ).
Try to understand what an adapter does and what a serializer does.
Create a custom serializer. Wonder why some of the methods don’t get called. Realize that should have used REST* base classes.
Change adapter and serializer to back from DS.RESTAdapter and DS.RESTSerializer correspondigly.
Hack hack hack …

Eventually I got an adapter and a serializer with only a small number of minor changes compared to the original DS.REST* versions:

Custom extractSingle and extractArray methods (which are called indirectly by DS.RESTAdapter.extract) that don’t look for subkeys, but use the payload value directly (as a direct value map, or an array of value maps, e.g. [...] vs. {"objects":[...]}).
keyForAttribute and keyForRelationship which turn Ember Data convention camelcase field names into underscored JSON data keys (from instanceId to instance_id).
pathForType that doesn’t do pluralization of resource name (e.g. project resource list is at /project and individual resource at /project/1).

(I still have to find a way to include the trailing slash in requests, Ember Data seems to be stripping them away, what causes extra redirects with Django REST framework. Or just specify trailing_slash=False for the API router.)

And that’s it. Total size is about 20 LOC. I’m pretty surprised about that the minimal changes needed over DS.REST* classes. What I have not done is saving models to the backend — the code might be missing functionality to make that possible.

You can check the code out yourself at github. At the moment the client-side UI code is in freezr/ui/static/freezr_ui/coffeescript directory.

P.S. I had one major gotcha while doing this. I’ve documented that one in an another blog post.

Gotcha!

2013-12-11T00:00:00+00:00

(Note: The code examples below use coffeescript instead of plain javascript. If you don’t know coffeescript here’s a quick cheat sheet: @foo ≅ this.foo and () -> stmt ≅ function () { stmt }. Additionally text in curly braces {{…}} is Ember’s templating language.)

Finnish road sign number 122, “Two-way traffic”. (Source: Wikimedia Commons)

While doing a retry on Ember for freezr user interface, I hit a problem I’d like to share with you. I didn’t find help on the internet on this so I hope if someone hits the same problem this post will help.

Anyway, I hit one major gotcha that had me scratching my head for a long time. I had used ember-time as a basis on how to implement a “since state change” time display. Converting the original code to coffeescript was straightforward (but see below for an update):

App.FromNowView = Ember.View.extend
  nextTick: null
  tagName: 'time'
  template: Ember.Handlebars.compile '{{view.output}}'
  output: (() -> (moment @get('value')).fromNow(true)).property('value')
  tick: () ->
    @nextTick = Ember.run.later this, (() ->
      @notifyPropertyChange('value')
      @tick()), 1000
  willDestroyElement: () ->
    Ember.run.cancel @nextTick
  didInsertElement: () -> @tick()

and it was used like this:

{{view "App.FromNowView" valueBinding="stateUpdated"}}

Which worked great when the page was first loaded but it failed to update the time view after updates. I was really really confused. The state value was itself updated in the rendered view correctly immediately after Project.reload() finished, but text derived from stateUpdated field was not. WTF?? This is what was happening in the browser:

Top row is what happened in the UI and the bottom ones showing what the server actually sent to the client on state change from running to freezing to frozen states. Why is it stuck on “for 4 hours”?

Time to debug. So,

I checked the JSON response. Yep, it had the correct, updated value.
I wondered whether the name was somehow conflicting (it was originally stateChanged), so I renamed the JSON field and model field. No effect.
I put tons of log output statements in Ember end Ember Data code. This was a great learning experience in itself, as now I have a lot better understanding how Ember propagates value changes. Nice stuff, I think. However digging deeper and deeper I kept seeing that the updated value was being passed correctly along, yet still refusing to show up in the actual web page.
I wondered whether the date attribute type was doing something fishy and switched to string instead. No effect, the “bad” value persisted.
I searched the net high and low to no avail.

I started to do voodoo coding. Poking at things and hoping the problem is mysteriously fixed.

Finally I added logging to DS.attr’s use of Ember.computed and …

… all was made clear to me.

All of the other fields were getting the value from @_data element (which contained the updated values set by DS.Model.setupData) except — except for stateUpdated which got its value from @_attributes!

At this moment I remembered what I earlier read about Ember bindings. And that there was a difference between normal bindings and one-way bindings. And that the valueBinding="stateUpdated" did a binding on App.FromNowView.value to Project.stateUpdated. And that this was a normal e.g. two-way binding meaning that updates on Project.stateUpdated are propagated to App.FromNowView.value and vice versa.

I was not getting the updated value from JSON response because I had already overwritten it myself.

This is the offending line:

@notifyPropertyChange('value')

This doesn’t actually change the value of value, but Ember doesn’t know that so it propagates the event to the bound field of Project.stateUpdated, which eventually results in Project.set('stateUpdated', «value») where the new value was actually the old value. I’ll try to put this into a picture.

In the figure below I’ve used green for events initiated by Ember Data and red for those initiated by App.FromNowView and the gray arrows show bindings between different Ember-controlled values. I refer to objects by their class names, so Project.stateUpdated below is not a class field but a field in an instance of Project class.

In the template the statement valueBinding="stateUpdated" creates the two-way fat gray arrow binding (top row). The binding from App.FromNowView.value to App.FromNowView.output is a one-way binding and comes from the use of property('value') on the output function (right column). Finally the App.FromNowView.output binding to {{view.output}} comes from somewhere deep inside the templating system (bottom row).

The initial value is loaded by Ember Data and is propagated from top left corner by the green arrows. First, Project.stateUpdated is changed, which then propagates to App.FromNowView.value, which in turn causes the value of App.FromNowView.output to change, which finally causes the {{view.output}} template to be (re-)rendered. This will in turn cause the get chain to propagate back in the chain, finally resulting in the nicely formatted time delta value to be written into the HTML page for user to see.

This is where the call to tick messes things up. It will be called every second, and it will call notifyPropertyChange('value') which in turn causes two propagations to occur — one back to the original Project.stateUpdated value thus overwriting it, and the other to propagate to the output template. This meant that the output value was correctly updated as time passed, but any change in the actual stateUpdated value as reported by the backend was not reflected in the human-readable output.

(I’m not sure, but I think Ember’s idea is that since I’ve overwritten the values myself it will keep them around until I call either save or rollback. I’m not sure whether it is sensible to call reload at all when you have uncommited changes in the model.)

Now that I had understood the true problem the solution came immediately. In the application I just wanted to ensure that updates on the bound value are propagated to App.FromNowView.output, which was already automatically updating when the bound value was changed. It also has to be refreshed as time progresses (“a few seconds” → “a minute”) which does not need to refresh the bound value, just the output value. The correct update sequence where display updates do not affect the actual state update time value is shown in the picture below:

Now tick will only cause the rendered value to be updated while all changes in the original model are also honored. The change is trivially simple with changing the property change event fired on the output element:

tick: () ->
  @nextTick = Ember.run.later this, (() ->
    @notifyPropertyChange('output')
    @tick()), 1000

With this simple change everything was finally made good!

So what’s the lesson learned? When using Ember, you need to understand how a value is bound, to where, and what type of binding makes sense for any particular situation. Also don’t use @notifyPropertyChange indiscriminantly on values that are bound from outside the caller’s control.

Update: Ember-time itself has since been fixed. You’ll need to look at bf3383c6 or earlier commit to see the original version.

Idempotent PUT is a fake

2013-12-05T00:00:00+00:00

Previously I poured my thoughts on REST/JSON protocol differences. I am still researching on how different server and client frameworks work, but as an interlude I’ll comment on the interpretation of the PUT operation in relation to its use on “RESTful” APIs.

I’ve seen a lot of people state that PUT /resource/<id> should create the resource if it does not exist. Like here, here and here and here and and.

This is absolutely wrong. This is a misinterpretation of idempotency. Following this logic to the extreme causes both semantic and practical problems.

Idempotency

I am making a strong statement here regarding PUT semantics here, so let me first introduce you to the idea of idempotency. I’ll quote from the wikipedia entry on idempotence:

Idempotence (/ˌaɪdɨmˈpoʊtəns/ eye-dəm-poh-təns) is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.

And here is a small light switch system which has both idempotent and non-idempotent functions:

(define lights-state #f)
(define (lights) lights-state)
(define (lights! on-or-off) ...) ; sets lights on or off
(define (lights-toggle!) (lights! (not (lights))))

If you repeatedly call lights, you’ll get the same value every time. The getter is both safe (no side effects) and idempotent (returns same value on repeated calls). Similarly lights! is not safe (it has a world-changing side effect) but is idempotent:

> (lights! #t) (lights)
> #t
> (lights! #t) (lights)
> #t

(lights-toggle!, of course, is not idempotent.)

Now you are asking me what’s in the lights! function I didn’t show you earlier. I’ll show you now:

(define (lights! on-or-off)
  (if (and (boolean? lights-state) (boolean? on-or-off))
      (set! lights-state on-or-off))
  lights-state)

This is an idempotent function. As long as lights-state stays boolean (guaranteed if only lights! or toggle-lights! are used to change light state) it will change the value of lights-state to match the request.

Now the surprising bit. If lights-state is not a boolean value, lights-state is still an idempotent function and lights and lights! are too!

Now consider a multi-user system (aka real world) where this happens:

Me> (lights! #t)
=> #t
Elsie> (set! lights-state 'explode)
Me> (lights! #t) ; just making sure
=> 'explode

Boom! What happened? Wasn’t lights! supposed to be idempotent? Yes, and it still is. But wait, I thought that idempotency means that any idempotent operation should work the same if repeated later!

Let’s go back to wikipedia entry and scroll a bit down:

A composition of idempotent methods or subroutines, however, is not necessarily idempotent if a later method in the sequence changes a value that an earlier method depends on – idempotence is not closed under composition.

“Not closed under composition”. Technically, when you call a function (method, procedure, script, whatever) in a real-world situation its result is a composition of the current system state, inputs you provided and the function implementation itself. Idempotency guarantees that any changes to the system state by the idempotent operation are such that calling the same operation with the updated system state will result in the same final result as calling with the unaltered system state.

What it does not guarantee is that if you call with some other system state you would get the same results. If anyone else has changed the system state between your calls to the idempotent routine, then the system state has changed and there are no guarantees that the result from your call will be the same. This is exactly what happened, Elsie changed the system state, so even though the lights and lights! functions are still idempotent, my operations from my viewpoint are not since the two calls were composed differently.

At this point you should realize that when standards talk about idempotency or behavior of repeated PUTs they are not guaranteeing you that all your PUTs will give the same response or have the same effect in the system every time under all conditions. What the operation idempotency guarantee can give you is that when the composition of your PUT has not changed (apart from the changes the original PUT made), subsequent PUTs should give you the same result. But only when that assumption holds, otherwise we are not talking about idempotency at all.

PUT doesn’t have to create resources

The normal life cycle of any object, entity or resource within computer systems is:

It does not exist.
It is created.
Stuff happens to it.
It is destroyed.
It is no more.

Interpretation of POST and DELETE operations are straightforward if you think of them as steps #2 and #4 respectively. They manage the life cycle of the resource. The resource exists between creation and destruction, and otherwise exists not.

If we take the viewpoint that these are the only operations to manage a resource’s lifecycle — and I urge you to take this viewpoint too — then PUT is valid only during step #3.

That is, PUT should not create a resource.

Now I can already hear an argument in the line of “but using PUT to create new resources is an idempotent operation” and you are right. If you define PUT to create non-existent resource and update an existing resource, then two sequential PUTs will always get the same result (even if the resource didn’t exist in the first place). But this is circular reasoning. You can’t argument that PUT should be a life cycle operation because it can be one while staying idempotent. We can define that PUT is not a life cycle operation and it still stays an idempotent operation (PUT on non-existent resource would result in the same result both times – a failure).

At this point it should be clear that saying that “PUT should create resources because of idempotency” is a false argument because idempotency holds even if this is not the case.

Which way PUT swings is a design choice. A choice.

I want to convince you that it should not create resources.

(Ed: Changed “strawman argument” to “false argument” above. Thanks Frederic for pointing out the semantic difference!)

PUTs on DELETEd resources

Now I’ll try to convince you why PUT as a life cycle operation is not a good idea from developer’s perspective because it just causes practical implementation problems (and if you are not aware of these, it can create hidden semantic traps in your system).

This is a real-world case (simplified though):

User 1 creates a message (POST).
User 1 edits the message (GET, PUT).
User 2 sees the message and decides to open it for editing (GET).
User 1 decides the message is crap and removes it (GET, DELETE).
User 2 updates the message (PUT).

If you allow PUT to implicitly create non-existent resources, you get what I’d call semantically inconsistent result. For users, the message exists when it should not. This is entirely consistent from system’s point of view, since the message created at step #5 is not the same message that was deleted at #3.

Unfortunately most of the systems that are written are meant for human consumption and need to work with human expectations. Thus in this case the implicit PUT most definitely was definitely not helping system development at all.

Oh no, wait! Here’s another!

User 1 has CREATE permission on messages.
User 2 has EDIT permission on messages.
User 3 has REMOVE permission on messages.

I think you can already guess where this is going. If there’s implicit create on PUT I have to check for CREATE permission in two different places, both POST and PUT. (This is another real-world scenario where some people can CREATE and EDIT, others can only EDIT and some DELETE but not create or edit. Auditability requirements…)

What then is PUT?

Simple:

It is idempotent. (See above on limits.)
It operates on existing resources.
It is not a life cycle operation. It cannot create or destroy resources.

Idempotent PUT still stays the very same and very powerful and useful feature as before as it allows you to just repeat the request in case of transient network or server failures. Just please don’t think PUT as a life cycle management operation, because it should not be.

REST MESS

2013-12-04T00:00:00+00:00

While working on a hobby project called freezr I came across a few assumptions I had made which turned out to be wrong. I’m going to write a bit about these assumptions, since I found solving the resulting problems very frustrating.

I had decided to write freezr API-first instead of UI-first. The reason for this decision was based on that

I had a very good understanding of the problem and what kind of actions it offered to the user, so there was no need to research the problem through UI prototypes etc. (If you do not have a good understanding of the problem, you should always start with UI mockups and prototypes.)
I am very much in favor of delegating the web server to provide interfaces to core logic and let the browser be the UI (e.g. HTML5 web application). This means that “web server’s” role is really to provide an HTTP API to the core services, and the only bit of user-visible “web serving” happens when it bootstraps the browser-based application.
I am not a UI designer nor UI developer. I’m much more a service architect and developer. I can do UIs that are best described as “engineering UIs”, functional, but not pretty and definitely not having thought too much about usability. In particular, I was hoping to get someone else to actually do the UI bit for me — another reason for postponing UI development.

So I worked on freezr for a few weeks, on and off, and got it to a situation where the service itself was functional (albeit lacking a lot of production-level stuff like authentication, access control etc.) and passed quite a lot of unit and integration tests. The integration tests drove the service via its defined REST API alone.

I wrote the server using Django. I had a few reasons to pick up Django, one of them being familiarity with it. That’s familiarity, not liking. I don’t like Django that much, I’ve struggled with it in the past, but I still find it a quick way to get an web app from zero to development demo. Albeit, it is always a bit of a frustrating experience. I could have used Flask, but I’m not as familiar with it, and the times I’ve used it I’ve found writing quite a bit of boilerplate code for things that come as default in Django.

(As a side note: I don’t like node.js, so I’m not going to use Meteor or its ilk. I find it frustrating to write in a language that has practically zero thought given towards developer friendliness, orthogonality or understandable error and exception handling. If I could decide, I’d replace JavaScript with a standardised and well designed bytecode interpreter where browsers would provide a JavaScript-to-bytecode compiler shim for backwards compability. It could even use LVVM representation directly. This would give it much better re-targetability from other languages.)

Anyway, I ended up using Django with the Django REST framework. I have worked with TastyPie and found it superlatively frustrating experience when trying to do anything “out of the tastypie box” so I was absolutely sure that I would not touch it even with a long stick (unless it had a very, very sharp end with a nuclear option installed). (TastyPie might have gotten better since, so you shouldn’t take my opinion as anything else than an opinion.)

So I wrote a REST interface using REST framework. I think it ended up nice and orthogonal. I especially liked the way how the framework made it easy to provide URIs for resource references. Like this (edited for brevity):


GET /api/account/1/
HTTP 200 OK

{
    "id": 1,
    "domain": "http://localhost:8000/api/domain/1/",
    "name": "AWS account",
    "access_key": "AKIAJH3LIPN74P3XO3UQ",
    "active": true,
    "projects": [
        "http://localhost:8000/api/project/1/",
        "http://localhost:8000/api/project/2/"
    ],
    "regions": [
        "us-east-1"
    ],
    "instances": [
        "http://localhost:8000/api/instance/13/",
        "http://localhost:8000/api/instance/14/",
        "http://localhost:8000/api/instance/15/",
        "http://localhost:8000/api/instance/16/",
        "http://localhost:8000/api/instance/17/",
        "http://localhost:8000/api/instance/18/"
    ],
    "updated": "2013-11-29T13:19:17.963Z",
    "log_entries": [
        {
            "type": "info",
            "time": "2013-11-28T20:58:19.900Z",
            "message": "Regions changed",
            "details": "Added: us-east-1\nRemoved: none"
        },
        {
            "type": "info",
            "time": "2013-11-28T20:58:38.929Z",
            "message": "Refreshed 1 regions in 1.15 seconds, total 6 / added 6 / deleted 0 instances",
            "details": null
        },
	...
    ],
    "url": "http://localhost:8000/api/account/1/"
}

Using URIs for resource references makes the whole API theoretically to have a very nice property: as long as the “root” point is known, it is possible to find all resources in the system without any need of the resource URL syntax. The interface itself will tell you that instance 15 is located at http://localhost:8000/api/instance/15/ without you having to know anything about the URL structure. For all you care, you could have instance 15 in a completely different URL from other instances like http://fnord:6643/ISTORE.JCL/?iid=i-6a56cd3. You, as a web browser application programmer would not have to do anything to support distributed resources!

I just love the idea. I though the REST API was just what REST is really meant to be — simple, using HTTP primitives, clean URLs, with the whole data model traversable without having to know about the particular service’s URL structure (the web server would tell the root URL during bootstrapping).

Hooray. Time to go do some UI development.

For the UI side I decided to try out Ember.js. I knew its data layer wasn’t yet final, but I thought, what the heck, I’m doing pretty simple REST API here, that shouldn’t be a problem.

It was.

This is not Ember’s fault in itself. It is just that Ember’s REST interface is designed to work with a particular flavor of REST interfaces. The REST API that I had defined did not conform to this model. I searched the net for a solution, and found ember-data-django-rest-adapter which … didn’t work out too well either. It is not final either so I should not expect too much, but it had the same problem as with Ember’s default REST adapter: it was making a lot of assumptions about the REST protocol. In particular, it didn’t work with resource URIs. Well, no problem, I can just HyperlinkedModelSerializer to get IDs instead. And it wanted to pluralize resources in URIs, e.g. a project was fetched from /api/project/ID/ but list of projects from /api/projects. Oh god. Then I found it actually was expecting hasMany relations as [{"id":1},{"id":2}] and not just [1,2].

No, I’m not going down that rabbit hole.

If there is competition for most stupid convention ever, I’d nominate the idea that computers are required to pluralize human words when using a computer-oriented API to distinguish between fetching a resource versus many resources. Quick, what’s the plural of “locus”? What if your API describes shoe pairs (e.g. shoes), is the resource point for fetching records of many shoes then “shoess”?

Frustration and amazement.

I came to realize that:

Most of the backend to browser development is done in a tightly linked manner. They are collectively developed and either both of those work well together (you picked rails-friendly framework for rails backend), or either the browser side gives in (custom resource adapter) or the server side caves in (doing whatever is required for the responses to conform to client expectations).
There is no universal “way of doing REST”. I though I understood this, but I had just thought the disperancies were in resource access and action definitions, not so much in how the resources are serialized and deserialized to/from JSON format.

(Example of different action definitions: In freezr, a project is frozen with a POST to /api/project/1/freeze/. Another and entirely valid choice would have been to apply PATCH to /api/project/1/ with content of { 'state': 'freezing' }, where instead of defining an action, the request would declare the desired state.)

In reality there are many ways to do these, and most frameworks are designed to work only with one particular REST protocol without thought given to reconfigurability for different use cases. (The configuration of REST adapters mostly concerns with endpoint URL and what combinations of operations is used for different idioms like is partial change PUT or PATCH, can you POST over an existing record?)
I don’t know jack shit.

To fix the last problem I’m going to do some research on different REST interface patterns and which server- and client-side frameworks use then, and write a follow-up blog post on what I find out.

Working on freezr

2013-11-29T00:00:00+00:00

Just a quick post — I’m working now on something I call freezr. It is is a) a hobby project and b) based on a real world need. Since it’s a hobby project it is also naturally an open source project (or at least will be when I get around tacking license information into the repository) and located, of course at GitHub.

My old flame ... from 17 years past

2013-11-03T00:00:00+00:00

Before I get about talking old stuff, here’s a few important suggestions for anyone who came here looking for my old MD5 Java implementation (in case you don’t know, MD5 is a cryptographic hash function):

Firstly, do not use it. There are plenty of good alternatives starting from Java’s standard library’s java.security.MessageDigest up to separate open-source implementations such as org.bouncycastle.crypto.digests.MD5Digest.
Secondly, do not use MD5 at all. MD5 has not been considered secure enough for new applications for over a decade now. Use SHA256 or better instead. It’s of course another issue if you’re working with legacy protocols, but for any new implementation you just do not use MD5.
Thirdly, if you are planning to do what seems to be every new web site programmers thing, that is, when you’ve come to realize that storing plaintext passwords is bad and have come up with the idea of storing passwords as hashes, and were thinking of using MD5 for that — and of course now you’d be thinking of SHA256 instead — do not use a cryptographic hash function for password hashing. Use a hash function specifically designed for long-term secure password hash storage such as PBKDF2. Just don’t use a plain hash function for password hashing. Trust me.
And of course if all of the above is new news for you, please please please get some education. I pathologically hate closet cryptographers e.g. people who think they know everything about cryptography since they’ve finally succeeded in breaking out of a wet paper bag.

The reason for this post is that I still keep getting e-mails for support and questions about an MD5 Java implementation I wrote a looooooong time ago (1996). It is so long time ago that in a few years people starting their professional programming careers will be younger than that piece of code.

(Oh boy, am I old.)

So if you have come here to report a bug in my MD5 implementation or have a question about how to use it, now you have the full story why it is effectively abandonware (which it sort of isn’t, since it is under LGPL and has thus been incorporated to many other codebases) — and that it is usable in and relevant to only a very narrow set of programming problems, which yours most likely is not one of them.

I don’t even have the source online anymore. I find keeping it online pointless for the reasons listed above. If you are super-interested in the source, just search for md5 paavolainen, although a lot of the hits are actually derived (mostly better!) implementations.

Why did I write an MD5 Java implementation in the first place?

(Imagine the following spoken with a hoarse, oldtimer voice, worn rough by the dust inhaled by years in solitude service among racks and racks of ancient servers.)

Well, this happened in 1996 when Java was still at version 1.0.2 and did not have java.security package at all (it came along in 1997). I just needed MD5 in a browser Java applet for the purpose of signing a request sent to backend server (implemented in C as a module to CERN httpd, oh the times) on a research project into micropayments. I didn’t find a ready-to-use MD5 Java class (the Internet was much smaller back then) so I wrote one following almost 1:1 the RSA C-language reference implementation. (The micropayment project never got anywhere practical, but that’s a whole another story…)

(Back to normal voice.)

P.S. I’m not saying that I’m offended by people e-mailing me about the MD5 class — on the contrary, it does feel good to know you have made - albeit very small - but a lasting contribution to the “great internet of buzzwords”. What I sincerely hope for is that those e-mail would stop, not because I don’t like them, but because MD5 in general, and definitely not my unmaintained implementation of it should not be used anymore. Move along, don’t dwell in the cryptographic past.

P.P.S. I’ll be more positive on the next post, I promise.

Life's easier if you can code

2013-10-28T00:00:00+00:00

If you have never ever programmed anything, you can find the title of this post strange. Are programmers somehow superhuman, capable of lifting railroad cars with their bare hands? Are they more intelligent, more capable than other people? Or is there a secret cabal of programmers where by joining you’ll get secret discounts at electronics stores and easier promotions at work?

Oh I just wish even just the second to last would be true, but alas, none of the above.

Being a programmer does not make you fitter (strangely often the opposite), nor stronger. But it does help quite a lot in many things. It’s also possible to do some really cool things if you mix in with some physical world stuff with the programming. However that’s not the kind of “making life easier” stuff I’m really talking about.

I was inspired to write this post because I’m trying to sell some stuff, mostly old magazines boxed up (why I have kept them in the first place, though?). There’s a nice free-to-advertise craigslist-style site used here in Finland that I’ve used before but. The but is there’s a length limit on the ad and I’ve got tons of those magazines to sell. Itemizing them goes over the length limit many times over.

So what do I do?

I whip up a Python script using Genshi formatting an YAML input file. The output is a bunch of text files, broken down by magazine names. Here it is, a total 9 lines of code:

#!/usr/bin/env python3
import genshi.template, yaml

data = yaml.load(open("data.yaml"))
tmpl = genshi.template.NewTextTemplate(open("template.txt"))
i = 1

for mags in data:
    names = list(map(lambda e: e['name'], mags))
    result = str(tmpl.generate(magazines=mags, names=names))
    print(result, file=open('out/' + (", ".join(names)) + '.txt', 'w'))
    i += 1

Time spent:

Script: 10 minutes
Writing text template: 5 minutes
Reformatting data to YAML: 30 minutes

Not bad — I did have the magazine data already available (text file, needing some reformatting for YAML) so this went quite nicely. After this effort I basically need just to log into the classified site and copy-paste the data file by file. (If I had really literally tons of ads to place I’d scripted the uploading part too, but I have just tens. Not worth optimizing that.)

Eventually I had to reformat the output several times before the site grokked it. I would hate even the potential of having to re-do something that tedious by hand, so I’m positive about the result.

There is great benefit in optimizing repeated tasks (xkcd has a nice illustration about it). Here I needed the script only once, so I’m not sure whether I came out ahead time-wise, but definitely I didn’t get to experience the tedium of doing so.

Come to think about it, the reason I did use templating was probably to avoid a tedious task by turning it into a programming problem. Writing a small script to do the task at least gave me a feeling of being productive even if it might not have been so.

Ha! Maybe that’s it:

Being able to write programs makes life easier by allowing you to turn (some) tedious tasks into interesting programming problems.

I’m happy with that.

P.S. So what about Excel? I do actually find spreadsheets quite useful as an miniature programming platforms when the data I’m manipulating is already in tabular form. Doing =if($B4<>"";TRUE;FALSE) and copy-pasting it over a row is often faster than writing and debugging an imperative program.

One viewpoint on cloud computing

2013-07-10T00:00:00+00:00

Recently I was consulting a client on cloud strategy. When we were trying to explain to the client how the risk landscape with growing adoption of cloud computing (being the case that it affects them even if they don’t themselves use cloud services) … I had an idea.

An idea that I think gives some insight why enterprises and especially IT companies were slow on cloud uptake and why small and agile startups were quick to take up on it.

Before I get to the actual idea, I need to go through some background information first. If you’re super duper familiar with risk management in IT service procurement, feel free to skip ahead.

Bloody long introduction

So, you know what risk is? Wikipedia puts it this eloquently:

risk = probability × loss

That is if you have a 0.5% yearly probability event that costs you $1M, and another with 50% probability and a loss of $10,000 these are crudely equal with expected yearly losses of $5,000 for both. So you’ll take both the probability of a bad thing happening and the consequences of that thing happening together as a risk.

Caveat emptor: This is only one viewpoint on risk.

This view of risks comes with an attached, implicit viewpoint. It is viewed as my risk. For example, the risk to me of your house catching fire is neglible (being non-neglible only if you happen to live within 100 meters), because the loss to me of your house burning is zero → my risk is zero too.

In larger businesses and government agencies it is common to push enterprise’s risk (“my”) to the vendor (“your risk”) through contractual means. In areas of IT service procurement this means the service provider assumes liabilities on not meeting service level agreements / deviating from rules and regulations / other failures. In exchange to .. well, of course, higher fees.

So, the risk probability can be divided into two components: mine and yours.

The service provider may mitigate its risks by many means. It might employ quality process models and employ good quality hardware as well as cover residual risks with insurances, for example. (More cynically oriented might expect the vendor to not do so.)

Anyway this does provide two more aspects to consider when understanding the loss component. For this discussion I’ll split it into methods and means. Processes and hardware, if you like.

So we get to:

risk coverage = (me ⇆ you) × (methods ⇆ means)

Ignore the pseudo-scientific notation for a minute. What I mean here is:

(me ⇆ you)

You can push risk probability to someone else, or handle it yourself. Not surprisingly corporate and governmental organizations tend to push risks away from themselves. After all it is easier to say “We had a contract with ACME Corp. to cover all bases! They fucked up!” than to “It was our fault.”

You can always count on people in large entities to cover their asses without regard to global optimum - it’s not their money, after all.
Methods and means relate .. here’s an example.

One network security risk aspect is the management of firewalls. To have good security you need to have good processes to ensure that only the minimum set of required holes are used, knowledge to understand the security model, an audit trail of changes, and so on.

Good processes don’t mean anything without some means to turn those processes into the desired action. In this example that would require an actual network firewall (hardware). You can wish and design and change manage all you want but without a firewall those policies would have zero effect (e.g. you’d have no network security, or alternatively no network, both of which would be bad for business).

Finally we are getting close to the cloud. So bear with me.

The “traditional” way to manage IT service risks was to let the service vendor handle the risk. The risk coverage model was a bit like below (with dashes on non-relevant things):

my risk coverage =   (me ⇆ ---) × (------- ⇆ -----)
your risk coverage = (-- ⇆ you) × (methods ⇆ means)

(Note: I’m not sure whether the word “coverage” is a good choice here. Can’t figure out anything better, though…)

When shit then did hit the fan it was you (the vendor) that had to handle bad publicity and the resulting loss of income (sanctions, paybacks etc.). There are some risks that cannot be transferred (opportunity costs etc.), but generally my losses would be small-ish.

(There is a bank in Finland which has an IT service vendor handle its computing needs. All the standard high quality goodies: hot standby fail-over data center with redundant connectivity between the two. The link was so redundant and reliable that when one of the redundant links actually did fail, it reliably caused the other link to fail at the same time. This kind of mind-blowing cluster-fuckup cost the service vendor, but cost the bank probably quite a lot too. Small-ISH is relative.)

With the introduction of cloud computing and its commodity computing model the the coverage handling of risks has changed:

my risk coverage   = (me ⇆ ---) × (methods ⇆ -----)
your risk coverage = (-- ⇆ you) × (------- ⇆ means)

Now a cloud computing provider’s job is to provide the technical services I have purchased at an agreed SLA. However the cloud vendor does not take the responsibility to ensure that I would use its services either correctly or effectively! In a cloud computing environment I must now handle processes that make effective use of the means provided by the cloud vendor.

Going back to the firewall example with Amazon Web Services:

AWS is liable if it fails to either a) provide the firewall services (security groups, VPC network ACLs) with agreed availability or b) they have other functional problems (like passing traffic not explciitly allowed).
AWS is not liable if I do “allow all from all” and someone hacks the system when I didn’t do the methods bit properly. I have to understand and implement the methods to use the means AWS provides to meet my own business goals.

Finally, a point

Out of this comes the synthesis of the great idea I referred earlier:

The introduction of cloud computing doesn’t substantially change IT service risks, but it does change the distribution of these risks between the client and the service provider.

What’s so bloody difficult in this for many enterprise and governmental clients is that for years they have oursourced all IT risk management processes and now they would have to learn to handle it themselves (or find someone else to do that — a market that didn’t exist when cloud computing came around).

Alternatively said:

Earlier, the negotiation of distribution of risk between clients and service providers was a business negotiation, an exchange of responsibilities and liabilities versus fees required to accept those responsibilities and liabilities.

Cloud computing in contrast is a commodity market where the service provider tries to minimize negotiations with the clients by providing a limited set of contract options for its clients.

So WTBF about startups?

Well think about it.

10 years ago

You’re a startup. You need IT service. You go to a IT provider. You are so bloody small, they give you a crap deal. You can’t negotiate — it’s either their way, or the highway. You call some other vendors, but in the end you’re really negotiating just different shades of blue. So you sign.

Then they blow up. You go out of business. (The contract? Yeah, good luck in trying. Even if you win, their standard contract you had to accept doesn’t give you back the business you lost. Remember, you’re a startup, you don’t have the capital to survive someone else’s fuck-up. Your house was next to my tinderbox.)

When cloud computing comes around

You’re a startup. You need IT service. You go to a cloud provider. They give you just one deal, the same deal everybody gets. You can’t negotiate — it’s either the cloud way, or get a TARDIS and go back 10 years (previous chapter).

(Then they blow up. Same situation as 10 years back, minus the lawyer.)

So …

Startups have never ever had the chance to negotiate risks on the same level as enterprises. Earlier they had to take the crap deal. (Alternatively they had to live in the shadows of the “real” IT economy, that is, hugging servers and trying to negotiate a reasonable deal with ISPs to get fat enough Internet pipe and worry the hair gray about their cheap hardware and colo deals.)

When cloud computing came around it offered no worse risk distribution than startups ever had to handle, yet it offered new capabilities that the earlier model lacked.

No wonder startups embraced the cloud. Even with an unknown future, the cloud was guaranteed to be no worse than what was available before.

Afterword

This is just one viewpoint. Making an assumption that this would be the only reason for success and fast adoption of cloud computing in startups is both wrong and retrofitting the facts to a fabricated historical narrative. Don’t fall for that. Reality is much, much more complex.

Enter Jekyll (where's Hyde?)

2013-07-08T00:00:00+00:00

Live and learn. Earlier I wondered whether I could use a github repository as a place to publish a blog. Essentially my original plan was to use the plain repository view as a way to render ReST formatted pages. But it got a bit unwieldy very fast after that.

A blog needs a feed. A blog without a feed isn’t a blog, at least by my definition. So I thought about writing a short script to read the page titles from RST files and output an .atom and .rss formats (and wrote one).
You still need a “master” page for random visitors so they can see what you’ve written lately. Ok, not a biggie either.
And … !!!!

No! I’m not going that way, again. I’ve written static website and blog generators before and I know where this path would lead me to. There has to be a better way! Surely my idea of using github as a blogging platform, surely there must be programmers who also DRY.

Of course I had seen and heard about GitHub Pages but had somehow completely bypassed them earlier. Funny how you can ignore so completely something you sort of know. I guess if someone had come to me a month back and asked “You know about github pages?” I’d certainly answered “Yep, you can host static web sites and blogs there.” Somehow I just didn’t make the connection between what I was looking for and what’s available. C’est la vie.

Alas, github pages isn’t a complete solution to your blogging needs. It does come with Jekyll static site generator which will help a lot in creating a website by either automating a lot of the legwork or by providing ready-made abstractions for wrapping custom stuff in Liquid templates.

I took a look at some of the example Jekyll-generated sites. Some are very pretty, and I’m impressed by the fact that sites like Development Seed can be generated via Jekyll. (Or maybe I shouldn’t. HTML5 makes that very possible. Perhaps it is more impressive that the site has been made despite problems that Jekyll has probably created… there’s no perfect tool, and I would assume Development Seed’s creators have hit a few gotchas along the way.)

“But still”, I was thinking, “do I have to write all those templates just to get a working static blog generator?”

So, if you’re thinking about creating your blog on GitHub pages using Jekyll, here’s what I found out: Jekyll Bootstrap. Quickly, do this!

$ sudo gem install jekyll
$ git clone https://github.com/plusjade/jekyll-bootstrap.git USERNAME.github.io
$ cd USERNAME.github.io
$ rake post title="Hello World"
$ jekyll server -w

and then browse to http://localhost:4000. You’ll see an example post and the one you just created before (Hello World). You’ll find the sample post in _posts directory. Edit it. Reload the page in browser. You can already see results!

The next step is to push your cloned repository to your own account under github — you’ll need to 1) create the repository, 2) update repository url at your checked-out Jekyll bootstrap repository and 3) push.

$ git remote set-url origin git@github.com:USERNAME/USERNAME.github.io.git
$ git push origin master

Note that USERNAME really should be your own github username when you push. Earlier when cloning it was just a directory name, but in set-url it must match your github username. You won’t see your pages in github pages unless you push to <your username>.github.io repository.

By the way — Jekyll bootstrap uses USERNAME.github.com in its examples, yet GitHub Pages keeps talking about USERNAME.github.io (com vs. io). Apparently there was a renaming operation moving user and project pages from github.com to github.io in April 2013. I tested that both schemes (e.g. USERNAME.github.io and USERNAME.github.com repository names) work, but accesses to the USERNAME.github.com URL will redirect you to github.io address. Note that Jekyll bootstrap instructions are likely to be updated at some point in time, so this note about the inconsistency might be obsolete by the time you read this.

After you’re done pushing, wait a while and navigate to http://USERNAME.github.io/.

P.S. You can take a look at the repository for this blog.

P.P.S. If you want to keep your blog’s version history clean from Jekyll Bootstrap’s commit history, do cd USERNAME.github.io; rm -rf .git; git init; git add ., commit changes with git commit -m 'Historyless clone from Jekyll Bootstrap.' update origin with git remote add origin git@github.com:USERNAME/USERNAME.github.io.git and do git push -u origin master. Just don’t do this if you ever wish to merge updates from the original jekyll-bootstrap repository.

Can you blog in github?

2013-04-12T00:00:00+00:00

I’ve written quite a lot in my life. Some of it has ended up in blogs (now since gone, although I have archive copies), alas most not. Mostly what I write is targeted towards a small audience — customers, co-workers, friends. For those venues a blog isn’t actually the best way (customers like .doc files, for co-workers email or intranet work just fine, and Facebook and Twitter are great for friends). Yet, sometimes I have a mysterious urge to spill my guts to the wider world.

I added a blog to wordpress.com some time earlier, then pulled its plug without actually writing anything meaningful in there. Somehow I felt a whole blog site, where I’d worry over presentation (I’m a stickler for good design) over the content was somehow.. unsatisfactory. Yet I’d definitely not want to host anything myself, either.

While this was swirling around my head (in a background process), I thought about github. It is public. It can natively render restructuredtext files natively to HTML — perhaps not so pretty, but what the hell. Thanks got git itself, all of the data would be version-controlled and securely replicated. Would that work?

So that’s what I’m now trying to check out. Could I use github itself to host a blog, with minimal maintenance and effort?

I do see already some problems. How would you do a nice RSS feed? How would I do linking between posts easy? And of course, how would I handle comments?

I’m not sure. I’ll find out soon enough.