Cave was released back in 1985 by XOB Software. Other things that happened in 1985: The UK got its first cellular phone network, DNS was launched and Richard Stallman published the GNU Manifesto.

I ended up going to the same school, and thus got to spend several years with those cutting-edge BBC Micros. As time moved on, they were replaced with Archimedes machines, and then I went on to other adventures. I never had a BBC of my own, instead taking the Commodore 64 route! (I was annoyed at the time but now concede that my parents very much made the correct choice!)

But… I always did miss Cave. It was beautiful in its simplicity. And so, in recent times, I decided to recreate it.

The first thing I needed was the original source code. I knew that Cave was mostly written in BASIC but I didn’t have my own copy (although I did once print out the entire thing on a dot-matrix printer!)

Thankfully, some perusing of the Stardot forum] turned up an original disc image which meant I could get to work. A handy little tool called Beeb Image let me access the numerous files directly on my own system.

The original tokenised BASIC files were converted to be human readable using this handy script: [BBCBasicToText.py].The resulting repository can be found here: https://github.com/tgreaves/cave-bbc-econet](https://github.com/tgreaves/cave-bbc-econet.

There are a whole range of modern MUD engines that can be heavily customised. Which one to use for this project? In the end, I went with Ranvier:

Ranvier is a MUD game engine whose goal is to be a simple but powerful way to build whatever MUD you want with special care given to extensibility. The core code strives to be completely unopinionated toward any specific style of game while using the bundle system to build the game you want without having to dig through the engine’s code.

Perfect!

By going through the original BASIC code, I could then implement equivalent functionality within the Ranvier system.

The original Cave used one file per ‘room’. This didn’t scale well on BBC Econet networks that used floppy disk based filestores. Imagine 15+ players all moving around, and all that contention on one poor floppy disk! Things got a lot better at school when the system was upgraded to one with a hard disk.

These files needed converting to a format that worked with Ranvier. I’ll take any excuse to write a new Python script, so did so! convert_rooms_to_ranvier.py t was born.

Now it’s your turn to explore! The source code is on GitHub, and it includes instructions on how to play the game.

You can also use your favourite client to connect directly! `cave.extricate.org:4000` is the address.

An example of a client that you can use is [Mudlet](https://www.mudlet.org/).

It’s not a game that has aged well compared to modern multi-player experiences. But I am happy to have done my part in preserving part of the historical record. What made this game fun was playing it with those 15 or so other people all in the same room — it’s all about that experience!

I hope you find some enjoyment with it too!

In this example, I walk through setting up an IPsec site-to-site VPN where the two sides are as follows:

1. AWS – A private VPC, containing one EC2 server (to allow me to test everything is working!)
2. Home network – With an OPNsense firewall sitting in front of it.

Warning: There are AWS charges for running VPNs as described in this article. Please refer to https://aws.amazon.com/vpn/pricing/ for the specifics.

AWS: Spinning up the VPC

I am going to let AWS do some of the heavy lifting for us here, as there is a handy Wizard for this scenario.

The Wizard handles the following:

1. Creation of a private VPC (not addressable from the Internet)
2. IPsec VPN configuration (including Customer Gateway, Virtual Private Gateway and Site-to-Site VPN)

As the above hints, there are several different components involved in bringing up the IPsec VPN on the AWS side. The Getting Started documentation is very good at explaining how these slot together, should you wish to get this going on an existing VPC.

From the VPC section of the EC2 console, I hit the Launch VPC Wizard button, which takes me here:

I am selecting the VPC with a Private Subnet Only and Hardware VPN Access, as that is exactly what I want!

Now, I can configure the details of the VPC:

I go with the defaults, as I’m happy with the 10.0.0.0/16 range (My home network is on 192.168.1.0/24 so no clash). I use the very imaginative name of ‘Test’.

I then need to tell the Wizard where it can find my home network. Note that AWS does not initiate the VPN – I will handle that later when configuring OPNsense.

‘Customer Gateway IP’ is the public-facing IP address of my network (but replaced here for example purposes). I am using Static routing, so need to tell the Wizard what that looks like – a pretty standard 192.168.1.0/24 which will be the case for a lot of home networks.

Clicking Create VPC will build it all out. This will take a few minutes.

Spinning up a test instance

Once this is done, I spin up a test instance. I won’t go through all the steps here, as I assume that if you are at the point of setting up a VPN like this, you are already familiar with spinning up instances within AWS.

The key point is that I ensure the instance is created within my new private VPC:

Once the Instance is up, I can verify this:

Perfect.

I now need to download the details that I need in order to configure OPNsense on my home network.

I go to VPC -> Site-to-site VPN connections and select my VPN. There is a handy Download Configuration button.

I hit that, and select pfSense (the product that OPNsense was forked from – the details within are compatible).

With the file in hand, that completes the work required within AWS. Now to the home network!

OPNsense: Firewall

There are a few different things to do here. This section is based on the official OPNsense documentation.

Let IPsec traffic into the network

By default, all incoming WAN traffic is blocked. There are several rules that need setting up to allow:

• IPSec ESP
• IPSec ISAKMP
• IPSec NAT-T

This is done within Firewall -> Rules -> WAN.

Here’s a snippet where I am setting up the first of these:

Once all three are in place, the summary screen will look like this:

OK, great! Now I need to set up the IPsec connection itself.

OPNsense: VPN setup

I navigate to VPN -> IPsec -> Tunnel settings, and hit the (+) symbol to add Phase 1.

I now adjust all the settings to match those provided within the downloaded AWS configuration file.

There are quite a few parameters, so I take my time to get it right! (This includes unchecking some of the pre-checked boxes).

Once saved, I’m not done! I need to do Phase 2, which is done by hitting the (+) next to the Phase 1 entry:

Again, I consult the downloaded file from AWS to get the settings right.

Once done, it all looks like this:

Note that I have now ensured that Enable IPsec is clicked before Saving everything.

Now, the moment of truth! Over to VPN -> IPsec -> Status overview:

Nothing yet… as the VPN does not come up by default (the little arrow on the right is still amber). I click on it, then wait and hit the little ‘i’ symbol to expand all the information.

This is good news! The VPN is up.

Final Firewall Bits

There is one more thing to do, however. I need a firewall rule to allow traffic through to the LAN network too.

Over to Firewall -> Rules ->IPsec to add this rule in. Once done, it shows in the Summary as follows:

Job done!

At this point, it’s possible to reach the private EC2 instances from my home network, by addressing them using their 10.0.0.0/16 addresses. I test this by running a ssh connection from a machine on my home network.

Some closing points:

1. You get 2 tunnels when setting this up with AWS to ensure redundancy. This guide only sets one up on the home side of the network. As an exercise for the reader, you can set up the second! (The details are in the downloaded Connection details file, as with the first one).

2. The method described here uses a Shared secret approach. This could be strengthened by using Certificates. Here is some AWS documentation to get started with implementing that.

Note: Although I work for AWS, these are my own personal findings and thoughts. Please do not consider them as official benchmarks in any way!

AWS announced the latest M6g instances in December 2019. These feature Arm-powered Graviton2 processors, as well as fully encrypted DDR4 memory. Arm processors are everywhere in terms of mobile devices, and my favourite Raspberry Pi computers, but have not traditionally been featured within the cloud. This is changing.

Arm-powered processors are not new to AWS: The first iteration (a1) was announced in November 2018.

Why would you care? In short, the announcements claim that you will see similar (or better!) performance to comparable Intel hardware, for a lower cost. What’s the catch? Your workloads have to be compatible with an Arm architecture. Chances are, if you are running standard Linux workloads on a popular distribution (e.g. Amazon Linux or Ubuntu), these are going to be a real option for you.

With M6g now Generally Available, I decided to take a closer look for myself. How easy would it be to get going? What peformance would I find?

Getting up and running

For a head-to-head comparison, I decided to go with the following instances:

• m5.xlarge (Intel 4 VPUs, 16 gig RAM, $0.214 per hour On Demand)
• m6g.xlarge (Graviton2 4 VPUs, 16 gig RAM, $0.172 per hour On Demand)

The above prices were correct at the time of writing for the eu-west-1 (Ireland) region. At launch, the m6g instances are also available in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Frankfurt), and Asia Pacific (Tokyo).

This article assumes that you are already familiar with spinning up Linux instances within AWS, including ssh access once they are available. If you need it, here’s the official Getting Started Guide.

Secondly, these instances are not Free Tier eligible! You will be incurring costs by spinning them up. Please remember to tidy up after yourself afterwards!

You know you are in the right place when, after electing to launch a new instance from the EC2 console, you see this:

Yes, the choice of either x86 or Arm architectures!

For this exercise, I went with Ubuntu Server 18.04 LTS. As some of the benchmarks are storage heavy, I changed the root EBS volumes to be 30 gigabytes in size, but still the default gp2 SSD storage.

I was then able to ssh in to start the benchmarking. The process for each instance was identical. Ubuntu looks and operates just the same, whether you are using the x86 or Arm editions. In fact, I would check /proc/cpuinfo every so often as a sanity check to ensure I was logged into the right machine at times (!)

1. Linux kernel compilation

Ah, that old favourite! Let’s grab the latest Linux kernel and see how each machine does at compiling it!

Here are the commands to invoke:

sudo apt update

sudo apt-get install -y git build-essential kernel-package fakeroot 
libncurses5-dev libssl-dev ccache bison flex

wget https://git.kernel.org/torvalds/t/linux-5.7-rc5.tar.gz
tar xf linux-5.7-rc5.tar.gz
cd linux-5.7-rc5
make menuconfig
time make -j 4

Some notes on the above:

1. After running make menuconfig, simply Exit and save the default settings.
2. We use -j 4 for the make process to use all 4 available CPU cores for the building process.

The results?

In this test, the Arm system was 2.25% slower. That’s pretty close. Especially when you consider it is 21.7% cheaper in On Demand costs!

There’s a slight caveat with this test: The exact files compiled could vary between x86 and Arm architectures. It’s not exactly scientific. So let’s move on to some more…. traditional…. benchmarking.

2. MariaDB performance

For benchmarking exercise number 2, it was the turn of MariaDB. As a quick reminder:

MariaDB Server is one of the most popular open source relational databases. It’s made by the original developers of MySQL and guaranteed to stay open source. It is part of most cloud offerings and the default in most Linux distributions.

{{< figure src=”mariadb-logo.png” >}}

Here, I combine installing MariaDB as well as sysbench in order to load test it. A read / write test is performed. Here we go!

sudo apt-get install -y sysbench mariadb-server
sudo mysql -u root -e 'create database sbtest'

sudo sysbench /usr/share/sysbench/oltp_read_write.lua --db-driver=mysql --threads=4 --mysql-host=localhost --mysql-user=root --mysql-port=3306 --tables=5 --table-size=10000000 prepare

sudo sysbench /usr/share/sysbench/oltp_read_write.lua --db-driver=mysql --threads=16 --events=0 --time=300 --mysql-host=localhost --mysql-user=root --tables=5 --delete_inserts=10 --index_updates=10 --non_index_updates=10 --table-size=10000000 --db-ps-mode=disable --report-interval=1 run</code></pre>

Some caveats here:

1. This is the test that is very hungry on disk space. Most of the 30 gigabyte storage will be utilised.
2. You will notice I don’t secure the MySQL instance, with the commands running as root. This is not Best Practice for general systems. However, in this case the instances are disposable, and they are being destroyed after the testing process.

The results?

m5.xlarge:

SQL statistics:
    queries performed:
        read:                            436520
        write:                           1247200
        other:                           62360
        total:                           1746080
    transactions:                        31180  (103.91 per sec.)
    queries:                             1746080 (5819.10 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          300.0588s
    total number of events:              31180

Latency (ms):
         min:                                  6.92
         avg:                                153.96
         max:                                975.64
         95th percentile:                    314.45
         sum:                            4800464.88

Threads fairness:
    events (avg/stddev):           1948.7500/19.35
    execution time (avg/stddev):   300.0291/0.02</code></pre>

m6g.xlarge:

SQL statistics:
    queries performed:
        read:                            451836
        write:                           1290960
        other:                           64548
        total:                           1807344
    transactions:                        32274  (107.56 per sec.)
    queries:                             1807344 (6023.09 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          300.0680s
    total number of events:              32274

Latency (ms):
         min:                                  5.66
         avg:                                148.74
         max:                                833.04
         95th percentile:                    303.33
         sum:                            4800535.10

Threads fairness:
    events (avg/stddev):           2017.1250/22.81
    execution time (avg/stddev):   300.0334/0.02</code></pre>

If we pull out the key performance metrics:

| Instance | Metric | Result |
| m5.xlarge | Transactions | 103.91 / second |
| | Queries | 5819.10 / second |
| m6g.xlarge | Transactions | 107.56 / second |
| | Queries | 6023.09 / second |

Here, the Arm-powered m6g instance was 3.45% faster for both Transactions and Queries. Decent! And, again, still 21.7% cheaper!

3. Redis performance

Finally, it was the turn of Redis.

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams.

I was particularly interested in this one: It’s in-memory. Would there be a performance impact with the full encryption of RAM involved with the Graviton2 processors?

This one is nice and easy to do:

sudo apt-get install -y redis-server
redis-benchmark -q

The results:

m5.xlarge:

PING_INLINE: 100000.00 requests per second
PING_BULK: 97370.98 requests per second
SET: 101522.84 requests per second
GET: 100908.17 requests per second
INCR: 101729.40 requests per second
LPUSH: 92678.41 requests per second
RPUSH: 101112.23 requests per second
LPOP: 94517.96 requests per second
RPOP: 92421.44 requests per second
SADD: 101010.10 requests per second
HSET: 95419.85 requests per second
SPOP: 84530.86 requests per second
LPUSH (needed to benchmark LRANGE): 85910.65 requests per second
LRANGE_100 (first 100 elements): 54614.96 requests per second
LRANGE_300 (first 300 elements): 23929.17 requests per second
LRANGE_500 (first 450 elements): 16672.22 requests per second
LRANGE_600 (first 600 elements): 11687.71 requests per second
MSET (10 keys): 99206.34 requests per second

m6g.xlarge:

PING_INLINE: 135135.14 requests per second
PING_BULK: 127388.53 requests per second
SET: 135318.00 requests per second
GET: 132100.39 requests per second
INCR: 136798.91 requests per second
LPUSH: 142653.36 requests per second
RPUSH: 136239.78 requests per second
LPOP: 141442.72 requests per second
RPOP: 136612.02 requests per second
SADD: 134228.19 requests per second
HSET: 142247.52 requests per second
SPOP: 132802.12 requests per second
LPUSH (needed to benchmark LRANGE): 141043.72 requests per second
LRANGE_100 (first 100 elements): 64850.84 requests per second
LRANGE_300 (first 300 elements): 21734.41 requests per second
LRANGE_500 (first 450 elements): 14100.39 requests per second
LRANGE_600 (first 600 elements): 10858.94 requests per second
MSET (10 keys): 107991.36 requests per second

There are a lot of stats here, so just pulling out a few:

| Instance | Metric | Result |
| m5.xlarge | SET | 101,522 / second |
| | GET | 100,908 / second |
| m6g.xlarge | SET | 135,318 / second |
| | GET | 132,100 / second |

A big difference here! The Arm-equipped ms6.xlarge is 26.7 – 28.5% faster on these metrics. In fact, it’s faster on all of them, except the latter two LRANGE tests. That could be worth looking deeper into.

Conclusion

From these tests, my conclusion is that the m6g.xlarge is punching very well in terms of performance, in comparison to the m5.xlarge. It was ever-so-slightly slower on the Linux kernel compilation, ever-so-slightly faster on the MariaDB tests, and notably faster on the Redis tests.

Then we factor in the price. The m6g.xlarge It’s 21.7% cheaper in terms of On Demand pricing!

I would definitely be looking to move any appropriate Linux workloads to the Graviton2 setup. The price / performance ratio is stellar here. I can see these processors becoming increasingly popular as ‘word gets out’ on just how good the ratio is!

I hope you have found this journey into some very simple benchmarks useful. I’d be very interested to hear on any workloads you are considering moving to Graviton – perhaps you’ve already moved them!

This got me thinking about implementing a decent home network firewall solution, above and beyond the default one you get when running your average broadband router.

Security aside, I’m also a big fan of stats and pretty graphs, and these are quite simply hard to come by with some of the basic consumer broadband router solutions.

While the Raspberry Pi, especially in its 4th incarnation, has a decent amount of power behind it, it’s not quite in the ballpark to be running pfSense. Mainly as it only has one Ethernet port out of the box, and while there are methods to extend this, I wanted a piece of hardware that had what I needed ‘out of the box’ but was still minimalistic in nature.

Enter the PC Engines APU boards. These are system boards designed for this sort of thing, complete with multiple Ethernet ports, decent amounts of RAM (2 gig upwards) and AMD G Series processors. They can boot from an SD card or added m-Sata SSD storage.

Getting on board

Instead of putting all the bits together myself, I purchased a pre-built system from LinITX. The spec was as follows:

• APU2 E2 board (2 gig RAM, 3 Ethernet ports)
• 16 gig M-Sata SSD
• Case
• Power supply (12V)
• pfSense pre-installed on the SSD.

Direct link on LinITX’s site.

My reasoning on going with this spec:

• 2 gig RAM is easily enough to run a consumer firewall for a typical home family network. (The system is running at around 430 Mb in use most of the time).
• SSD storage rather than SD card. This is for resilience. SD cards are just not that great when it comes to computer usage. I had two die within a week which left a bit of a sour taste in the mouth.

The kit arrived next day, which was wonderful service.

In terms of size, you are looking at something twice the width of a typical Raspberry Pi case. As the board pictures show, there are no cooling fans, so it is rigged for silent running.

One slight annoyance is that there is no power switch – as soon as the power cable is plugged in, the thing will start booting!

Getting Flashy

The system booted up into pfSense with no issues. The Dashboard showed that the APU board was running a very old coreboot (BIOS). In fact, 4.0.7 was so old it didn’t even appear as a Legacy version on the PC Engines coreboot web pages.

In general the rule is not to update your BIOS unless there is an actual reason to, due to the potential risk of bricking the device. For the APU boards, however, there were various fixes since 4.0.7 that seemed pretty relevant — especially if you wanted to perform a re-install of anything without messing around with kernel and boot parameters.

But how? I didn’t at this point have a null-modem cable to control things when booting from a USB stick to perform the flashing procedure.

I then realised that, duh, pfSense is running on FreeBSD which is more than capable of running flashrom directly.

Which version? 4.11.0.6 is the latest at the time of writing. Is it safe? Well, if in doubt, ask Twitter…

I followed ‘Method 1’ as documented here. No scary messages appeared in the process, and I was met with:

Verifying flash... VERIFIED.

One reboot later, and pfSense remained happy, and reported the new version of the BIOS was there.

Switching to OPNsense

pfSense was working well. PPPoE worked first time with my broadband setup (once I had an appropriate modem) along with all the other home networking bits that you would expect.

However, mainly for political reasons, I wanted to switch to OPNsense. Some of the actions taken by those involved with pfSense over the years had been, to be blunt, pretty unprofessional and the community element of OPNsense felt a lot better to me.

As mentioned earlier, I didn’t have a null modem cable, which is needed to access the APU boards over the serial cable: There is NO monitor output!

I sourced a StarTech USB null modem cable from Amazon: https://www.amazon.co.uk/gp/product/B008634VJY

The drivers were provided on CD but were thankfully also easily found via the StarTech web site. The cable worked first time, once hooked up to a laptop running Windows 10 and MobaXTerm. The serial port settings are provided by PC Engines.

This allowed the quick installation of OPNsense. Configurations between the two products are not directly compatible, but re-creating my pretty simple setup really didn’t take very long. As with pfSense, the basics such as PPPoE worked straight out of the box.

Performance

I did a quick speed test via https://www.speedtest.net/ – this actually provided my fastest download speed yet! That’s not scientific at all as there are so many variables with these sorts of things, but at least it shows no reduction, right?

The box is running happily at about 22% RAM utilisation, so plenty left for additional bells and whistles.

The CPU is up and down, due to Netflow logs being enabled, but peaks at 25% when data analysis is going on.

Verdict

I’m happy!

This is one of those ‘does what it says on the tin’ products. LinITX provide a good package deal (even if the BIOS was very old, but easily rectified), and delivered very quickly too.

The hardware has proven rock solid so far: Despite flashing, re-installing etc, there have not been any surprises at any point.

Based on my limited use so far, both pfSense and OPNsense run well on the hardware. You would be happy no matter what your choice of flavour. Note that you can install other things too… here’s an Ubuntu example.

If you are looking to have a home firewall box, this is definitely a good way of doing it. I’d be interested to hear if you are doing it in any other way though!

If your modem can’t do this, then essentially you have to do it via IP bridging instead, and end up with a horrible ‘double NAT’ situation. I really wanted to avoid that.

You would hope that the shiny (Sagecom manufactured) Hub One from plusnet would support my desired bridging mode…. but no. It doesn’t. Could it be achieved with alternative firmware? Yes, but… There’s no way of flashing the firmware on these without getting your soldering iron out, and I didn’t fancy that.

Let’s go back to 2011…

Thankfully, modems exist that do support bridging mode. Some research revealed that a particularly well-regarded one is the Huawei EchoLife HG612. It dates back to 2011 and was the originally provided modem for the first fibre broadband installs in the UK.

When initially provided, they were very much locked down with no access to configure it. Custom firmware has since arisen which completely unlocks the GUI. Great news.

I picked one up from eBay: Go for one of the ‘3B’ models if you can as these solve a previous overheating (!) issue. They seem plentiful on there. Surprisingly, the one that arrived seemed brand new, so was either unused or had been very well looked after!

It arrived unlocked. This meant some work was needed.

Flash Attack!

Thankfully, the flashing procedure is straightforward. Power on the router while holding down the RESET switch, then connect up to a laptop via Ethernet. Upload new firmware and you are done.

The whole process is explained here.

This was successful. I obviously held my breath for the correct amount of time…

One step at a time…

My first test was to use this modem as a ‘drop-in’ replacement for my original Hub One router. This meant a few changes via the GUI:

1. Change the LAN IP address: It defaults to 192.168.1.1, whereas the Hub One uses 192.168.1.254. I did not want to adjust the Gateway on my home network until I was sure the HG612 actually worked properly! (Note: DHCP is handled elsewhere in this scenario).
2. Switch to Router mode: The HG612 comes with bridging selected by default. So it was a simple case of switching to ROUTER mode, and entering my Plusnet credentials. No other changes were needed.

With this done, the HG612 successfully negotiated everything and my network was working.

The Final Bridge

After an appropriate amount of soak testing (the line remained rock solid), it was time to go ‘full-on bridge’.

Thankfully, this was again straightforward:

1. Switch the HG612 to PPPoE Bridged mode: This was as simple as changing the mode to ‘PPPoE Bridge’ in the GUI. I set Port binding to LAN1 only – this meant that LAN2 would remain free as a way to access the GUI should I need it.
2. Configure firewall with my Plusnet credentials. Straightforward and no fiddling with any other parameters (e.g. MTU) required.

It worked first time and, touch wood, I have not observed any drop-outs to date.

Final thoughts…

It’s frustrating that there seems to be a trend of ISP-provided routers locking out this sort of functionality: Plusnet are not alone in this. Getting better functionality from a device from 9 years ago is not how these things should be.

Technology should always be an enabler to make things easier. Not making PPPoE available (even in Advanced Settings) just made things that bit harder.

Now get off my lawn.

What Enterprise background is that exactly?

I work for StepStone. AWS is our cloud provider of choice. We have approaching 100 accounts and an expenditure level within AWS that you would expect given our €3,000+ million revenues (2018).

StepStone have many different brands, and multiple architectures powering those brands within the technological estate. This is historically due to acquisitions. There is always ongoing consolidation work, such as shared services for different brands to use, but this adds to the complexity that we are dealing with.

One of the linchpins of my role is Cost Optimisation, and working with the teams across the Group to do this right. Hey, this is one of the pillars of the AWS Well-Architected Framework after all.

Life Before Savings Plans

As is typical within Enterprises in AWS, a large portion of our spend is in EC2. That has mostly originated from numerous ‘lift and shift’ migrations.

We introduced targets for reservations across the group. This is really simple:

• The minimum target is 60% of savings being realised. I.e. 100% would be “All of my EC2 capacity is reserved”. Note: As this is talking about Savings that means Production servers (typically more powerful and therefore expensive) have greater weight than Development servers.
• “What about Spot?” — That’s OK, as they aren’t included in the calculations, so go for it.
• “What if I can’t reserve because….” – Yep, there are always valid reasons for this. One would be building something new and the capacities aren’t known yet. Another would be a site which is known to be sun-setting within a few months, and the old reservations have just expired.

This approach has worked really well for us. We’ve been regularly exceeding this target. In fact, hitting 80%+ is typical.

This relies on the individual Account Owners, and their teams, really focusing on which reservations should be made, and using the tools available to do so. CloudHealth has historically helped a lot with this, and Cost Explorer has got better at it too.

But it’s not perfect….

When Reservations Attack

The whole reservation system can cause pain.

Firstly, there is naturally a degree of manual work in implementation. The various recommendation engines make this a lot easier now, but there still has to be a manual analysis on what the future brings. What’s the future of the architecture? Is it being migrated very soon? Exactly what type of reservation should you be going for?

Okay, some amount of work like that is a given, to be fair, as reserving capacity is all about pre-planning.

The main issue is when things go wrong and you are having to work with Amazon to get things swapped around.

This is typically when we have needed Reservations refunded due to situations changing. Now, I’m not talking about where we have changed our mind — I wouldn’t expect a refund in that instance. We’ve had situations where we have made purchasing decisions based on advice from Amazon that was later shown to be wrong. The good news is that we got refunds, but it was a long and torturous process, involving far too many approval layers on the ‘other side’. So much so, in fact, that I’m wondering whether the money we got back even paid for the salaries of everyone that was involved. Frustrating.

We’ve historically been told that this is due to the EC2 capacity team relying on Reservations to accurately plan capacity. I’m now a little bit dubious on that, purely as Savings plans do not have to specify instance types any more – purely an expenditure level.

Are Reservations going to go?

Amazon are making it clear that they don’t really want customers using Reservations any more (from the FAQ):

Can I continue to purchase EC2 RIs?

Yes. You can continue purchasing RIs to maintain compatibility with your existing cost management processes, and your RIs will work along-side Savings Plans to reduce your overall bill. However as your RIs expire we encourage you to sign up for Savings Plans as they offer the same savings as RIs, but with additional flexibility.

Remember: Not only are Savings plans more straightforward for the customer, they are going to cause less drama for Amazon themselves. No longer will they have to field reservation refund requests.

I can imagine Amazon will make Reservations no longer available to purchase within a few years, and push Savings Plans very strongly before that via Account Managers and TAMs as they detect renewal windows coming up.

What are we going to do?

This is a great question. As of right now, we haven’t purchased any Savings Plans.

The reason for this is that, at an Account level, those teams can be very precise in their reservation strategy and therefore unlock the best discounts.

Executing a Savings Plan purchase at our payer account level (where we really have a great view of total spend) means being a bit more generic, and going in at the top level, rather than Instance types. There’s a balancing act here, as we would probably get better coverage but the discount level might end up being lower. Oh, and throw into the mix the fact that the human cost of analysing everything would be reduced too.

Also, if we purchase at the payer account level, we need to look at how we cross-charge our various entities their share of the upfront payment. Yes, we cross-charge our AWS bill. We’re not alone in our segment for doing this, that’s for sure.

In short, we need to do some more analysis and then take a consistent way forward, to avoid any confusion given the number of people that we have working on cost optimisation within the business.

What are YOU going to do?

Please let me know! I’d welcome your thoughts on Savings Plans and the changes that you have made, or are going to make, as a result of their introduction!