The post Sysadmin Sunday 131 appeared first on Server Density Blog.

Server Density at conferences

Our key conference belief is not to sell anything – just nurture an interest. So we never pay for speaking slots and with the limited financial resources available to us, we opt to sponsor more with our time than with money. For that reason we keep an eye on lanyrd, organise talk proposals and send our engineers around the world.

Following this theme, we try to offer custom and useful marketing swag out to attendees upon registration; not lazy corporate marketing gumf. Instead of resorting to paying thousands of dollars to plaster our logo everywhere and gain exposure through forced tweets, we try and provide attendees with real value – whilst keeping the costs as low as possible.

Providing valuable swag

Bearing in mind the overwhelming marketing presence at conferences, the aim for us is to offer something that attendees actually like, take away and use. In the past that has been, tape measures, USB lego men and more recently our custom (dot) books.

The (dot) book

The dot book is a 4 colour dot grid notebook. We’ve designed and printed 2000 so far and offered them out at various conferences over recent months, customised to the theme of the conference.

1. .bson notebook
2. .devops notebook
3. .pcap notebook
4. .nib notebook

The name

The dot name reinforces the point that we’ve designed and printed the books specifically for people with a technical knowledge – these names represent the filetype, language or framework that the books relate to:

.bson is an interchange format used mainly as a data storage and network transfer format in MongoDB.

.devops was not based on a file extension but just took the name of the idea behind combing operations and development.

.pcap (packet capture) is based on the results of a packet capture session from something like Wireshark.

.nib is the file extension for NeXT interface builder which is primarily used for iOS development.

The paper

Striking the balance between thick enough to convey a message of quality, yet thin enough to close the book was difficult. After many iterations we found that balance with the following paper spec:

• 300gsm uncoated 100% recycled paper cover
• 90gsm uncoated 100% recycled

It was important for us to use 100% fully recycled paper, although we had to be careful that with uncoated finishing, the paper was able to hold the colour that was being applied, in this instance 4 colour digital.

The pages

We’ve been able to customise the inner notebook pages as much as we are the outer spreads. Following the (dot) file extension theme, the writing paper is formatted in a dot grid. It has more value than simply reinforcing the name – dot grid paper has all the benefits of squared / lined so you can write straight and do technical drawings, but the visual guides are more subtle – you should definitely try it if you haven’t!

In the example of our .nib books made for the ALTWWDC conference, we designed iOS wireframes for each right spread – as well as a dot grid of course!

The cover pages

We’ve made the covers minimal, but with colours that subtly hint at the context:

• The .bson is green because of MongoDB.
• The .nib books are the bluey grey used by Apple on its developer pages.
• The .pcap books are red because they were initially made for one of our engineers talks at Atmosphere conference.
• The .devops books are blue to follow the devopsdays.org colours.

The back of the notebooks showcases our only real attempt at ‘official / traditional marketing’ with the company logo taking pride of place.

The inside covers is where the books get interesting and provide some real value, they are the pages we customise heavily for each book. They provide quick quips, tips and hints that people using the language will find useful.

Inside the .bson

Inside the .devops

Inside the .pcap

Inside the .nib

The problems with production

We’re a lucky lot working with the internet – we can build and test things instantaneously, deploy things with one click and change / fix bugs post production. Unfortunately when sending an email with packaged artwork to a printers, you better be 100% confident that you haven’t made a tpyo typo.

This was a major concern, not because we’ve had to send 1000′s back, but because we needed to get things right the first time. I would encourage any one undergoing a journey like this to leave your self time for at least 2 iterations. Luckily we had enough time (and the printers patience) to go through 3 iteration stages. Here’s why:

1. The cover pages were too thin and they lacked a quality feel.
2. The cover pages were too thick so the books wouldn’t close properly. In addition, we chose an 80gsm inner page stock which was too thin and didn’t withstand the fountain pen “seep test”.
3. Because the cover page had to wrap all of the inner pages, the design had to increase by 6mm and the inner pages reduce by 6mm towards the middle so all pages sit flush together when the book is closed. This also caused problems with content being off centre, so alignment was altered on a few occasions.

The future of the (dot) book

So far we’ve received some great feedback about the (dot) books that we’ve had produced. We want to keep them coming and make them a Server Density staple. To close this off, is there a single notebook in particular that you might like to see / find useful in the near future?

The post Designing and printing (dot) notebooks appeared first on Server Density Blog.

The post Sysadmin Sunday 130 appeared first on Server Density Blog.

Network considerations for multi data center redundancy

The network is the connecting layer between your customers and you, and all your internal components across data centers. This means there are a few areas to consider how failover works:

Connecting you to your customers – IPs and DNS

If one of your data centers fails then you need a way to redirect traffic to the secondary data center. We achieve this for Server Density by using the global IP service offered by Softlayer. This is similar to Amazon’s Elastic IP service but allows you to point the IP to any server in any of their data centers, rather than being restricted by region. This allows us to redirect traffic to an entirely different data center without any customer impact – the IP stays the same, it’s the internal routing that changes.

However, there is a failure scenario where this doesn’t work. The change requires the original data center routers to acknowledge the new configuration before the new route is applied, which might fail if there is a network problem in the original data center. This is mitigated by using DNS failover.

Our DNS is set with a low TTL so that we can adjust the IPs we send traffic to. In the event we can’t reroute our global IP, we can update the DNS to point to an IP in another data center, which we have already reserved and are ready to run as hot standbys. The downside is this can cause some downtime because of the DNS caching for the length of the TTL. Some ISPs cache more aggressively so it’s not guaranteed to update for all customers at the same time.

An alternative would be to use round robin DNS to provide all the IPs at once but this relies on the client to time out on failed connections and means some connections would always go to the secondary data center, which may not be optimal if you run a primary/secondary data center setup.

Internal connectivity

This is mostly about latency. Within the data center you can usually expect sub-1ms round trip times but as soon as you start to go between data centers then this increases based on distance.

From Washington, DC to San Jose, CA in the USA we see round trip times of around 72ms. Across the Atlantic you can expect anything up to 100ms, trans-Pacific up to 150ms and between Europe and Japan up to 300ms.

The implications of this are relevant for database replication – can you survive eventual consistency or do you need to guarantee the data gets to all your data centers? Network partitions can also be a concern.

It’s also relevant for file transfers, particularly for things like backup. If you have to restore an offsite backup to your secondary data center, how long will the file transfer actually take?

Server & ops considerations for multi data center redundancy

Load balancer failover

We run load balancer pairs using nginx – one active, one hot standby. These are monitored externally and if there is a failure, our global IP is automatically rerouted to the standby load balancer and an on-call alert triggered.

Gateway access

Almost none of our servers can be connected to via SSH from the public internet – all access goes through a single gateway server, including across multiple data centers. Of course, if the data center where this is located goes down then you lose all access. So we have a duplicate in our secondary data center.

An alternative to this is to use a VPN to connect directly into the environment network. I don’t like this approach because it opens up the entire network to your local system, which can lead to mistakes e.g. connecting to production by accident.

Self hosted tools e.g config management

We run Puppet to manage configuration across all our servers and amongst other things, it manages our internal hostnames through a centralised /etc/hosts file. When servers need to be replaced or IPs changed, we can do this using Puppet but it has no built in redundancy so we have to set up a replacement Puppet slave ready as a hot standby.

This applies to other tools you might be using. Do you have a central logging server? A backup management system? Maybe you even run your own monitoring! The advantage of using SaaS products is you don’t have to think about redundancy and monitoring for these tools, but some are best run yourself. In those cases, you need to consider the failover of each tool and how your response might be impacted if that tool was unavailable.

Monitoring

You need to know when you’re completely down, but also when certain failure scenarios take place: usually low traffic, high latency, increased errors. This requires a combination of monitoring across your entire infrastructure: from remote tests for response time through to end to end testing of request pipelines. Ideally, you will detect problems before customers start noticing.

Communication

The worst thing as a customer is to see problems with a service but have no idea what is happening. This is where public status pages come in. But it’s not enough just to post when there are problems – if your service is critical to your customers, they need to be able to subscribe to be warned when there are issues.

Server Density monitoring is critical to our customers so if we have a service issue where our monitoring stops, they need to know immediately so they can manually monitor their systems. The same with services like PagerDuty – if their alerting system is down, you have no way to know when your own services are down!

You can build your own fancy status pages like Heroku and AWS do. But there are services like StatusPage.io which provide a hosted page for you.

It’s also worth considering where else you should be telling your customers – many will be looking to social media if there are problems.

Automatic failover

We have automated failover of our nginx load balancer pairs but a full data center failover requires a manual process. The problem with automated failover is the potential for flapping, which can make a situation even more confusing.

Going to a full multi data center deployment all serving live traffic is probably the best way to do this because the requirements of that setup naturally mean each data center can serve traffic. Then it’s just a case of removing a data center from the rotation if it goes down, rather than having to deal with switching and failover.

Whether you do manual or automated failover, checklists are valuable. They help avoid mistakes with manual processes and allow you to run through to make sure everything is confirmed working when an automated failover happens. We have checklists for our manual data center failover process, load balancer failovers and also a recovery checklist to ensure all core functionality is working after an outage.

Conclusions

Full multi data center redundancy is a long process and always increases raw hosting costs. The stage of the business determines when this should be done. As you grow in revenue and customers, it becomes more appropriate, especially if people are relying on the service.

It may not be worth it depending on how critical your service is to your customers if you assume your downtime will be relatively short (hours). You have to weigh up whether the likelihood of a prolonged outage from extreme events like Hurricane Sandy justifies the effort vs days of downtime.

You’ll also probably find other things not mentioned in this post, so feel free to comment with anything else you find in your own quest for full multi data center redundancy!

The post Multi data center redundancy – sysadmin considerations appeared first on Server Density Blog.

The post Sysadmin Sunday 129 appeared first on Server Density Blog.

There are x3 main types of multi data center deployments:

1. Disaster recovery: you maintain servers in an additional data center which allows you to recover data in the event your primary data center is destroyed. It’s designed as a last resort after a catastrophic event and doesn’t usually allow you to run normal operations from the secondary data center. This is more like a backup.
2. Hot standby: you maintain duplicate servers in an additional data center which are running and ready to take over immediately in the event the primary data center fails. The entire application can fail over very quickly and means you can survive the destruction of the primary data center, or failover in the event of planned maintenance; basically any event where the primary data center is unavailable.
3. Live traffic handling: the same as the hot standby option but this data center (or data centers) serve traffic too, so there is often no real “primary”. This is often used to locate the application closer to the user and tends to use some kind of geographical load balancing, such as using anycast DNS to route users to their closest location.

These are ordered in terms of complexity and cost and are generally implemented in that order e.g. to have a live traffic handling data center you need all the same things as a hot standby facility. Each one gets more expensive because you have to duplicate servers with sufficient resources to take over live traffic, even if they’re not used (in the case of hot standby). Server Density has had a disaster recovery setup for several years and we recently upgraded to hot standby ability, with a view to moving up to live traffic handling in the future.

Application considerations for multi data center redundancy

There are two main aspects of implementing multi data center redundancy. The first is the sysadmin, network and server engineering work that needs to be done to deploy multiple servers and set up the failover mechanism, which will be covered in a future post. But before that, some preliminary work needs to be done to get your application ready to handle switching data centers.

Database failover

Databases are perhaps the most complicated component to scale and database failover is very product specific. So there are a number of generic things to consider in relation to which database you’re actually using:

• Replication: this is almost certainly how you’re going to handle failover on the database. Consider how replication is implemented with regards to master/slave and how failover is triggered.
• How far behind are your slaves? Across regions there will be some replication lag due to network latency. Can your application handle some data being “lost” because it hasn’t been replicated yet, or do you need to ensure strong consistency?
• How does your database handle split brain conditions when the network partitions? Do you need an independent node in a third data center to arbitrate over which node becomes master?
• How does your application detect a change in database master? Does this even matter? Will your users get errors or will it happen automatically?

Functionality

In hot standby setups, you can make tradeoffs if you don’t want to/can’t afford to replicate every single component. This means your application will need to degrade gracefully depending on the failure scenario; something which works well with service orientated architectures. For example, you could temporarily disable profile image uploading rather than duplicating large numbers of photos across data centers.

This requires your application to know when there is a failover situation, which can be done using a manual config flag that’s set as part of your failover process. Alternatively, you could set environment variables so the application knows which data center it is being served from and handles the situation appropriately.

Assumptions

Your application might make some assumptions about the availability of local resources. Paths, hostnames or IPs might be hard coded. You’ll need to audit your code to find out what assumptions have been made and good testing will reveal anything you have missed.

Notifying users

It can be useful to display a banner to users when there is a failover condition, especially if certain features are disabled or performance drops because of increased latency or cold caches.

Local data e.g. sessions

Sessions are often implemented using some kind of local storage. This problem tends to be solved as part of needing to balance traffic across multiple servers but it’s also worth considering how this works across data centers. If you use a database for storing session data then it will be replicated already, but be careful if you are using file or load balancer based session handling.

Don’t forget websites

Your core application needs to be able to fail over to allow existing users to continue using it, but don’t forget your product websites and billing systems. It’s good practice to consider your website a separate product with its own redundancy and deployment mechanism because for many businesses, its the sole location for new customers to find out and signup to the service.

First step completed

With all these considered and “fixed”, the next step is to start duplicating servers in a secondary location…which will be the topic of the next blog post!

The post Multi data center redundancy – application considerations appeared first on Server Density Blog.

The post Sysadmin Sunday 128 appeared first on Server Density Blog.

The post Sysadmin Sunday 127 appeared first on Server Density Blog.

The post Sysadmin Sunday 126 appeared first on Server Density Blog.

Initially, these were installed locally on a developer’s machine and set up via Apache vhosts running each component separately.

This soon became unmaintainable on a daily basis without a lot of work. We were spending too much time discussing whether a certain component was up to date and fighting bugs caused by API incompatibilities between versions. As we added more services to deal with things beyond the core of the product, this just got worse.

The answer was Vagrant.

What is Vagrant?

Essentially, vagrant is a command line tool for managing virtualbox instances (other backends are now available in the latest version). You use a pre built box (or package your own) to create a fresh virtual machine with all your tools installed, accessible via SSH from the host machine.

Once a vagrant box is configured, it’s just a case of vagrant up, waiting a while and then a system is up and running.

Our vagrant box

We went through some iterations and experimentation to find something suitable for the way we wanted to work, combined with what we could actually achieve in the box. This is the end result for now, and I’ll list some modifications that we have planned but haven’t found the time to do.

The box is split in two, a base and an environment box:

The Base Box

The first stage in the vagrant build was to build a customised base box. We based it on Ubuntu 12.04, 64-bit build as that’s what we planned to use in production. Once that was chosen, I collected a set of dependencies and development tools that were required for each service (MongoDB, Apache, Node.js, vim, screen, etc). These were then deployed into the base box using a fairly standard puppet manifest with some available modules. At this stage, it installs just the tooling, dependencies and makes required system level config changes (networking setup, DNS entries).

Once this box is built, it’s uploaded to a development webserver so all the team have access to it. The Vagrantfile and the puppet manifests live in git, alongside the development box. This means that the base box can be recreated/tweaked/reviewed by anyone at any time if that’s a necessity.

Using a base box like this loses some flexibility. Every time you want to add something new at the base level, you have to rebuild and re upload the entire box. But it saves deployment time in the next stage, which overall results in a win.

You can temporarily work around this by adding a dependency into the environment box, but if you’re not strict with how you manage this it becomes a bit of a mess of where everything is, and you lose the advantage of having a pre-built base box.

The development environment

This is a little less straightforward than the base box build.

Starting with a Vagrantfile with the base box imported/declared, we added puppet modules to handle installing our agent into the box so it can report to itself, and a module to install apache vhosts.

Once that was done, we created a puppet module that can handle installing all of the services from git. This includes a clone/update, build process (buildout/composer), and finally run any test or development data scripts that we have. This was mostly a mash of already available bits and abuse of the exec puppet command.

The Vagrantfile is just a file of Ruby code, so it was easy to add something that set the box hostname based on the username of the host, and import a separate settings file for overriding the defaults for the vagrant settings (code checkout locations being the main one).

The final stage for this was to add a script that will kill and then start all the code that runs as a service (tornado/celery mainly). This grew out of an ugly hack involving starting lots of things in screen, and hasn’t really been updated to anything else. It does have a convenient advantage that `screen -list` will tell you exactly what is running, and a total number at the bottom for quick verification that everything started okay.

The code is checked out into a shared directory between the host and the guest, ultimately living on the host. This uses nfs for performance which means we can edit the code using the host editors and tools but the code will still run inside the vagrant box.

Debugging is taken care of by xdebug being configured to point to the host IP for the PHP services, and some work with WingIDE remote debugging for the python services, again with Wing configured to connect into the vagrant box.

Once the box was up and running, we added settings, configs and a custom domain (using vagrant-dns) to enable decent separation and ensure we don’t accidentally hardcode production/development URLs (or at least, that these are easier to catch if it does happen).

The main feature of this box is that the puppet manifests run with each provision. These update and redeploy the code each time, simplifying the update process to a single command, across every service and repository that we have deployed.

The Advantages

• Reproducible environment for everyone involved.
• The puppet manifests mean that just a vagrant provision then waiting is enough to bring everything up to date with the latest master.
• Self contained stack, you can see what is running at any point.
• Closer to production. We mostly develop on OSX, but deploy to Linux, this gives us both.
• Shared URLs for testing. We can pop a URL from our vagrant machines into Hipchat, and other members of the team can use it locally, without having to change it. vagrant-dns is a big win for us there.
• Easy to install. Install virtualbox, vagrant, get the Vagrantfiles, run vagrant up.
• While there’s nothing in the vagrant configurations that couldn’t be as easily done with some scripting for the host machine and remove the need for virtualisation, it’s handy that when it goes wrong a fresh rebuild is just a destroy and up away. Extremely useful for testing system wide settings.

The Disadvantages

• A from-nothing vagrant up takes 25 minutes and downloads about 2Gb (1.2Gb for the base box, the rest for code + dependencies + extras).
• vagrant provision to update to latest can take up to 10 minutes depending on speed of the connection and the host machine, so it’s not that easy to ‘just test’ something. You can update the individual services manually, but then you have the problem that we started with, making sure that everyone has the same code.
• It’s hard work for the host machine. The box we have configured has 2 cores and 2Gb of RAM allocated. On a 4Gb Macbook Air, that can start getting a little close to resource starvation, particularly with an IDE and a debugger running.
• Debugging isn’t as easy as I’d like, setting up the debugger is fairly involved in settings, and you can only really debug one thing at a time. Can be awkward when you’re trying to trace values across multiple services.
• The configuration we have isn’t as close to production as I’d like (no nginx, no caching, no centralised logging) but this is just a matter of spending more time.
• It doesn’t entirely solve ‘it works for me’. It just becomes ‘it works on my vagrant’. Fortunately, instances of that seem to be a lot less common.
• Random virtualbox/vagrant/host problems. We’ve had boxes crash, networks go away and all manner of strange things. At least with the code living on the host machine, we’ve not lost work when that happens.

While it seems that there’s more disadvantages than advantages, overall the reproducibility and simplicity of reducing updating to a single command far outweigh the drawbacks of working in a virtualised environment.

Future plans

Most of the future plans for this revolve around gradually bringing it in line with the production environment without losing the flexibility that we have gained.

• Use the production puppet manifests where possible. Our infrastructure is entirely puppet controlled, so I’d like to increase the reuse where possible.
• Create a version that uses the multi-vm capability of vagrant to simulate a cluster, with each service separately. Could be handy for looking at scaling/communication problems.
• See if we can reduce provision time even further, with possible build optimisations. This may then transfer to our deployment system.
• Move to vagrant 1.2 and test out some other backends.
• Remove the screen based development start script and move to something more production-like. (This is only used in the vagrant box, the live deployments use proper init scripts.)

Overall, the use of vagrant has been a big win for us as a company, and has reduced a lot of the problems we were having. There’s still some work to be done until we’re completely happy with it, but I’d recommend that anyone looking at building this type of project take a serious look to see if it suits them.

The post Many projects with Vagrant and Puppet appeared first on Server Density Blog.