With Ubuntu 16.04 and devicemapper as the docker storage driver, occasionally after a system boot the docker service won’t startup. The logs (systemd status docker.service) show something like:

Dec 05 15:18:23 xxx systemd[1]: Failed to start Docker Application Container Engine.
Dec 05 15:18:23 xxx systemd[1]: docker.service: Unit entered failed state.
Dec 05 15:18:23 xxx systemd[1]: docker.service: Failed with result 'exit-code'.
Dec 05 15:18:23 xxx systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Dec 05 15:18:23 xxx systemd[1]: Stopped Docker Application Container Engine.
Dec 05 15:18:23 xxx systemd[1]: docker.service: Start request repeated too quickly.
Dec 05 15:18:23 xxx systemd[1]: Failed to start Docker Application Container Engine.
Dec 05 15:18:39 xxx systemd[1]: Stopped Docker Application Container Engine.

You can react to this one of two ways. Spend ages poking around, trying to start the service and find an explanation, or just run lvscan and discover that that volume being used by devicemapper is inactive and docker therefore can’t start:

  ACTIVE            '/dev/ubuntu-vg/root' [51.76 GiB] inherit
  ACTIVE            '/dev/ubuntu-vg/swap_1' [8.00 GiB] inherit
  inactive          '/dev/docker/thinpool' [57.00 GiB] inherit

The solution is to run vgchange -ay <name_of_volume_group> (or just vgchange -ay if you don’t need to be specific) which activates the volume and will then allow you to start the docker service. <name_of_volume_group> can be retrieved via vgs.

Worth noting that the fix here is nothing magical, it’s just the standard way you activate a volume. The crux of the issue is why the volume was deactivated in the first place – it’s seemingly a race condition during the boot process (I’ve seen suggestions of various culprits), but I’ve yet to pin down the exact context.

The event wrapped up with a cracking keynote by Adrian Colyer entitled “Making sense of it all” in which he presented his perspective on where the world of containerisation is today, and where it’s headed, supported by evidence gleaned through direct discussions with many movers and shakers in the industry. You should definitely go check out the recording!

Having spent more time talking about containers in two days than the last two months, I thought I’d share some thoughts based on takeaways from the conference and my own experience, in case it helps others undertaking a similar journey. There’s a TL;DR at the end if you need it

Containers are in production

A common concern about using containers is that they’re simply not ready for mainstream production use yet – they’re too new, too immature, too insecure etc. That’s simply not the case. An O’Reilly survey carried out in May last year found 40% of respondents were using containers in production, with more than half planning to enter production within 6-12 months.

A similar survey by ClusterHQ around the same time found similar results, with 38% of respondents currently in production. Their 2016 survey (due for release shortly) shows this figure to have climbed to 76%, supporting last year’s predictions (see Adrian’s talk for more details).

So containers are no longer confined to dev/QA environments – they’re very much out there in the wild.

Beware the concept count

For many (most?) people, the first encounter they have with containers will be via Docker. When you read/watch the various introductory Docker tutorials out there, it can seem pretty simple to spin one up. There is a huge difference between running a container on your laptop and deploying a containerised application stack to production however.

Once you start down the containerisation path, the concepts come thick and fast, with an ever-increasing number of concerns you have to investigate, learn, make judgments and decisions about, and build confidence in (both within your team and the wider organisation). You need solutions to networking, security, service discovery, storage, logging, monitoring and so on.

Don’t underestimate the learning curve. If you, your team or your organisation shies away from learning new things, or has a tight deadline, chances are it’s not the right time for you to explore containers. If you don’t have the problems that containers solve, then it’s definitely not the right time.

Be pragmatic

As alluded to above, the Docker/container ecosystem is huge, with new tools and approaches emerging every day. It can be tempting to feel like you have to learn everything.

The reality is you don’t need everything. As exciting and shiny as a solution may seem, do you actually have the problem it’s trying to solve? A funky auto-scaling setup may sound perfect, but if your fluctuations are mild and predictable enough, then maybe a manual approach would suffice for now, leaving you with one less concept to care about?

Just because there are lots of people choosing Kubernetes, or extolling the virtues of Weave, or advocating Project Calico doesn’t mean that each of those is suitable or essential for your circumstances, especially if your environment is relatively simple rather than involving 1000s of containers across dozens of hosts.

Speaking for myself, after surveying the various options, I settled on Rancher for our particular orchestration needs – it’s pretty simple to setup, with a lot of stuff available out of the box (overlay network, service discovery, load balancing, clean UI). Sure there are lots of capabilities it doesn’t have that you can find elsewhere, some of which a very strong case could be made for. However we just don’t need anything more right now, so the added complexity has no justification.

Obviously Rancher might not be right for you – ultimately there is no single “best solution”. Context is king – be realistic about what your needs really are and take care not to over-engineer.

Miniliths vs microservices

No that isn’t a typo, it’s a word I made up… One of the main drivers for the adoption of containers is the adoption of a microservices architecture in place of a monolithic one. However, just like containers, microservices bring with them a whole world of painful overheads. For every microservice you introduce, there’s a cost to pay, and you need to be careful any benefits aren’t swallowed by the downsides.

Where appropriate, consider a “minilith” approach i.e. keep the number of microservices to a minimum, grouping related functionality as pragmatically as possible whilst leaving doors open to allow them to be split out further later if required (which may possibly be never).

Yes, this won’t be pure microservices, and it may hurt a bit inside, but it’ll mean reduced complexity with fewer processes to monitor, log, deploy, recover, scale and so on.

Stand on the shoulders of giants

With the ecosystem moving so fast, rolling your own solutions should be a last resort. It might be tempting to build a custom orchestration framework, but unless you keep it actively developed it won’t take long until it falls behind third party solutions and you’ll miss out on capabilities you would otherwise get for free. Not to mention having a harder job of getting others up to speed on how it works.

Once the pace of change slows down and the ecosystem matures, then sure, give it a go (if you’re sure it’s absolutely essential). But until then, don’t be tempted into striking out alone unless you can clearly quantify the benefits.

Don’t forget the humans

All the talk of automation makes it easy to overlook the human element of developing and maintaining systems. However slick a solution you may have in mind, without the right culture and mindset amongst your teams, you’re going to have problems.

Remember that DevOps is about communication and collaboration as much as automation, and can require a cultural shift to facilitate this. Having people on your team who don’t work well with others is a much more important problem to solve than a bit of automation. And if you can’t change behaviours, perhaps it’s time to manage people out.

Networking (the human kind)

Watching session recordings, reading blog posts etc is all well and good, but you really can’t beat networking with others and sharing thoughts and experiences. This might sound obvious – you could say the same about any topic. However, given how early-on we are in the process of adopting containers, it feels especially applicable at the moment.

There’s a lot of collective wisdom that is yet to be distilled down into blog posts and best practices that you can only really access by talking to people who have been there and done it (and quite possibly got burnt by it!). Put aside your British reserve (if you are British, obviously) and get out there and interact!

Storage is a persistent problem

When asked in the panel discussion which area we felt Docker is most lacking in, each of the three of us picked storage. This was a common theme amongst the many discussions I had too – it’s pain that is clearly widely felt. Even something as simple as inspecting the contents of a Docker volume from the command line is harder than it needs to be.

Yes, there are various storage solutions out there (e.g. Gluster, ClusterHQ, Convoy), and new ones entering the market all the time. But this is a tough problem to solve – if you want distributed storage you have to bear in mind the laws of physics in terms of latency and architect your application to handle your use cases and available IO accordingly.

Basically, for anything stateful, your life is going to be much harder than if it was stateless. Where possible, farm data off somewhere to a non-containerised external service.

Finding good people is hard

I suspect I’m preaching to the choir here (literally everybody I speak to has agreed with this), but one trend that just won’t go away is that finding good people with relevant skills is hard. Even harder than building distributed storage systems. Bearing in mind how new the ecosystem is, most knowledge and experience lives within developers and engineers that are not yet ready to re-enter the job market.

Consequently this means if you need extra pairs of hands, you’re most likely going to need to turn to the contractor or consultant markets. Which may not be the end of the world (though possibly the end of your budget). But don’t rely on filling a permanent vacancy in a hurry, especially if you’re outside London.

This is another reason to keep your concept count as low as possible and lean on third party offerings – if somebody on your team leaves, how quickly will it take somebody new to get fully up to speed?

TL;DR

I appreciate much of this is fairly conventional wisdom that could be applied to many more areas than containers. But given the penchant developers have for shiny things, it’s easy for it to fall by the wayside sometimes.

The world of containers is huge and can feel daunting, possibly a mountain you daren’t tackle. It’s definitely not a silver bullet, but once you cut through the hype and take the knife to the concept count to eliminate anything you really don’t need, it becomes far more approachable. Don’t be afraid to give it a go (if appropriate), but keep your eyes open and talk to others. And make sure you get that culture right.

I’d love to hear about your experiences – feel free to comment or contact me via whichever channel you prefer.

I wanted to show hidden files, so naturally I went to Preferences to look for an appropriate setting and found one that looked just the ticket: Settings > Core Settings > Exclude VCS Ignored Paths.

I figured that unticking this setting would do the job, but it made no difference – confusingly, it turns out that this is a global setting used by functionality such as fuzzy searching, and there is another setting that overrides this for the sidebar (a.k.a. tree view). Because Atom is modular, most of its components exist as independent packages. The setting you need is found in Settings > Packages > Core Packages > Tree View:

Here you’ll find “Hide Ignored Names” and “Hide VCS Ignored Files” settings – the latter is the one you want to stop the sidebar from ignoring files that your VCS ignores.

The first setting relates to another of the settings back on the main Settings panel (see first screenshot above) – Preferences > Settings > Core Settings > Ignored Names. This allows you to globally ignore file/folder patterns regardless of VCS ignore files. You probably want to leave this on unless you really want to be able to dig around in your `.git` folder within the editor.

Hope this saves somebody a bit of time and frustration as it took me ages to figure out! And I’m clearly not the only one, so hopefully Github will sort this soon.

Something it took me a while to figure out was how to stop Docker from bypassing UFW and exposing mapped ports to the world (due to specifying its own iptables chain). More often than not, I want containers to be restricted to private network access only. One option is to specify an ip address when mapping ports, but that’s a bit clunky and doesn’t work so well when you want to be able access the ports via multiple private ip addresses.

The challenge was making sure not to block outbound or inter-container connectivity in the process.

Having had to set up a number of servers and keep finding myself forgetting one of the steps, I figured it was about time I put this blog to good use and list the necessary commands here:

sudo ufw allow in on docker0
sudo sed -i s/DEFAULT_FORWARD_POLICY=\"DROP\"/DEFAULT_FORWARD_POLICY=\"ACCEPT\"/ /etc/default/ufw
sudo ufw enable
iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE

CSC error CS0182: An attribute argument must be a constant expression, typeof expression or array creation expression of an attribute parameter type

The same code built fine in Visual Studio 2012, and nothing had changed on the build server so clearly something in the most recent commits was triggering a difference in behaviour between the two environments.

It turns out that there’s a compiler bug in C# 4.0 (which our TeamCity server was using) involving null default parameters for attributes (an instance of which had been introduced in a recent commit) e.g.:

public SomeAttribute(SomeType someParameter = null)

When an attribute contains a parameter in its constructor that defaults to null, the compiler incorrectly treats this as being a typeless null literal rather than a constant expression and throws the exception in question.

The proper fix for this is to upgrade the compiler on the build server. However, as that isn’t always immediately practical, as a workaround you can simply cast the null parameter to the type in question e.g.:

public SomeAttribute(SomeType someParameter = (SomeType)null)

Eric Lippert provides more information about this particular bug in this StackOverflow answer.

If you have a max-width set and need to disable it in a cascaded style, you need to set it to none rather than auto i.e.:

img {
    max-width:100%;
}

#some-other-selector img {
    max-width:none;
}

This sounds obvious (and it is really), but because you usually use width:auto to override a previously specified width, I found myself inadvertantly doing the same with max-width and it took a while to realise what I’d done.

Googling didn’t turn up much, but after digging around I discovered that the “TeamCity Web Server” service was in a “Disabled” state. Attempts to do anything with it gave me the message “This service is marked for deletion.”

With the help of this StackOverflow answer, I quickly figured out that I’d started the upgrade process with Windows’ Services viewer open. For whatever reason, this kept a hook into the outgoing TeamCity service which couldn’t be fully uninstalled as a result.

The solution was simple – close the Services window and retry. The installation completed without any further issue.

Whilst this is a bit of an edge case, the fact that the TeamCity error message didn’t directly reference the “TeamCity Web Server” service didn’t help. A clearer message would have sped up the diagnosis.

Update 06/02/14: I’ve just done a quick review of this post and made some changes. There’s loads more I could add but time doesn’t permit right now

Every couple of years I go back to my old school to participate in a careers convention, where people from a range of industries get together to dish out tips and advice to pupils trying to make career choices. A great cause, especially with uni now being a more challenging prospect here in the UK due to the huge hike in tuition fees. Youngsters really need all the help they can get to make sure they pick the path that’s right for them.

I promised those I spoke to this year that I would post a list of links on my blog as a follow-up to find out more about web development. I should stress that this list is by no means exhaustive; I put it together quite quickly and there’s plenty more I could have included. For example, whilst I’ve tried to include links relating to a range of technologies, inevitably there’s more about the areas I have more experience with. Hopefully though it can still provide plenty of reading material and food for thought to help in evaluating whether web development is a subject worth investigating further as a potential career or perhaps just as a hobby.

I’ll try to update the list from time to time when I think of more (suggestions also welcome via the comments). I’m also hoping to follow this blog post up with some further thoughts on choosing web development as a career.

Without further ado, here are the links:

Programming Languages / Frameworks

• HTML5/CSS/JavaScript – Open technologies at the heart of the web – learn these first!
• Node.js – Increasingly popular serverside JavaScript platform
• Ruby – Widely used
• Ruby on Rails – One of the most popular web frameworks, built on Ruby
• Python – A good starter language, often taught on uni courses etc.
• PHP – Been around a long time and still widely used, though not considered too cutting edge these days
• ASP.NET – Microsoft’s web platform, used a lot in the enterprise (see also http://asp.net/ and http://msdn.com/)
• Java – Mostly found in the enterprise, not a good choice if you want to keep ahead of the curve!

Online video/interactive training

• Treehouse – Web design & development
• Codecademy – Interactive coding courses in a range of languages including JavaScript, HTML/CSS and Ruby
• CodeYear – Learn to program in a year (part of Codecademy)
• PluralSight – Large library of videos for a range of technologies, not free but reasonably priced
• TekPub – Screencasts on a range of topics, some of which are free (update: has been bought out by PluralSight)
• appendTo – jQuery/JavaScript training
• NodeTuts – Node.js screencasts
• RailsCasts – Ruby on Rails screencasts

(e)Books

Look out for deals of the day from most of these – they tend to have an ebook or two available each day for around £7-£10 (see my Twitter list covering many of the ebook publishers).

• O’Reilly – My favourite, some great JavaScript books such as Douglas Crockford’s “Javascript: The Good Parts“
• Manning Publications
• The Pragmatic Bookshelf
• Apress
• InformIT
• Also take a look at this huge list of free programming books

Resources

• Move the Web Forward – One stop shop for keeping abreast of web standards and getting involved in the community
• Webmaker – A Mozilla project “dedicated to helping you create something amazing on the web”
• frontend-dev-bookmarks – A huge list of frontend development resources
• jQuery – Most popular JavaScript library for rapid web development (though use with caution – easy to produce spaghetti code if you’re not careful)
• Twitter Bootstrap – UI framework that makes it really easy to get a good UI up and running very quickly
• Google Reader – RSS aggregator – subscribe to blogs and news sites and have all articles brought together in one place to save having to check each site manually (update: service has closed)
• Feedly – A drop-in replacement for Google Reader, a good way to aggregate articles from across the web into a single feed (or multiple if you prefer)
• TweetDeck – Useful Twitter client for monitoring topics and lists of accounts to help breakdown the wealth of information that flows through Twitter

Databases

• MySQL – Most popular open source relational database server (free)
• PostgreSQL – Another popular open source relational database (free)
• SQL Server – Microsoft’s SQL database, common in the enterprise (free cutdown editions available – Express and Compact Edition)
• MongoDB – Popular open source “NoSQL” database (free) – an alternative approach to “traditional” databases (has pros and cons)
• CouchDB – Another popular NoSQL database

News / Views / Tutorials

• Hacker News – Anything and everything of interest to a techie
• Reddit Programming – As above – see http://reddit.com/r/LearnProgramming/ and other related sections also
• Smashing Magazine – Large selection of articles, resources etc relating to web design
• HTML5 Rocks – Google resource with a host of articles, tutorials and resources on HTML5 and modern browser techniques
• DailyJS – JavaScript/Node.js news
• Ruby Inside – Ruby news and tutorials
• Techmeme – General tech news, usually one of the first places you’ll find the big stories
• Twitter – One of the best ways of keeping on top of everything – many blogs tend to post on their Twitter feeds when new articles are posted, plus there’s an endless stream of useful links being shared

Help

• StackOverflow – Q&A site for developers with specific problems, usually related to specific source code
• Programmers – Q&A site for programmers with higher-level conceptual problems

Mailing Lists

• JavaScript Weekly
• HTML5 Weekly
• Ruby Weekly

Podcasts

• Herding Code – Weekly podcast on a range of topics
• Boagworld – Longest running web design podcast
• The Changelog – Weekly podcast on the world of Open Source
• .NET Rocks! – .NET
• Hanselminutes – .NET/Windows focus
• NodeUp – Node.js
• 5by5 – A number of podcasts on a range of topics

Web Hosts

• Heroku – For Ruby, Node.js, Python, Java and more (requires understanding of source control – see below)
• AppHarbor – For .NET (requires understanding of source control – see below)
• Amazon Web Services (AWS) – Amazon’s cloud platform, a more granular and pay-as-you-go way of choosing your hosting infrastructure
• Azure – As above, but Microsoft’s equivalent platform
• WebHostingTalk – Great forums to help find/discuss any type of web host
• LowEndBox – Useful place to find a really cheap and basic (but functional) server to manage yourself

Jobs

• StackOverflow Careers
• CoderStack (update: service has closed)
• CW Jobs
• Also see this huge list of frontend developer interview questions

Source Code / Version Control

• Git – One of the most popular version control systems out there
• Mercurial – Similar to Git, slightly easier learning curve
• Github – The most popular source code host, based on Git
• BitBucket – Supports Git and Mercurial

There are a number of implementations available for different platforms, and the purpose of this blog post is to highlight a feature in dotless – the .NET implementation – which I’ve had to call on occasionally but often forget about in the interim, so I’m hoping this will stop me from forgetting

From time to time (when needs absolutely must) it’s necessary to use the proprietary filter attribute to accommodate IE (much better if you can use something like CSS3 PIE but like I say, sometimes needs must). For example:

.some-class {
    filter: progid:DXImageTransform.Microsoft.Shadow(Strength=15, Direction=0, Color='#000');
}

However, because of the abnormal syntax this involves, it causes parsing issues with dotless. I’ve read about a number of ways of working around this, but the only one I’ve found to be properly reliable (e.g. including the ability to use variables within the filter) is the use of dotless’ formatString function (which appears to be undocumented).

This has a similar signature to String.Format i.e. it takes as its first argument a string containing placeholders where values need inserting ({0}, {1} etc) followed by an argument for each value to be substituted. So using the above example, your CSS would look something like this:

.some-class {
    filter: formatString("progid:DXImageTransform.Microsoft.Shadow(Strength={0}, Direction={1}, Color={2})", @strength, @direction, @color);
}

Or perhaps more usefully, making use of a mixin:

.some-class(@strength = 15, @direction = 0, @color = '#000') {
    filter: formatString("progid:DXImageTransform.Microsoft.Shadow(Strength={0}, Direction={1}, Color={2})", @strength, @direction, @color);
}

Obviously formatString can be used for much more than this particular scenario, it’s usefulness isn’t restricted to working around annoying proprietary CSS attributes that really are way past their sell by date…

I got the following exception (where <URL_TO_GIT_REPOSITORY> is a placeholder I’ve substituted for the original URL):

jetbrains.buildServer.VCS – Error while loading changes for root <URL_TO_GIT_REPOSITORY> #master {instance id=43, parent id=3}, cause: ‘git fetch’ command failed.
stderr: The repository at directory ‘D:\Development\Infrastructure\.BuildServer\system\caches\git\git-4D05D25B.git’ cannot be opened or created, reason: The specified directory D:\Development\Infrastructure\.BuildServer\system\caches\git\git-4D05D25B.git is already used for another remote <URL_TO_GIT_REPOSITORY> and cannot be used for others (<URL_TO_GIT_REPOSITORY> ). Please specify the other directory explicitly.

What this essentially seems to be saying is that it can’t checkout changes from my remote Git repository because its local repository is currently tracking a different remote repository… which just happens to have the same URL! This didn’t make any sense – why was it thinking there were two different URLs here when they were clearly the same?

Well the eagle-eyed amongst you may spot that there is actually a subtle difference between the two reported URLs – the one in brackets has a trailing space after it. Turns out that I was setting the VCS root URL via a build parameter and had accidentally left the trailing space in there for one of my build configurations, giving me two different values in theory, even though to a human eye they looked the same. Clearly TeamCity doesn’t trim whitespace from VCS root URLs, otherwise it would have normalised them and treated them as one and the same.

Needless to say, removing the trailing space prompted everything to spring back into life.