Someone, somewhere, is the person who pressed go on that software update. And right now they know exactly what they did and what it has done. And as someone who knows that sinking feeling when you realise you’ve screwed up at work, I cannot imagine the state of them right now.

— Jim Waterson (@jimwaterson) July 19, 2024

Today’s tweet about the Crowdstrike incident, which seemingly brought the modern IT world to a standstill, reminded me of the darkest day of my professional life — when I accidentally knocked out internet access in a city of over 200,000 people.

It was my second year of university and I worked for a the largest local ISP in my home city as a junior system administrator. We had a large wireless network (~100km in diameter) covering our whole city and many surrounding rural areas. This network was used by all major commercial banks and many large enterprises in the area (bank branches, large factories, radio stations, etc).

To cover such a large area (in Ukraine in early 2000s), about 50% of which were rural villages and towns, we basically had to build a huge wifi network, that had a very powerful antenna in the center and many smaller regional points of presence would connect to it using directional wifi antennas and then distribute the traffic locally. The core router connected to the central antenna was located at the top floor of the highest building in the area about 20 min away from our office.

One day I was working on some monitoring scripts for the central router (which was basically a custom-built FreeBSD server). I’d run those scripts on a local stand I had on my table, make some changes, run it again, etc. We did not have VMs back then, so experimental work would happen on real hardware that was a clone of a production box. In the middle of my local debugging, I received a monitoring alert from our production saying that our core router had some (non-critical) issues. Since I was on-call that day, I decided take a look. Fixing the issue on the router, I went back to my debugging and successfully finished the job after about an hour.

And that’s where things went wrong… When I wanted to shut down my local machine, I switched to a terminal that was connected to the box, typed “poweroff”, pressed Enter… and only then realized that I did it on a wrong server! I had that second terminal window opened ever since the monitoring alert an hour ago, and now I ended up shutting down the core router for our whole city-wide network!

What’s cool is that there was no blame in the aftermath of the incident. The team understood the mistake and focused on fixing the problem. We ended up having to drive to the central station and manually power the router back on. Back then we did not have any remote power management set up for that server and IPMI did not exist back then. Dark times indeed!

As a result of that mistake, our whole city’s banking infrastructure and a bunch of other important services were down for ~30 minutes. Following the incident, we have made a number of improvements to our infrastructure and our processes (I don’t remember the details now) making the system a lot more resilient to similar errors.

Looking back now, huge kudos to my bosses for not firing me back then! This incident profoundly influenced my career in many ways:

First, the thrill of managing such vast infrastructures made me want to stay in technical operations rather than shifting to pure software development, a path many of my peers chose at the time. Then, having experienced such a massive error firsthand, I’ve always done my absolute best to safeguard my systems against failures, optimizing for quick recovery and being paranoid about backups and redundancy. Finally, it was a pivotal moment in my understanding of the value of blameless incident process long before the emergence of the modern blameless DevOps and SRE cultures — a management lesson that has deeply informed my approach to leadership and system design ever since.

1.  Who is Oleksiy?

I have spent most of my career in technical operations (system administration, later called DevOps, nowadays encompassed by platform engineering and SRE disciplines). Along the way, I worked at Percona as a MySQL performance consultant and then operated some of the largest Ruby on Rails applications in the world, all the while following the incredible story of Shopify’s development and growth.

Finally, after decades of work in operations, when a startup I was at got acquired by Elastic, I decided to move into software engineering. After 5 years there, I needed a bigger challenge, which felt like the right moment to join Shopify.

I started with the Storefronts group (the team responsible for Storefront themes, all the related infrastructure, and the Storefront rendering infrastructure) at Shopify at the beginning of 2022. Two years later, I can confidently say that Shopify’s culture is unique. I enjoy working with the team here due to the incredible talent density I have never encountered. Every day, I am humbled by the caliber of people I can work with and the level of problems I get to solve.

2.  What is the role of the Principal Engineer at Shopify?

Before joining Shopify, I was excited about all the possibilities associated with the Principal Engineer role. Immediately, I was surprised at how diverse the Principal Engineering discipline was at the company. We have a range of engineers here, from extremely deep and narrow experts to amazing architects coordinating challenging projects across the company. Even more impressive is that you have a lot of agency in the shape of a Principal Engineer you will be, provided that the work aligns with the overarching mission of making commerce better for everyone. After 2 years with the company, I found myself in a sweet spot of spending ~75% of my time doing deep technical work across multiple areas of Storefronts infrastructure, and the rest is spent on project leadership, coordination, etc.

3.  The recent tweet by Shopify Engineering shows impressive results achieved by your system. What is Shopify’s overall architecture?

The infrastructure at Shopify was one of the most surprising parts of the company for me. I have spent my whole career building large, heavily loaded systems based on Ruby on Rails. Joining Shopify and knowing upfront a lot about the amount of traffic they handled during Black Friday, Cyber Monday (BFCM), and flash sales, I was half-expecting to find some magic sauce inside. But the reality turned out to be very different: the team here is extremely pragmatic when building anything. It comes from Shopify’s Founder and CEO Tobi Lütke himself: if something can be made simpler, we try to make it so. As a result, the whole system behind those impressive numbers is built on top of fairly common components: Ruby, Rails, MySQL/Vitess, Memcached/Redis, Kafka, Elasticsearch, etc., scaled horizontally.

What makes Shopify unique is the level of mastery the teams have built around those key components: we employ Ruby core contributors (who keep making Ruby faster), Rails core contributors (improving Rails), MySQL experts (who know how to operate MySQL at scale), and we contribute to and maintain all kinds of open-source projects that support our infrastructure. As a result, even the simplest components in our infrastructure tend to be deployed, managed, and scaled exceptionally well, leading to a system that can scale to many orders of magnitude over the baseline capacity and still perform well.

4.  What is Shopify’s tech stack?

Given that databases (and stateful systems in general) are the most complex components to scale, we focus our scaling on MySQL first. All shops on the platform are split into groups, each hosted on a dedicated set of database servers called a pod. Each pod is wholly isolated from the rest of the database infrastructure, limiting the blast radius of most database-related incidents to a relatively small group of shops. Some more prominent merchants get their dedicated pods that guarantee complete resource isolation.

Over the past year, some applications started relying on Vitess to help with the horizontal sharding of their data.

On top of the database layer is a reasonably standard Ruby on Rails stack: Ruby and Rails applications running on Puma, using Memcached for ephemeral storage needs and Elasticsearch for full-text search. Nginx + Lua is used for sophisticated tasks, from smart routing across multiple regions to rate limiting, abuse protection, etc.

This runs on top of Kubernetes hosted on Google Cloud in many regions worldwide, making the infrastructure extremely scalable and responsive to wild traffic fluctuations.

Check the full Shopify tech stack at Stackshare.

What are Pods exactly?

The idea behind pods at Shopify is to split all of our data into a set of completely independent database (MySQL) clusters using shop_id as the sharding key to ensure resource isolation between different tenants and localize the impact of a “noisy neighbor” problem across the platform. 

Only the databases are podded since they are the hardest component to scale. Everything else that is stateless is scaled automatically according to the incoming traffic levels and other load parameters using a custom Kubernetes autoscale.

5. Is the monolith going to be broken into microservices?

Shopify fully embraces the idea of a Majestic Monolith—most user-facing functionality people tend to associate with the company is served by a single large Ruby on Rails application called “Shopify Core.” Internally, the monolith is split into multiple components focused on different business domains. Many custom (later open-sourced) machinery have been built to enforce coding standards, API boundaries between components, etc.

The rendering application behind all Shopify storefronts is completely separate from the monolith. This was one of the cases where it made perfect sense to split functionality from Core because it is relatively simple. Load data from a database, render Liquid code, and send the HTML back to the user – the absolute majority of requests it handles. Given the amount of traffic on this application, even a small improvement in its efficiency results in enormous resource savings. So, when it was initially built, the team set several strict constraints on how the code is written, what features of Ruby we prefer to avoid, how we deal with memory usage, etc. This allowed us to build a pretty efficient application in a language we love while carefully controlling memory allocation and the resources we spend rendering storefronts.

In parallel with this effort, the Ruby infrastructure team (working on YJIT, among other things) has made the language significantly faster with each release. Finally, in the last year, we started rewriting parts of this application in Rust to improve efficiency further.

Answering your question about the future of the monolith, I think outside of a few other localized cases, most of the functionality of the Shopify platform will probably be handled by the Core monolith for a long time, given how well it has worked for us so far using relatively standard horizontal scalability techniques.

6. How do you do testing?

Our testing infrastructure is a multi-layered set of checks that allows us to deploy hundreds of times daily while keeping the platform safe. It starts with a set of tests on each application: your typical unit/integration tests, etc. Those are required for a change to propagate into a deployment pipeline (based on the Shipit engine, created by Shopify and open-sourced years ago.

During the deployment, a very important step is canary testing: a change will be deployed onto a small subset of production instances, and automation will monitor a set of key health metrics for the platform. If any metrics move in the wrong direction, the change is automatically reverted and removed from production immediately, allowing developers to figure out what went wrong and try again when they fix the problem. Only after testing a change on canaries for some time the deployment pipeline performs a full deployment. The same approach is used for significant schema changes, etc.

7. How do you do deployments?

All Shopify deployments are based on Kubernetes (running on GCP), so each application is a container (or a fleet of containers) somewhere in one of our clusters. Our deployment pipeline is built on the Shipit engine (created by Shopify and open-sourced years ago). Deployment pipelines can get pretty complex, but it mostly boils down to building an image, deploying it to canaries, waiting to ensure things are healthy, and gradually rolling out the change wider across the global fleet of Kubernetes clusters.

Shipit also maintains the deployment queue and merges multiple pull requests into a single deployment to increase the pipeline’s throughput.

8. How do you handle failures in the system? 

The whole system is built with many redundancy and horizontal auto-scaling (if possible), which helps prevent large-scale outages. But there are always big and small fires to handle. So, we have a dedicated site reliability team responsible for keeping the platform healthy in the face of constant change and adversarial problems like bots and DDoS attacks. They have built many automated tools to help us handle traffic flashes and, if needed, degrade gracefully. Some interesting examples: they have automated traffic analysis tools helping them scope ongoing incidents down to specific pods, shops, page types, or traffic sources; then the team can control the flow of traffic by pod or shop, re-route traffic between regions, block or slow down requests from specific parts of the world, prioritize particular types of traffic and apply anti-adversarial measures across our network to mitigate attacks.

Finally, each application has an owner team (or a set of teams) that can be paged if their application gets unhealthy. They help troubleshoot and resolve incidents around the clock (being a distributed company helps a lot here since we have people across many time zones).

9. What challenges are you working on right now in your team?

We have just finished a large project to increase the global footprint of our Storefront rendering infrastructure, rolling out new regions in Europe, Asia, Australia, and North America. The project required coordination across many different teams (from networking to databases to operations, etc.) and involved building completely new tools for filtered database replication (since we cannot replicate all of our data into all regions due to cost and data residency requirements), making changes in the application itself to allow for rendering without having access to all data, etc. This large effort has reduced latency for our buyers worldwide and made their shopping experiences smoother.

Next on our radar are further improvements in Liquid rendering performance, database access optimization, and other performance-related work.

For most of my career (for at least 15-17 years before joining Elastic) I have worked in small to medium-sized startups, always in SaaS, moving really fast and having my impact on the business be mostly tied to my ability to ship. I loved that, even though it was often painful and stressful. My brain ended up being trained to derive dopamine from the constant feeling of shipping, constant feeling of overcoming challenges and solving problems.

Then, we got acquired by Elastic – a truly amazing company, honestly, the best company I have ever worked for and, unfortunately, a company that ships packaged software with its inherent effects on development process. At first, while my projects revolved around security and compliance, then around internal code migrations, integration into the Elastic ecosystem, etc, I felt really happy – I would do the really challenging work and derive pleasure from overcoming those challenges.

Unfortunately, over time the initial rush of excitement faded away and I have realized, that the most challenging problems within our product have been solved and I ended up in a position of working on a packaged product, building features with a release cycle of 6-8 weeks. The product is amazing, the features are really exciting, but the very long feedback cycle simply did not work for me after so many years in SaaS.

I still believe in Elastic, I love the company, the team and the amazing culture we have built over the years. But I need to ship faster, move faster, get feedback from my users sooner. After considering different options, I have decided to join Shopify – a large and fast moving Rails-based company, where I hope to get a chance to once again experience the thrill of fast delivery and tight feedback loops. Let’s see how it goes

For any modern technology company, a comprehensive application test suite is an absolute necessity. Automated testing suites allow developers to move faster while avoiding any loss of code quality or system stability. Software development has seen great benefit come from the adoption of automated testing frameworks and methodologies, however, the culture of automated testing has neglected one key area of modern web application serving stack: web application edge routing and multiplexing rulesets.

From modern load balancer appliances that allow for TCL based rule sets; local or remotely hosted varnish VCL rules; or in the power and flexibility that Nginx and OpenResty make available through LUA, edge routing rulesets have become a vital part of application serving controls.

Over the past decade or so, it has become possible to incorporate more and more logic into edge web server infrastructures. Almost every modern web server has support for scripting, enabling developers to make their edge servers smarter than ever before. Unfortunately, the application logic configured within web servers is often much harder to test than that hosted directly in application code, and thus too often software teams resort to manual testing, or worse, customers as testers, by shipping their changes to production without edge routing testing having been performed.

In this post, I would like to explain the approach Swiftype has taken to ensure that our test suites account for our use of complex edge web server logic
to manage our production traffic flow, and thus that we can confidently deploy changes to our application infrastructure with little or no risk.

Our Web Infrastructure

Before I go into details of our edge web server configuration testing, it may be helpful to share an overview of the infrastructure behind our web services and applications.

Swiftype has evolved from a relatively simple Rails monolith and is still largely powered by a set of Ruby applications served by Unicorn application servers. To balance traffic between the multitude of application instances, we use Haproxy (mainly for its observability features and the fair load balancing implementation). Finally, there is an OpenResty (nginx+lua) layer at the edge of our infrastructure that is responsible for many key functions: SSL termination and enforcement, rate limiting, as well as providing flexible traffic management and routing functionality (written in Lua) customized specifically for the Swiftype API.

Here is a simple diagram of our web application infrastructure:

Swiftype web infrastructure overview

Testing Edge Web Servers

Swiftype’s edge web server configuration contains thousands of lines of code: from Nginx configs to custom templates rendered during deployment, to complex Lua logic used to manage production API traffic.Any mistake in this configuration, if not caught in testing, could lead to an outage at our edge, and considering that 100% of our API traffic is served through this layer, any outage at the edge is likely to be very impactful to our customers and our business. This is why we have invested time and resources to build a system that allows us to test our edge configuration changes in development and on CI before they are deployed to production systems.

Testing Workflow Overview

The first step in safely introducing change is ensuring that development and testing environments are quarantined from production environments. To do this we have created an “isolated” runtime mode for our edge web server stack. All changes to our edge configurations are first developed and run in this “isolated” mode. The “isolated” mode has no references to production backend infrastructure, and thus by employing the “isolated” mode, developers are able to iterate very quickly in a local environment without fear of harmful repercussions. All tests are written to run as part of the “isolated” mode employ a mock server to emulate production backends and primarily focus on the unit-testing of specific new features that are being implemented.

When we are confident enough in our unit-tested set of changes, we could run the same set of tests in an “acceptance testing” mode when the mock server used in isolated tests is replaced with an Haproxy load balancer with access to production networks. Working on tests and running them in this mode allows us to ensure with the highest degree of certainty that our changes will work in a real production environment since we exercise our whole stack while running the test suite.

Testing Environment Overview

Our testing environment employs Docker containers to serve in place of our production web servers. The test environment is comprised of the following components:

• A loopback network interface on which a full complement of production IPs are configured to account for every service we are planning to test (e.g. a service foo.swiftype.com pointing to an IP address 10.1.0.x in production is tested in a local “isolated” testing environment with IP 10.1.0.x assigned to an alias on the local loopback interface). This allows us to perform end-to-end testing: DNS resolution, TCP service connections to a specific IP address, etc. without needing access to production, nor local /etc/hosts or name resolution changes.
• For use cases where we are testing changes that are not represented in DNS (for example, when preparing edge servers for serving traffic currently handled by a different service), we may still employ local /etc/hosts entries to point the DNS name for a service to a local IP address for the period of testing. In this scenario, we ensure that our tests have been written in a way that is independent of the DNS configuration, and thus that the tests can be reused at a later date, or when the configuration has been deployed to production.
• An OpenResty server instance with the configuration we need to test.
• A test runner process (based on RSpec and a custom framework for writing our tests).
• An optional Mock server. (As noted above, this might be docker in a local test environment, or in CI, and is likely to be used as part of the test runner process, where it emulates an external application/service; serves in place of a production backends; or acts as a local Haproxy instance running a production configuration and may even route traffic to real production backends.

Isolated Testing Walkthrough

Here is how a test for a hypothetical service foo.swiftype.com (registered in DNS as 1.2.3.4) is performed in an isolated environment:

1. We automatically assign 1.2.3.4 as an alias on a loopback interface.
2. We start a mock server listening on the localhost configured to respond on the same port used by the foo.swiftype.com Nginx server backend (in production, there would be haproxy on that port) with a specific stub response.
3. Our test performs a DNS resolution for foo.swiftype.com, receives 10.1.0.x as the IP of the service, connects to the local Nginx instance listening on 10.1.0.x (bound to a loopback interface) and performs a test call.
4. Nginx, receiving the test request, performs all configured operations and forwards the request to a backend, which in this case is handled by the local mock server. The call result is then returned by Nginx to the test runner.
5. The test runner performs all defined testing against the server response: These tests can be very thorough, as the test runner has access to the server response code, all headers, and also the response body, and can thus confirm that all data returned meets each test’s specifications before concluding if the process as a whole has passed or failed test validation.
6. Specific to isolated testing: In some use cases, we may validate the state of the Mock server, verifying that it has received all call we expected it to receive and that each call represented the data and headers expected. This can be very useful for testing changes where our web layer has been configured to alter requests (rewrite, add or remove headers, etc.) prior to passing them to a given backend.

Here is a diagram illustrating a test running in an isolated environment:

An isolated testing environment

Acceptance Testing Walkthrough

When all of our tests have passed in our “isolated” environment, and we want to make sure our configurations work in a non-mock, physically “production-like” environment (or during our periodic acceptance test runs that must also run in a production mirroring environment), we use an “acceptance testing” mode. In this mode, we replace our mock server with a real production Haproxy load balancer instance talking to real production backends (or a subset of backends representing a real production application).

Here is what happens during an acceptance test for the same hypothetical service foo.swiftype.com (registered in DNS as 1.2.3.4):

1. We automatically assign 1.2.3.4 as an alias on a loopback interface.
2. We start a dedicated production Haproxy instance, with a configuration pointing to production backend applications, and bind this dedicated haproxy instance to localhost. (This exactly mirrors what we do in production, where haproxy is always a dedicated localhost service).
3. Our test performs DNS resolution for foo.swiftype.com, receives 10.1.0.x as the IP of the service, connects to a local Nginx instance listening on 10.1.0.x (bound to a loopback interface), and performs a test call.
4. Nginx, receiving a test request, performs whatever operations are defined and forwards it to a local Haproxy backend, which in turn sends the request to a production application instance. When a call is complete, the result is returned by Nginx to the test runner.
5. The test runner performs all defined checks on the response and defines whether the call and response are identified as passing or failing the test.

Here is a diagram illustrating a test call made in an acceptance testing environment:

A test call within the acceptance testing environment

Conclusion

Using our edge web server testing framework for the past few years, we have been able to perform hundreds of high-risk changes in our production edge infrastructure without any significant incidents being caused by the deploying of an untested configuration update. Our testing framework provides us the assurances we need, such that we can make very dramatic changes to our web application edge routing (services that affect every production request) and that we can be confident in our ability to introduce these changes safely.

We highly recommend that every engineering team tasked with building or operating complex edge server configurations adopt some level of testing that allows the team to iterate faster without fear of compromising these critical components.

Business, Management and Leadership

After changing my job at the beginning of 2017 and returning to Swiftype to focus on Technical Operations team leadership, I continued working on improving my skills in this area and read a number of truly awesome books:

• “The Effective Executive: The Definitive Guide to Getting the Right Things Done” by Peter F. Drucker — this classic has immediately become one of my favourite leadership books of all time. There are many useful lessons I learned from it (like the notion that all knowledge workers should consider themselves executives in some sense), but the most powerful was the part on executive time management.
• “Hatching Twitter: A True Story of Money, Power, Friendship, and Betrayal” by Nick Bilton — A truly horrifying “Game of Thrones”-like story behind the early years of Twitter. I didn’t think shit like that actually happened in real life… I guess the book made me grow up a little and realize, that simply doing your best to push your company forward is not always enough. I’d highly recommend this book to anybody working in a fast growing company or thinking about starting a VC-backed business.
• “Shoe Dog: A Memoir by the Creator of NIKE” by Phil Knight — a great story of a great company built by regular people striving for quality results. Heavily reinforces the notion that to be an entrepreneur you need to be a bit crazy and slightly masochistic. Overall, a very fascinating tale of a multi-decade development of a company — a strong contrast with all the modern stories about internet businesses. A must read for people thinking about starting a business.

Health, Medicine and Mortality

I have always been fascinated by the history of medicine, medical stories and the inner workings of the modern medical system. Unfortunately, this year I’ve had to interact with it a lot and that made me seriously consider the fact of our mortality. This has led me upon a quest to learn more about the topics of medicine, mortality and philosophy.

• “When Breath Becomes Air” by Paul Kalanithi — Fantastic memoir! Terrifying, depressing, beautifully described story of a young neurosurgeon, his cancer diagnosis, his battle with the horrible disease and up to the very end of his life. I found the story of Paul very relatable and just like with Atul Gawande’s book I’ve read last year, it brought forth very important questions on how should we deal with our own mortality. Paul gave us a great example of one of the options for how we may choose to spend our last days — the same way we may want to spend our lives: “You can’t reach perfection, but you can believe in an asymptote toward which you are ceaselessly striving”.
• “The Emperor of All Maladies” by Siddhartha Mukherjee — probably the best book on cancer out there (based on my limited research). The author takes us on a long, very interesting and terrifying trip through the dark ages of human war against cancer and explains why after so much time we are still only starting to understand how to deal with it and there is still a long road ahead. Highly recommended to anybody interested in the history of medicine or wants to understand more about the reason behind a malady that kills more than 8 million people each year.
• “Complications: A Surgeon’s Notes on an Imperfect Science” by Atul Gawande — once again, one of my favourite authors manages to explain a hard problem of complications in healthcare and give us a sobering look at the limits and fallibilities of modern medicine.
• Bonus: “On The Shortness Of Life” by Seneca — It is amazing how something written 2000 years ago can have such profound relevance today. I found this short book really inspiring and it has led me to start my road to adapting some of Stoic techniques including mindfulness and meditation.

Miscellaneous

Few more books I found very interesting:

• “Born a Crime: Stories From a South African Childhood” by Trevor Noah — Listened to this book on Audible and absolutely loved it! Hearing Noah’s voice describing his crazy childhood in South Africa mixing fun and absolutely horrifying details of his life there and the struggles he had to endure being a coloured kid under and right after Apartheid.
  Even though it was never as scary as what Noah is describing in his book, I have found in his stories a lot of things I could relate to based on my childhood in late USSR and then in 1990s Ukraine which was going through an economic meltdown with all of the usual attributes like crime and crazy unemployment.
• “I Can’t Make This Up: Life Lessons” by Kevin Hart — I have never been a particular fan of Kevin Hart. Not that I disliked him, just didn’t really follow his career. This book (I absolutely recommend the audiobook version!) ended up being one of the biggest literary surprises ever for me: it is the funniest inspirational read and the most inspiring comic memoir I’ve ever read (or, in this case, listened to). Kevin’s dedication to his craft, his work ethic and perseverance are truly inspiring and his success is absolutely well-earned.
• “Kingpin: How One Hacker Took Over the Billion-Dollar Cybercrime Underground” by Kevin Poulsen — Terrifying read… I’ve never realized how close the early years of my career as a systems administrator and developer took me to the crazy world of underground computer crime that was unfolding around us.
  I’ve spent a few weeks week wondering if doing what Max and other people in this story did is the result of an innate personality trait or just a set of coincidences, a bad hand the life deals a computer specialist, turning them into a criminal. For many people working in this industry, it is always about the craft, the challenge of building systems (just like the bind hack was for Max) and I am not sure there is a point in one’s career when you make a conscious decision to become a criminal. Unfortunately, even after finishing the book I don’t have an answer to this question.
  The book is a fascinating primer on the effects of bad and the need for good security in today’s computerized society and I’d highly recommend it to everybody working with computers on a daily basis.
• “Modern Romance” by Aziz Ansari — very interesting insight into the crazy modern world of dating and romance. Made me really appreciate the fact that I have already found the love of my life and hope will never need to participate in the technology-driven culture today’s singles have to deal with. Really recommend listening to the audiobook, Aziz is very funny even when he’s talking about a serious topic like this.
• “The Year of Living Danishly: My Twelve Months Unearthing the Secrets of the World’s Happiest Country” by Helen Russell — Really liked this book. It offers a glimpse into a society surprisingly different from what many modern North Americans would consider normal. Reading about all kinds of Danish customs, I would think back to the times I grew up in USSR and realize, that modern Danish life is very close to what was promised by the party back then. The only difference — they’ve managed to make it work long term.
  Even though not many of us could or want to relocate to Denmark or to affect our government policies, there is a lot in this book that many of us could apply in our lives: trusting people more, striving for a better work-life balance, exercising more, surrounding ourselves with beautiful things, etc.

I hope you enjoyed this overview of the best books I’ve read in 2017. Let me know you liked it!

Business, Management and Leadership

Considering the new role I’ve started in January 2016 (first-time CTO of a growing startup company), my reading last year was heavily geared towards business, management and leadership topics. Here are my favourite books in this category:

• “The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers” — in my opinion, a must-read book for anybody interested in starting a company or already building one. A treasure trove of great advice for startup founders on building and managing their teams.
• “Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers” — the author explains why so many companies, that find an initial product-market fit, subsequently fail to grow into leaders of their respective markets and often die a slow and painful death. The concept of a chasm and, especially, the idea of the whole product were very powerful for my understanding of what I felt in many companies I worked for — mainstream customers could not use your product unless they are provided with a minimum set of components and services to solve their problem. Very important read for leaders of modern SaaS companies, especially for API/platform enterprises.
• “Turn the Ship Around!: A True Story of Turning Followers into Leaders” — an inspiring story of a navy captain transforming one of the worst-performing crews in the fleet into a perfectly functioning team by pushing control down to individual team members.
• “The Score Takes Care of Itself” — inspiring story of one of the best sport team transformations and the man behind it, legendary coach Bill Walsh.

Miscellaneous

Few more books I found very interesting:

• “The Collapse of Parenting: How We Hurt Our Kids When We Treat Them Like Grown-Ups” — maybe it is just a confirmation bias, but I absolutely loved this book. The author focuses on a few serious problems in today’s parenting and the resulting decline in the achievement and psychological health of American children. He finally managed put into words something that was bothering me for 10 years since moving to Canada. Now that I became a parent and would have to raise a child in this environment, I was glad to hear that I wasn’t crazy not to agree with the approach that is being pushed on modern parents by American society.
• “Being Mortal: Medicine and What Matters in the End” — one of my favourite authors, Atul Gawande, explores the current state of end of life care in the USA, Canada, and Western Europe. Terrifying at first, the book makes you consider your own mortality and think about the choices you are bound to make eventually for yourself and, potentially, for your close family members.
• “Sapiens: A Brief History of Humankind” — a captivating overview of our history as human species: from 70,000 years ago until the 20th century: how we evolved, how we affected other species on the planet and how did we end up where we are today. A long, but very interesting read!
• “The Road To Sparta: Retracing the Ancient Battle and Epic Run that Inspired the World’s Greatest Foot Race” — fascinating story of Dean Karnazes (one of the most famous ultra-distance runners in the world) and his exploration of the legend of the Marathon. Highly recommended to anybody interested in running.
• “Catastrophic Care: How American Health Care Killed My Father — and How We Can Fix It” — very detailed overview of what is broken in US healthcare today. Even if you don’t have anything to do with US healthcare market, the book is a great collection of stories about side-effects of what initially looked like great ideas, but ended up screwing the system even further.

Fiction

I was always a huge fan of sci-fi fiction and this past year I have discovered a few real gems that ended up on my all time favourite list:

• “Remembrance of Earth’s Past (aka The Three Body Problem)” series by a Chinese author Liu Cixin — huge universe, highly-detailed and powerful characters, timeline spanning centuries — you can find all of it here. But on top of the standard components of a good space opera, there is this previously unknown to me layer of Chinese culture, language, philosophy.
  This trilogy has become an instant classic for me and is in the top-10 of my all time favourites next to Asimov’s “Foundation” and Peter F. Hamilton’s “Void”.
• Everything from Niel Gainman! Up until this year when I got exposed to his writing, I’ve never realized how much pleasure one could get from reading prose. I’m not sure how he does it, but if he were to publish a book of obituaries or classifieds, I’d be willing to read that too — I enjoyed his English so much! Favourite books so far: “The Graveyard Book” and “The Ocean at the End of the Lane”.

I hope you enjoyed this overview of the best books I’ve read in 2016. Let me know you liked it!

For the next few years a lot of new functionality we needed has been added to the gem, making it more complex and immensely more powerful. I’ve enjoyed working on it, developing those features, contributing to the community. But then I left Scribd, stopped being a user of DbCharmer and the situation drastically changed. For quite some time (years) I would keep fighting to make the code work with newer and newer versions of Rails, struggling to wrap my head around more and more (sometimes useless) abstractions Rails Core team decided to throw into ActiveRecord.

Finally, in the last 2 years (while trying to make DbCharmer compatible with Rails 4.0) it has become more and more apparent, that I simply do not want to do this anymore. I do not need DbCharmer to support Rails 4.0+, while it is very clear that many users need it and constant nagging in the issues and the mailing list, asking for updates generated a lot of anxiety for me, anxiety I couldn’t do much about (the worst kind). As the result, since I simply do not see any good reasons to keep fighting this uphill battle (and developing stuff like this for ActiveRecord IS a constant battle!) I officially give up.

After some long and painful consideration I’ve decided to officially suspend the project. Here is what the suspension means in this case:

• I will stop making any changes in DbCharmer code
• Pull Requests and Issues functionality on the project repository will be disabled (I will dump the issues somewhere for future reference, but no new messages could be added)
• There will be a huge message in the project README explaining that no Rails versions beyond the latest 3.2.x are supported and there are no plans to do any development to make the code work with Rails 4.0+
• Project mailing list will be disabled
• Project website will be moved to a github domain (with the same message explaining the project status)

I’m really sorry if any of the users of the project still had some hopes regarding the Rails4 branch and potential upgrade to the newer Rails versions, but 3.2.x will be the last version officially supported by the project.

Now, here are my answers to the questions that people have asked me about this decision (I’ve talked to a few of the largest users of the project already):

Are you going to kill the repo? – No, the repository (and even the rails4 branch) are going to stay intact. I’m just going to clearly mark it as inactive to make sure people do not try to use it with new Rails versions or expect the project to support those in the future.

What about Rubygems? – All rubygems versions released up until this point will stay active and accessible as long as rubygems is alive. Though no updates will be provided for any Rails versions beyond 3.2.x.

Why not crowdfund the development? – This is a really tough issue I struggled with for a long time. The problem here is that for many years now I had been very fortunate to be in a position when I’m not motivated by the money anymore. So crowdfunding the development would only increase my anxiety 10x, while not really changing the situation on the motivation side. That’s the opposite of what I need at this point.

What if you need DbCharmer functionality on your current or some next job? – This is why I’m not deleting the repo, gems, etc and calling it a suspension and not a closure There is a chance, that one day I will end up in a situation, where I will really need all those wonderful features I’ve enjoyed with DbCharmer for years. And I’m pretty sure, that unless there will be another project available on the market, I will try to revive the project (or build something new upon the most important pieces of DbCharmer codebase). But nobody knows what will happen, so for now the project is suspended.

Can I help with the development? Maybe send in a patch? – Another very tough issue. Accepting patches still requires a lot of time and dedication to review, understand and test them. And that is not something I want to do at this point. The only real way to resume the development of the project at this point would be to transfer the ownership to somebody else. But unless someone creates a fork, shows a true dedication to the project (making sure all the incoming changes are 100% test-covered and battle-tested, etc), I’m not ready to do that. If you have some ideas on this matter, you could ping me any time.

Recently, one of the candidates asked me to share my lists with him and I thought this information could be valuable to other people so I have decided to share it here on my blog.

Must-Read Books List

First of all, I would like to share a list of books I believe every professional in our field should read at some point in their life. You may notice that many of these books are not too technical or are not really related to the pure systems administration part of a techops job. I still think those are very important because technical operations work on senior levels involves much more than just making sure things work as expected. A lot of it involves time management, crisis management and many other topics that are equally important for a professional in this field.

So, here is the list (with not particular ordering, grouped by topics):

Systems and Networks Administration

• Advanced Programming in the UNIX Environment
  by W. Richard Stevens and Stephen A. Rago
• High Performance MySQL: Optimization, Backups, and Replication
  by Baron Schwartz, Peter Zaitsev and Vadim Tkachenko
• UNIX and Linux System Administration Handbook
  by Evi Nemeth, Garth Snyder, Trent R. Hein and Ben Whaley

Technical Operations, Architecture, Scalability

• Web Operations: Keeping the Data On Time
  by John Allspaw and Jesse Robbins
• Release It!: Design and Deploy Production-Ready Software
  by Michael T. Nygard
• Scalable Internet Architectures
  by Theo Schlossnagle
• The Art of Capacity Planning: Scaling Web Resources
  by John Allspaw

Project, Release and Time Management

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win
  by Gene Kim, Kevin Behr and George Spafford
• Kanban: Successful Evolutionary Change for Your Technology Business
  by David J. Anderson
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
  by Jez Humble and David Farley
• The Power of Full Engagement: Managing Energy, Not Time, is the Key to High Performance and Personal Renewal
  by Jim Loehr and Tony Schwartz

Other

• Failure Is Not an Option: Mission Control from Mercury to Apollo 13 and Beyond
  by Gene Kranz
• Team Geek: A Software Developer’s Guide to Working Well with Others
  by Brian W. Fitzpatrick and Ben Collins-Sussman
• Antifragile: Things That Gain from Disorder
  by Nassim Nicholas Taleb
• The Field Guide to Understanding Human Error
  by Sidney Dekker
• Behind Human Error
  by David D. Woods, Sidney Dekker, Richard Cook, Leila Johannesen

For more information on interesting books for technical operations engineers, you can check out the following book lists on GoodReads:

• DevOps Reading List
• Surge 2013 Reading List

Interesting Conferences

Conferences, in my opinion, are an essential part in professional development of any engineer. Here is a list of conferences that could be useful for techops engineers:

• Surge Conference – in my opinion, this is definitely one of the best conferences dedicated to building and maintaining large web architectures. If I were to choose one conference a year to go to, it would definitely be Surge. Videos from previous years are freely available online: 2010, 2011, 2012. 2013 videos should be available soon as well.
• Oreilly’s Velocity Conference – biggest and, probably, the oldest web operations and web performance event. In my opinion, recently it became too focused on web frontend performance, though it is still a really interesting event. Complete video compilations from the conference are available for sale: 2011, 2012, 2013.
• Monitorama Conference – pretty new, but already very popular conference with interesting content for everyone interested in monitoring (which most ops engineers are). Sides and videos from the first ever Monitorama conference in 2013 are available online.
• Percona Live Conference – really awesome event for anybody who has MySQL in their stack. Huge multi-track event with talks from the best and brightest people in MySQL community. Slides and keynote videos from 2013 event are available online.
• DevOps Days – small events happening all around the world and becoming more and more popular. The major topic of these conferences is the DevOps movement, related team/project management practices, etc. Videos and slides from some of the events are available online.

Even if you do not have time to watch any of those conference videos, I think every operations engineer out there would really enjoy 2011 Surge Conference closing plenary session video where Theo Schlossnagle (one of my personal heroes in IT community) described a typical debugging session many of us go through every once in a while:

Interesting Web Resources

And last, but certainly not least, I would like to share a list of web resources I like to follow to stay up to date on the most recent news and fresh ideas within the web operations community and related areas:

Leading Industry Sites and Blogs

• MySQL Performance Blog from Percona – one of the best resources on MySQL performance
• High Scalability – awesome resource with a lot of great articles on scalability, performance and design of large scale systems
• Kitchen Soap – Blog by John Alspaw (another of my personal heroes in IT field)
• DevOps Community Planet – feed/news aggregator for the DevOps community
• DevOps Community on Reddit – not too active, but still a useful resource for getting interesting news
• Agile Sysadmin – Blog of Stephen Nelson-Smith
• obfuscurity – Blog by Jason Dixon, maintainer of Graphite, author of Descartes, Tasseo and other useful tools for metrics collection and displaying
• The Agile Admin – Many interesting thoughts on agile web operations and devops
• Operation Bootstrap – Blog of Aaron Nichols talking about many different aspects of working in operations

Engineering Blogs of Large Web Companies

• Code as Craft – Etsy Development and Operations blog
• Twitter Engineering Blog
• Netflix Tech Blog
• LinkedIn Engineering Blog

Podcasts

• Changelog – member-supported podcast on 5by5 network talking about interesting open source projects
• Food Fight – bi-weekly podcast for Chef community
• DevOps Cafe – interviews with interesting members of DevOps community
• The Ship Show – twice-monthly podcast, featuring discussion on everything from build engineering to DevOps to release management, plus interviews, new tools and techniques, and reviews

And this is it! I hope these lists would be useful for young engineers going into the technical operations and for people who already work in this space. I am going to try to regularly update this post in the future to make sure it stays relevant for a long time.

Since I have joined Swiftype, I have been mainly focused on improving the infrastructure through better automation and monitoring, and worked on our backend code. Now I am looking for a few good operations engineers to join my team to work on a few key projects like building a new multi-datacenter infrastructure, creating a new data storage for our documents data, improving high-availability of our core services and much more.

To help us improve our infrastructure we are looking both for senior operations engineers and for more junior techops people that we could help grow and develop within the company. Both positions could be either remote or we could assist you with relocation to San Francisco if you want to work in our office.

If you are interested, you can take a look at an old, but still pretty relevant post I wrote many years ago on what I believe an ops candidate should know. And, of course, if you have any questions regarding these positions in Swiftype, please email me at kovyrin@swiftype.com or use any other means for contacting me and I will try to get back to you as soon as possible. If you know someone who may be a great fit for these positions, please let them know!