Stay tuned – I really hope to finally get a chance to do more blogging this year.

For more information on my background, please check my Github profile, my linkedin profile or the resume section on this blog. If you need to contact me, feel free to use any channels listed on the contacts page.

Update: After a few initial interviews I’d like to update this post with a bit more details on what I’m looking for in the new position.

Though even more I’m interested in being able to make strategic technical decisions for an operations team and apply my experience and knowledge for solving challenging tasks with a dedicated team of ops engineers. This could be anything from a tech ops team lead role (in a medium/large companies) to a director of technical operations (in a small-to-medium sized startups).

Update: Ok, I’ve found a new job – I work for Swiftype now!

About 2 months ago I’ve joined LivingSocial technical operations team and one of my first tasks there was to figure out a way to make our MTAs perform better and deliver faster. We use a really great product called Momentum MTA (former Ecelerity) and it is really fast, but it is always good to be able to squeeze as much performance as possible so I’ve started looking for a ways to make our system faster.

While working on it I’ve created a set of scripts to integrate Momentum with Graphite for all kinds of crazy stats graphing, those scripts will be opensourced soon, but for now I’ve decided to share a few tips about performance-related changes we’ve made to improve our performance at least 2x:

1. Use EXT2 Filesystem for the spool storage – After a lot of benchmarking we’ve noticed that amounts of I/O we’ve been doing was way too high compared to our throughput. Some investigation showed that EXT3 filesystem we were using for the spool partition had way too high metadata update overhead because of the fact that the spool storage uses a lot of really small files. Switching to EXT2 helped us gain at least 50-75% additional performance. Additional performance gain was caused by turning on noatime option for our spool.

   There are some sources that claim using XFS for spool directories is a better option, but we’ve decided to stick with EXT2 for now.

2. Do not use %h{X} macro in your custom logs – Custom logs is an awesome feature of momentum and we use it to log our bounces along with some information from mail headers. Unfortunately the most straighforward thing to do (using %h{X} macro) was not the best option for I/O loaded servers because every time Momentum needs to log a bounce it needs to swap message body in from the disk and parse it to get you the header value.

   To solve this issue we’ve created a Sieve+ policy script that would extract the headers we need from a message during initial spooling phase (when the message is still in memory) and put those values to the message metadata. This way when we need to log those values we wouldn’t need to swap message body in from the disk. Here is the Sieve script to extract header value:

   1 2 3 4 5 6

   require [ "ec_header_get", "vctx_mess_set", "ec_log" ]; # Extract x-ls-send-id header to LsSendId context variable # (later used in deliver log) ($send_id) = ec_header_get "x-ls-send-id"; vctx_mess_set "LsSendId" $send_id;

   After this we could use it in a custom logger like this:

   1 2 3 4 5 6

   custom_logger "custom_logger1" {   delivery_logfile = "cluster:///var/log/ecelerity/ls-delivery_log.cluster=>master"   delivery_format = "%t@%BI@%i@%CI@D@%r@%R@%m@%M@%H@%p@%g@%b@%vctx_mess{LsSendId}"   delivery_log_mode = 0664 }

3. Give more RAM to Momentum – When Momentum receives a message, it stores it to the disk (as required by SMTP standard) and then tries to deliver the copy it has in memory, if delivery succeeds, on-disk copy is unlinked. The problem with a really have outbound traffic load is that momentum needs to keep tons of emails in memory, but by default it could only hold 250 messages. With a load of 250-500 messages a second this is just too small.

   To change this limit we’ve increased Max_Resident_Active_Queue parameter and changed it to 1000000 (of course we made sure have enough RAM to hold that many messages if needed) and Max_Resident_Messages to 0 (which means unlimited). This allows Momentum keep as many messages resident as possible and reduce the load caused by swap-in operations required for re-delivery attempts, etc.

4. Choose a proper size for your I/O-related thread pools – in default Momentum config they set SwapIn and SwapOut thread pool sizes to 20. Under really high load even on our 4xSAS15k RAID10 this tends to be too high value. We’ve switched those pools to 8 threads each and it helped to reduce I/O contention and overall I/O throughput.
5. Disable adaptive delivery if you do not need it (especially critical in 3.4 with cluster-aware adaptive delivery) – in our case it allowed us to increase single server throughput from 1.5M to 2M messages/hour for servers with warmed-up bindings that do not need adaptive warmup and other fancy features. The reason behind this difference is in the fact that to be able to do adaptive delivery magic, your MTA needs to spend considerable amounts of resources on all those decisions that make adaptive really valuable for warming-up new IPs (adaptive backoff on ISP blocks, moving mail between nodes based on bindings suppression information). After you disable Adaptive, make sure yo restart your ecelerity process and check that module is not loaded (using module list console command).
6. Remove Replicate clauses from your configs – if you do not use data replication features in momentum (inbound_cidr, metrics, etc), disabling all replicate stanzas in your config files could give you a great performance boost because apparently this replication stuff takes a lot of resources and greatly reduces throughput (though it could be useful in some cases).

As a summary, I’d like to note, that as with any optimizations, before tuning your system it really helps to set up as much monitoring for your MTA servers as possible: cacti graphs, graphite, ganglia or something else – does not matter. Just make sure you see all the aspects of your system performance and understand what is going on with your system before changing any performance-related settings.

Ruby core team released new 1.8.7-p357 version with the problem fixed. JRuby development team came out with the new 1.6.5.1 release. Unfortunately 2 days after the release there is still no information from Ruby Enterprise Edition team on what to do with all the ree deployments.

So, since there is no patch for Ruby Enterprise Edition 2011.03 to prevent an attack, I’ve quickly ported ruby core patch to be used with the latest ree release. Here it is:

You can view it or download it from github. Disclaimer: This is provided as is, no guarantees are provided, etc.

For more information about the issue, here is a video of the official presentation:

Update #1: Updated the patch to apply cleanly on REE (w/o a need to switch directory to source).
Update #2: The patch is currently deployed in production at Scribd and is being tested and deployed on LivingSocial servers.
Update #3: Phusion released 2011.12 release of REE now, so this patch is not needed anymore.

New features in this release:

• Rails 3.0 support. We’ve worked really hard to bring all the features we supported in Rails 2.X to the new version of Rails and now I’m proud that we’ve implemented them all and the implementation looks much cleaner and more universal (all kinds of relations in rails 3 work in exactly the same way and we do not need to implement connection switching for all kinds of weird corner-cases in ActiveRecord).
• Forced Slave Reads functionality. Now we could have models with slaves that are not used by default, but could be turned on globally (per-controller, per-action or in a block). This is a new feature that brings our master/slave routing capabilities to a really new level – we could now use it for a really mission-critical models on demand and not be afraid of breaking major functionality of our applications by switching them to slave reads.
• Lots of changes were made in the structure of our code and tests to make sure it would be much easier for new developers to understand DbCharmer internals and make changes in its code.

Along with the new release we’ve got a brand new web site. You can find much better, cleaner and, most importantly, correct documentation for the library on the web site. We’ll be adding more examples, will try to add more in-depth explanation of our core functions, etc.

If you have any questions about the release, feel free to ask them in our new mailing list: DbCharmer Users Group.

For more updates on our releases, you can follow @DbCharmer on Twitter.

It was spring 2010 when we decided that even though Softlayer‘s server provisioning system is really great and it takes only a few hours to get a new server when we need it, it is still too long sometimes. We wanted to be able to scale up when needed and do it faster. It was especially critical because we were working hard on bringing up Facebook integration to our site and that project could have dramatically changed our application servers cloud capacity requirements.

What buzzword comes to your mind when we talk about scaling up really fast, sometimes within minutes, not hours or days? Exactly – cloud computing! So, after some initial testing and playing around with Softlayer’s (really young back then) cloud solution called CloudLayer and talking to our account manager we’ve decided to switch our application from a bunch of huge and at the time pretty expensive 24-core monster servers to a cluster of 8-core cloud instances. To give you some perspective: we had ~250 cores at the start of the project and at the end of 2010 we’d have more then 100 instances – we weren’t a small client with a few instances).

For those who are not familiar with Softlayer cloud: they sell you “dedicated” cores and memory, which is supposed to give you an awesome performance characteristics comparing to shared clouds like EC2.

Long story short, after a month of work on the project we had our application running on the cloud and were able to scale it up and down pretty fast if needed. And since the cloud was based on faster cpu and faster memory machines, we even saw improved performance of single-threaded requests processing (avg. response time dropped by ~30% as far as I remember). We were one happy operations team…

But then the problems started. Pretty painful and weird problems. The problems all looked exactly the same – some day (or night) some of our cloud instances would start blocking on I/O operations forever or for really long periods of time. Then some time after that our filesystems would switch to read-only mode because of write errors (timeouts) on their storage.

Just to make it clear: our application does not log much to local disks and almost does not read anything. But sometimes we still need to read a config file or write some logging information, and syslogd was still there and sometimes needed to write to disk. So, many times a week we would wake up, see an instance dead (all critical processes locked in D-state), call Softlayer, and spend hours (literally) to bring it back up. Sometimes it’d happen to one server, sometimes to ten of them. First time in my life I was afraid to go to bed because I was almost certain a monitoring SMS would wake me up within a few hours. This was probably one of the most stressful periods of my professional life to date.

What was making the situation even worse, nobody on Softlayer’s side seemed to care about the trend we all were seeing pretty clearly – their cloud was seriously broken. Every ticket about a dead instance was processed as if it was the first one with such problem ever. Support people would poke around an instance for some time, then try to restart it, etc. After some time they’d give up and escalate it to so called InfoSys team. Which would take its time (usually hours) to finally respond with a template answer telling us there was a problem with their SAN and we were the only customers experiencing the problem and that they are working with their SAN vendor to fix this problem. We’ve got dozens of tickets and they all looked exactly the same (compared word by word) even though they were sent weeks apart.

After a month (I believe) of this hell in our operations team’s life and being close to a nervous breakdown I’ve contacted our dedicated account manager and told him we were going to start leaving Softlayer for some other provider (not only the cloud but all servers) if they would not fix the problem within one week. And this is where things started moving really fast: I’ve received dozens of calls from all kinds of managers, was called-in to a few meetings with a senior manager responsible for the cloud, etc and within a week they’ve found a solution. They’d setup local disks on their hosts systems and move our filesystems to local disks instead of putting them on their SAN. This was supposed to “fix” the problem until they figured it out completely and, as expected, it did help us. The problems went away and if they were still happening again – that’d be some instance they forgot to switch to local storage.

Time was passing by and for a few months we did not have any serious issues with the cloud. Then we’ve started growing. Being pretty sure the problem had been solved by that time we didn’t actually realize that new cloud instances were created on SAN again. And really soon we’ve got bad news – their problem was still there. Any issues happening with our instances were processed within the same template:

• an instance breaks;
• we receive a notification and file a ticket;
• Softlayer support guy spends up to an hour trying to figure out what’s happening;
• the ticket is escalated to the InfoSys team;
• we wait up to 1 day (yes, two cases out of dozens we had with them it took more than a day!) for an answer from this systems team and the problem resolution.

This InfoSys team from my point of view was a really interesting case of a poorly integrated third party service within a huge company (this is my vision of the problem, I do not know how this happened in reality). Every time we’d have a problem that needed their attention, support guy would escalate the problem to the systems team and this is it – nothing he could do and the only thing we’d hear hours after the escalation was “we can’t contact systems team to get an update, we should wait”. Nobody from this team ever contacted us directly, it was impossible to reach out to any of them, they never seemed to care about those downtimes we had – they’d follow the same “fix one instance at a time” patterns again and again and whole Softlayer support department would not be able to get a few words of an update from them.

At this point we’ve started thinking about moving away from the cloud, but it’d involve a month or two of work and having a small operations team with lots of concurrent tasks limited your ability to allocate so many resources to such a project so we didn’t do that back then.

After a few more months of work with their cloud and having intermittent downtimes (rare, but still painful) at the beginning of 2011 two things happened:

First, we’ve got Softlayer guys to admit that their “dedicated” cores are actually Intel Hyper-Threading threads (1/2 of a cpu core). This made us realize that the cloud wasn’t actually cheap comparing to the new hardware available on the market.

Second, our account manager and a few guys from the systems team in clear text asked us to leave their cloud because they could not support our I/O requirements anymore (remember I told you before – we barely used their storage).

This was enough and we’ve finally made the decision to give up and go back to real hardware only we control and manage. It took us more than a month of work, but we think we’ve got pretty good system built as the replacement for Softlayer cloud based solution.

Now, the reason I’ve decided to finally write this blog post is funny: I was going to write it a few times and didn’t have a chance to do it. But today (April 30, 2011) one of 5 cloud instances we have left in Softlayer cloud has died. We use those instances for all kinds of experimental “servers”, etc and it wasn’t uber-critical downtime. So, this instance died and so far (4.5 hours after its death) the problems looks really similar to what we’ve experienced many-many times in the last year and again – they follow the same template: support guys have spent an hour messing with the instance, then they’ve transferred the ticket to their InfoSys team and 3 hours later there weren’t any updates and they would not be able to reach that team to get anything from them. Update: It took them 4.5 hours to recover the instance and the only explanation we’ve got was “there were some problems with the host system”.

I really hope every single operations team in the world considering using Cloudlayer as their solution for some problems would google first, see my experience and would make their own conclusions. These are the facts and my own experience and it is up to you guys to decide how to interpret them and apply them to your own cases.

Scribd has an agile, startup culture and an unusually close working relationship between engineering and ops. You’ll regularly find cross-over work at Scribd, with ops people writing application-layer code and engineers figuring out operations-level problems. We think we’re able to make that work because of the uniquely talented people we have on the team.

To allow us to keep scaling, we’re now looking to add a strong, experienced operations guru to the team. As a member of Scribd operations, you’ll have tremendous ownership and responsibility for one of the web’s most popular applications. Because Scribd is a startup, you will wear many hats and have broader responsibility than you would at a larger company.

If you read this blog, you should already have a good sense of the kind of work you’ll be doing on this position.

The Ideal Profile

You are an experienced operations professional and have run ops at at least one large-scale website. You have comprehensive knowledge of a broad variety of system tools, from MySQL and Nginx to Squid and Memcached. You should also have strong software development skills and be well-versed in major programming languages. You should be strongly motivated, a creative solution finder, and ready to jump into the thorniest technical problems whenever necessary.

Responsibilities

• Develop and maintain all aspects of Scribd’s operations infrastructure, including system monitoring, backups, server configuration, databases, and caching systems
• Collaborate with engineering to create next generation infrastructure to support changing requirements
• Predict scaling problems before they occur and work with engineering to prevent them
• Write and debug application level ruby code
• Participate in an on-call rotation
• Quickly diagnose server problems and employ preventive measures to maintain high availability servers

Qualifications

• Bachelors degree in CS or equivalent experience
• 3-5 years of professional experience in site operations
• Strong software engineering skills, including knowledge of major programming languages
• Strong database skills, preferably with MySQL, and overall linux knowledge
• Experience with most of the following technologies: MySQL, Nginx, Ruby, Memcached, Squid, git, Solr, HBase, Postfix
• Proven ability to quickly learn and implement unfamiliar technologies
• Strong desire to work hard at a rapidly growing company

Location: You are preferably located near San Francisco, CA. Relocation assistance is designed on a per-case basis. In short, we’ll be creative to get you here.

Contact: Please send your email cover letter and resume with the subject “Your name – Senior Site Operations Engineer – via Kovyrin.net” to jobs@scribd.com or contact me directly using any of my contacts. All communication and correspondence is held in the strictest confidence to ensure that you can connect and learn more without exposure.

First of all, here is why you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your users perform on your site. If your content was on your web server, you could have used simple controlled downloads functionality built-in to nginx out of the box. But the problem is that your content is remote.

Here is what we do to solve this problem.

First, we create a special location on our nginx server. This location will be used as a proxy for all our accelerated file downloads:

After adding this location to our nginx config we could start sending responses with headers like the following:

Here is an example code you could use in a Rails application to use our internal redirect location:

As you can see, nginx is really powerful tool and when you turn your creativity on you can make it even more powerful. Stay tuned for more Nginx-Fu posts.

• Her web site (of course she is the one who designed it)
• Her portfolio
• Her resume
• Her LinkedIn profile

We’re physically located in Toronto, Canada, but she has a great experience of working remotely too. So, if you need a web designer or a junior web designer, feel free to contact Tanya.

It is a complex problem that usually consists of two smaller ones: individual cache elements invalidation (you need to keep an eye on your data changes and invalidate cached pages when related data changes) and full cache purges (sometimes your site layout or page templates change and you need to purge all the cached pages to make sure users will get new visual elements of layout changes). In this post I’d like to look at a few techniques we use at Scribd to solve cache invalidation problems.

Since we use haproxy to balance our traffic in the cluster it is hard to predict where should we send a purge request. So we fan those out to all cache servers.

To make sure cache purging won’t slow the site down, especially considering we need to do more that just a simple cache purge (submit documents to search indexes, etc, etc), we just spool a “document changed” request to a queue and then have a set of asynchronous processes that do all the work in background.

Next, The Hard Problem – handling full cache purges w/o killing our backend servers with 5x-10x traffic (our normal hit ratio is ~90-95%).

We’ve spent a lot of time thinking about this problem and the first idea we came up with was to have a loop process somewhere that would iterate all documents we have cached and purge them one by one… but that does not seem to be a practical solution when you have tens of millions documents (and few page versions per document) and obviously the solution would not scale with constantly growing documents corpus.

So we kept brainstorming and finally got one idea that works just perfectly for us: what if we’d be able to take our traffic and define a function f(t) that would return a percentage of the traffic that should be purged at any moment in time. So we did it – we’ve implemented a nginx module that would version our cache by assigning every cached page a revision (using a custom HTTP-headers + Vary-caching) and would be able to slowly migrate the cache from one revision to another over a pre-defined period of time.

Here is an overview of the requests/responses flow in our web/cache/application cluster:

X-Cache-Revision value selection algorithm is the following:

Having this kind of logic in place allows us to do so called “slow” cache purges that could take any time from a few minutes (that still helps to reduce the load spike generated by the hottest content) up to many hours (this is what we normally use) or days (never used this option, but it is definitely possible).

Here is an example 100% cache purge over an 8 hour interval:

1. Daily hit ratio graph:
   
2. Weekly hit ratio graph:
   

As you can see, during those slow purges our cached pages would be slowly updated without putting too much pressure on the backend. Cache hit ratio would slowly degrade and then slowly get back to its normal levels, but with our normal (6-8 hours) purges hit ratio never gets lower that 65-70% which makes it possible for us to save huge amounts of money on not having 90% spare capacity just for the cache purge load surges (we used to have lots of spare application cluster capacity before introducing this approach).

Update (June 5th, 2010): Request/response flow and cache revision selection algorithm added.