Originally loops plugin was created to make our (Scribd.com) own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin itself.

The major idea behind this small project was to create a deadly simple and yet robust framework to be able to run some tasks in background and do not think about spawning many workers, restarting them when they die, etc. So, if you need to be able to run either one or many copies of your worker or you do not want to think about re-spawning dead workers and do not want to spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your loop as a separate process controlled by monit/god/etc), then I’d recommend you to try this framework — you’ll like it.

For more information, visit the project site and, of course, read the sources

So, first of all I would like to explain what kind of interviews I really hate Ever since I was thinking of myself as of a developer (many years ago) and was going to “software developer position” interviews I really hated questions like “What is the name and possible values of the third parameter of the function some_freakin_weird_func() from some_weird.h” or “How to declare a virtual destructor and when it could be useful?”… All my life I had pretty practical thinking and never bothered to learn APIs or some really deep language concepts that are useful in 1% of the development time in really weird edge case situations. I always thought that a Real Engineer should know where to find an answer and should be able to find a way to implement something without knowing all possible ways to do it - engineering mind should be flexible!

So, if I’d interview a developer for our company, I’d never ask them this weird kind of theoretical questions because honestly speaking web development is not a rocket science and in most of the cases you don’t need to have a CS Master degree to be a good developer.

So, let’s get closer to our discussion in Scribd. During this discussion we were talking about our standard hiring practices, about questions we used to ask our candidates and such. As the result I decided to write a set of questions I’d really like an ideal candidate to be able to answer. Many of them are not rails specific, but rather high-load and high-scale specific. Let me share some of those questions here.

1. You have a generic web application (does not matter if it is rails or not) with a set of layers and parties involved in each request:
   • Database
   • Application
   • Internet
   • User

   What security problems could occur on each level and what action should developer take to prevent those problems from happening? For example take a login action in your application and explain possible problems.

2. You have a table with 100 millions of records, you need to make some changes in each records and it is obvious that you won’t be able to do that using some SQL queries, so you need to do in some application-level code (one time use script). Your database is MySQL.
   • Where would you put the code in the application: model, module with a bunch of methods, migration, some dedicated script?
   • How would you process all those records: Model.find(:all).each {}, a loop with different offsets and a limit, a loop walking through primary key values with some step, something else?
   • What would you choose for this task and why: Model.find or Model.find_by_sql? What is the major difference between those two? When would you definitely use find_by_sql?
   • What would you use: update_attribute, attribute assignments + saves, Model.connection.execute or something else to update the table? When and why would you use each of those?
3. You have a rails action that accepts uploads and then transfers them to a pretty slow storage (S3, some slow nfs, something else). If you have a lots of uploads, pretty soon many of your mongrels (if not all) will be used for uploads and your application would become unusable. What options do you see to solve this problem?
4. You have a huge table in mysql, you need to select a few randomly selected records (and this is not a one-time action, it’ll be used pretty often, like on the home page), how’d you do that?
5. You have a document model with the following fields: id, user_id, title, body, created_at, views_count, download_count. There are 10000000 documents. You need to implement a download counters on your site so when one downloads a document, you’d increase the download_counter field. How would you implement counter management?
6. How’d you write an SQL query to find all users in a system that have no docs (standard User has_many docs relationship)?
7. Caching:
   • What kinds of caching techniques do you know?
   • What caching options does Rails provide, when would you use those?
   • What caching options does HTTP protocol provide?
   • You have a pretty popular action on your site which uses some heavy SQL query which should be cached. But the problem is that sometimes when the cache expires and too many users hit the action, all those SQL queries will start hammering the database trying to refresh your caches… That causes huge problems. How’d you try to solve the problem?

As you can see, these questions aren’t asking you about REST or new cool Rails buzzwords (ActiveResource, named_scope, etc, etc) because IMHO - you can learn those in a week of skimming through any rails-related mailing list. My questions are aimed at the most often happening problems in high-traffic startups where not all common practices could be used and you need to think before writing some code that’d be executed millions of times per day.

So, if you’ve read the questions and you think you know (or is able to find in Google/books/etc) answers and you think you know Rails I think you should contact us because we’re hiring. At this moment we need a few really great Ruby-developers that are not afraid of working on high-load web applications and who isn’t just a buzzword junkies, but a developers that could use their tools effectively.

For more information and more formal position information, please check out our jobs page or just email me with any questions related to these positions.

P. S. If you’re really good, we could work with you remotely w/o relocation to San Francisco (we have some team members in Canada and Ukraine now).

So, long story short, after a lots of docs and source code reading we’ve found really interesting thing. There is a “feature” added in the recent ActiveMQ release (5.X) called “flow control“, which is used by ActiveMQ core to slow down too fast producers when consumers lag or just aren’t as fast as ActiveMQ wants them to be. This results in unpredictable producers stalls and all kinds of weird problems.

If you’re experiencing the same problems, you could try to add the following to your activemq.xml file to make it stop using flow control and let your producers spool data to the queue as fast as it is possible (that’s exactly what I’d expect from a queue server):

When this is added, all your queues (if they take more than 64Mb of RAM) will start persisting their messages to the disk and freeing your RAM to consume more messages w/o any slowdowns.

Another useful feature we use in our producer is Async Sends which basically means that your producers won’t require any acknowledgments from ActiveMQ after it pushes your messages to a queue. This makes push() operations MUCH faster and your producers’ throughput will become tremendously higher. If you use Java as your producer language, you could use the following code to make your connection async:

After we’ve made the changes explained above, our queue server and producers became rock stable and blazing fast so there is no point to look for another solution because looks like I was right - ActiveMQ is an amazing piece of software - you just need to learn it before using.

This is it for today, if you have any questions, feel free to ask them in the comments and I’d be glad to answer.

After a short research I’ve found a pretty simple to set up and easy to use approach to such a connection problems I’d like to describe here.

First, you’ll need someone (or you can do it before leaving home) to start an ssh daemon on port 80 on one of your servers. I use one of my Slicehub slices for this to permanently have an ability to use it. You can do it like this (if it is a temporary solution):

Notice: this `which sshd` was used because on some OSes sshd does not want to start w/o an absolute path to its binary.

If you’d like to have it permanent, just add one line to your /etc/ssh/sshd_config file and restart your ssh server:

OK, this step is finished and you have sshd listening on port 80. Now, let’s imagine you go somewhere and need to use ssh (or ICQ/GTalk/Jabber/MSN/AIM, or torrents) and some weird admin banned all tcp ports but port 80. Here’s what you’ll need to do:

1. You open terminal on your machine (most likely it’ll be your laptop)
2. You run the following command:

   1	$ ssh -D 1080 -p 80 -v user@your-host.com "sleep 1000000"

3. Use 127.0.0.1 (port 1080) as your local socks4/5 proxy server (no authorization) in any software you want

When you have theses steps are finished, you can go to any place you want and wherever you have an access to web servers, you’ll have an access to anything you need (tested on: ssh, safari, adium, vuze torrent client, rubygems, macports).

P. S. On my macbook to make this thing even more useful I’ve installed tsocks port (non-mac users, check it out here).

This is why when I received an email from Packt company asking if I’d like to read and review their book on Lighttpd I decided to give it a shot (I usually do not review any books because I do not always have enough time to read a book thoroughly to be able to write a review). So, here are my impressions from this book.

First, when I received the book, I was in doubt: how such a small book could cover so flexible and multi-purpose piece of software like Lighttpd? But after reading a few first chapters I understood that the book is written as any professional book should be written - no weird long chapters explaining what is Internet, web server, HTTP or HTML. From the first pages you dive to the world of Lighttpd starting from installation and going through the most popular and useful configuration options to really interesting virtual hosting configurations.

Pretty interesting security-related chapters explain typical process of obtaining and installing SSL certificates. This always was the most painful process for me in almost any web setup I worked on, don’t know why though, this process seemed kind of messy for me, but I think those chapters could help novice web server administrators understand the process of securing a web server.

For many new lighttpd users the most interesting chapters should be ones that explain typical problems and use cases of using apache with or instead of apache servers and describe how to set up lighttpd to be used with the most popular backend web software (Ruby on Rails, wordpress, phpmyadmin and more). From my consulting experience I should say that this is the most popular lighttpd deployment scheme (lighttpd frontend + some backends) and this is why these chapters should be the most interesting for new users. As a RoR server administrator I wish authors would explain more different deployment options for Rails, but of course I understand that the book is dedicated to lighttpd, not Rails so it is OK to miss some options.

And finally, for experienced users like me who understands all the internals of the web traffic handling and needs to get more control over the traffic there is just awesome chapters on using Lua with lighttpd and writing custom lighttpd modules. I really wish I had such an explanations when I was doing nginx modules development for our projects! If I’d write some lighttpd module I’d definitely buy this book just because of those Lua and modules-related chapters - there is not so much info on these topics on the Net and such a solid explanation of the basics of web server extension could help a lot.

So, as the final words on this book, I’d like to recommend it to all web services administrators (even if they do not use lighttpd yet) - this book explains many fundamental things that you’d really like to know. As for professional developers/admins, this books could be useful if they are going to work on some advanced configurations or modules for lighttpd (yes, I loved those lua and modules sections).

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) and queue start growing, the thing could kill your server and you won’t be able to do something about it - it just eats all your memory and this is it. Since then I’ve started looking for a better solution for our queuing, the technology was too cool to give up. I’ve tried 5 or 6 different popular solutions and all of them sucked… They ALL had the same problem - if your queue grows, this is your problem and not queue broker’s :-/ The last solution I’ve tested was ActiveMQ and either I wasn’t able to push it to its limits or it is really so cool, but looks like it does not have this memory problem. So, we’ve started using it recently.

In this small post I’d like to describe a few things that took me pretty long to figure out in ruby Stomp client: how to make queues persistent (really!) and how to process elements one by one with clients’ acknowledgments.

First, we need to connect to the queue server and this code is the same for clients and servers:

Second form looks better for me because it allows you to specify if you want client to be reliable (lock on errors and try to reconnect, etc) or just want client to raise an error in case of any problems.

When you’ve connected to the server, you can push your data to a queue:

And this is where you have an ability to specify if you want your data to be persistent (survive server crashes, etc): headers is a hash that could have an element :persistent => true, which would do the thing. So, your code would looks like this (for example):

Now, when you have your data submitted to the queue, you need to be able to read it and process with some script. This is as simple as the following code:

Again, we wanted to receive messages one by one and acknowledge successful processing in our code. This is simple too. You need:

• Pass :ack => :client as an element in your headers hash
• Call client.acknowledge(msg) in the loop if you’re sure that an element could be removed from the queue

This is basically it with the stuff I wanted to explain today. If you’ve never tried to work with any queue servers, try today and maybe tomorrow you won’t be able to imagine your systems architecture without such a component

Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.

Few months ago my good friend has joined Scribd and we’ve started thinking on this problem together. As the result of our brainstorming we’ve managed to figure out what were the problems preventing us from doing efficient caching:

• First of all, a lots of code in the action is changing the page view if our visitor is a bot (no, not a cloaking, just some minor adjustments of the view).
• Second problem was a set of differences in the view for anonymous and logged in users.
• And finally, third problem was the fact that the page has a few blocks that change pretty dynamically: document stats pane and comments lists.

All these problems when combined were creating a lots of pain when I was trying to cache a whole page. When we’ve figured them out, we’ve started thinking on how could we generalize possible combinations of those factors and possible approaches to caching.

There is a well known idea in web applications development: the fastest web app action is an action that does not require any code to be executed on your application server. So, first idea we’ve tried to think about was some approach that would definitely reduce the number of hits on our app servers. This idea was based on HTTP protocol features related to Last-Modified and E-Tag headers. But there was a problem - not so many users go to the same page twice so even if we’d make the page cacheable, it wouldn’t help too much. But the idea of full page caching outside of the application was really good and we’ve started playing with it to figure out how to use it in production.

Long time ago, when Internet was slow and expensive many ISPs and large companies were trying to reduce their traffic w/o hurting users’ experience. Then caching proxy servers were born. The idea of those servers was to handle all web requests going from a network (ISP or a company office) and try to cache as much content as possible so when the same or some other user would request a cached page, proxy server would return it really fast. If we’d implement support for those Last-Modified headers, all proxy servers would be happy to cache our pages. But there was a problem - no one uses caching proxies in 2008 So, we’ve got an idea - why can’t we place such a server in front of our application and make it cache content for all users in the world? (Yes, we knew about a caching reverse proxies before - I’m just trying to explain the flow of our thoughts and words when we were brainstorming the problem).

The only problem with this approach would be to differentiate logged in users, anonymous users and bots. Considering the fact that our proxy server could be placed between the app and our web servers (nginx), we’ve decided to create a nginx module that would translate the same document page URLs to a set of URLs, which would be different for all those 3 kinds of users.

When all those problems with different kinds of users were solved, we’ve decided to solve the last one - non-cacheable dynamic stats pane. The solution was pretty simple - we’ve added a small ajax call to the page which would update stats on the cached version of our page for all real users while bots will see the same page, but with a bit stale stats pane.

Long story short, the results is really great. Application servers load reduced by 50-70%, database servers load is reduced by 30-60%, response times dropped down to 150-200 msec from 500-750 msec. As an additional positive effect of the caching we’ve managed to remove all fragments caches from the application and free more of memcached resources for data caches. Here are a few cacti graphs of our servers load/traffic (the caching was introduced on Oct 9th at night):

Main MySQL command counters:

Uploaded with plasq’s Skitch!

One of our Application Servers CPU Usage:

Uploaded with plasq’s Skitch!

One of our Application Servers Load Average:

Uploaded with plasq’s Skitch!

Unfortunately there are a lot of things to share related to this caching experience, so I’ve decided to make a series of posts that would explain all the problems we had and solutions we’ve found for each of the following parts of the caching system:

• Nginx module development
• Squid Server setup (configs and hardware)
• Rails code to support Last-Modified headers and how we purge caches

So, if you’re interested in details, subscribe to this blog’s RSS feed and in a few days you’ll see the first article from this series.

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

The package itself consists of two parts:

• Perl scripts to process incoming email:
  • bounces processor — could be assigned to process all your bounce emails
  • feedback loops messages processor — more specific for Scribd, but still - could be modified for your needs (will be released soon).
• Rails plugin to work with mailing blacklists

For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.

Notice: The box can’t be restarted to check with clean caches and stuff, but I was doing full reads from this huge table on a machine with 16Gb RAM so all caches were washed out by this load anyways.

As you can see, less work linux does on its side to optimize disk I/O, slower it works Actually it was pretty expected, but still - surprising result. The problem (as guys from Youtube explained on the last year MySQL Conf) is there because Linux knows nothing about RAID’s internals and specific drives’ queues so when it is trying to re-arrange requests in its own queue it wastes CPU resources and could potentially prevent RAID controller from doing its own queue optimizations.

After this test I’ve tried it on a few other I/O-bound servers (both read and write bound) and the result was the same - noop (do nothing) I/O scheduler gave me the best results. Long story short, I’ve decided to try this scheduler on all our boxes for a week and look at the results in cacti graphs to see how it works.