Alexey Kovyrin's Blog

DB Charmer – ActiveRecord Connection Magic Plugin

Scoundrel — Tue, 03 Nov 2009 15:36:26 +0000

Today I’m proud to announce the first public release of our ActiveRecord database connection magic plugin: DbCharmer.

DB Charmer – ActiveRecord Connection Magic Plugin

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

Allows you to easily manage AR models’ connections (switch_connection_to method)
Allows you to switch AR models’ default connections to a separate servers/databases
Allows you to easily choose where your query should go (on_* methods family)
Allows you to automatically send read queries to your slaves while masters would handle all the updates.
Adds multiple databases migrations to ActiveRecord

Installation

There are two options when approaching db-charmer installation:

using the gem (recommended)
install as a Rails plugin

To install as a gem, add this to your environment.rb:

1 2	config.gem 'db-charmer', :lib => 'db_charmer', :source => 'http://gemcutter.org'

And then run the command:

1	sudo rake gems:install

To install db-charmer as a Rails plugin use this:

1	script/plugin install git://github.com/kovyrin/db-charmer.git

Easy ActiveRecord Connection Management

As a part of this plugin we’ve added switch_connection_to method that accepts many different kinds of db connections specifications and uses them on a model. We support:

Strings and symbols as the names of connection configuration blocks in database.yml.
ActiveRecord models (we’d use connection currently set up on a model).
Database connections (Model.connection)
Nil values to reset model to default connection.

Sample code:

1
2
3
4
5
6
7

class Foo < ActiveRecord::Model; end

Foo.switch_connection_to(:blah)
Foo.switch_connection_to('foo')
Foo.switch_connection_to(Bar)
Foo.switch_connection_to(Baz.connection)
Foo.switch_connection_to(nil)

The switch_connection_to method has an optional second parameter should_exist which is true by default. This parameter is used when the method is called with a string or a symbol connection name and there is no such connection configuration in the database.yml file. If this parameter is true, an exception would be raised, otherwise, the error would be ignored and no connection change would happen.

This is really useful when in development mode or in a tests you do not want to create many different databases on your local machine and just want to put all your tables in a single database.

Warning: All the connection switching calls would switch connection only for those classes the method called on. You can’t call the switch_connection_to method and switch connection for a base class in some hierarchy (for example, you can’t switch AR::Base connection and see all your models switched to the new connection, use the classic establish_connection instead).

Multiple DB Migrations

In every application that works with many databases, there is need in a convenient schema migrations mechanism.

All Rails users already have this mechanism – rails migrations. So in DbCharmer, we’ve made it possible to seamlessly use multiple databases in Rails migrations.

There are two methods available in migrations to operate on more than one database:

1. Global connection change method – used to switch whole migration to a non-default database.
2. Block-level connection change method – could be used to do only a part of a migration on a non-default db.

Migration class example (global connection rewrite):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class MultiDbTest < ActiveRecord::Migration
db_magic :connection => :second_db

def self.up
create_table :test_table, :force => true do |t|
t.string :test_string
t.timestamps
end
end

def self.down
drop_table :test_table
end
end

Migration class example (block-level connection rewrite):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class MultiDbTest < ActiveRecord::Migration
def self.up
on_db :second_db do
create_table :test_table, :force => true do |t|
t.string :test_string
t.timestamps
end
end
end

def self.down
on_db :second_db { drop_table :test_table }
end
end

By default in development and test environments you could skip this :second_db connection from your database.yml files and rails would create the tables in your single database, but in production you’d specify it and get the table created on a separate server and/or in a separate database.

This behaviour is controlled by the DbCharmer.migration_connections_should_exist configuration attribute which could be set from a rails initializer.

Using Models in Master-Slave Environments

Master-slave replication is the most popular scale-out technique in a medium-sized and large database-centric applications today. There are some rails plugins out there that help developers to use slave servers in their models but none of them were flexible enough for us to start using them in a huge application we work on.

So, we’ve been using ActsAsReadonlyable plugin for a long time and have made tons of changes in its code over the time. But since that plugin has been abandoned by its authors, we’ve decided to collect all of our master-slave code in one plugin and release it for rails 2.2+. DbCharmer adds the following features to Rails models:

Auto-Switching all Reads to the Slave(s)

When you create a model, you could use db_magic :slave => :blah or db_magic :slaves => [ :foo, :bar ] commands in your model to set up reads redirection mode when all your find/count/exist/etc methods will be reading data from your slave (or a bunch of slaves in a round-robin manner). Here is an example:

1
2
3
4
5
6
7

class Foo < ActiveRecord::Base
db_magic :slave => :slave01
end

class Bar < ActiveRecord::Base
db_magic :slaves => [ :slave01, :slave02 ]
end

Default Connection Switching

If you have more than one master-slave cluster (or simply more than one database) in your database environment, then you might want to change the default database connection of some of your models. You could do that by using db_magic :connection => :foo call from your models. Example:

1
2
3

class Foo < ActiveRecord::Base
db_magic :connection => :foo
end

Sample model on a separate master-slave cluster (so, separate main connection + a slave connection):

1
2
3

class Bar < ActiveRecord::Base
db_magic :connection => :bar, :slave => :bar_slave
end

Per-Query Connection Management

Sometimes you have select queries that you know you want to run on the master. This could happen for example when you have just added some data and need to read it back and not sure if it made it all the way to the slave yet or no. For this situation and a few others there is a set of methods we’ve added to ActiveRecord models:

1) +on_master+ – this method could be used in two forms: block form and proxy form. In the block form you could force connection switch for a block of code:

1
2
3
4

User.on_master do
user = User.find_by_login('foo')
user.update_attributes!(:activated => true)
end

In the proxy form this method could be used to force one query to be performed on the master database server:

1
2
3

Comment.on_master.last(:limit => 5)
User.on_master.find_by_activation_code(code)
User.on_master.exists?(:login => login, :password => password)

2) on_slave – this method is used to force a query to be run on a slave even in situations when it’s been previously forced to use the master. If there is more than one slave, one would be selected randomly. This method has two forms as well: block and proxy.

3) on_db(connection) – this method is what makes two previous methods possible. It is used to switch a model’s connection to some db for a short block of code or even for one statement (two forms). It accepts the same range of values as the switch_connection_to method does. Example:

1 2	Comment.on_db(:olap).count Post.on_db(:foo).find(:first)

Associations Connection Management

ActiveRecord models can have an associations with each other and since every model has its own database connections, it becomes pretty hard to manage connections in a chained calls like User.posts.count. With a class-only connection switching methods this call would look like the following if we’d want to count posts on a separate database:

1	Post.on_db(:olap) { User.posts.count }

Apparently this is not the best way to write the code and we’ve implemented an on_* methods on associations as well so you could do things like this:

1 2	@user.posts.on_db(:olap).count @user.posts.on_slave.find(:title => 'Hello, world!')

Notice: Since ActiveRecord associations implemented as proxies for resulting objects/collections, it is possible to use our connection switching methods even without chained methods:

1 2	@post.user.on_slave # would return post's author @photo.owner.on_slave # would return photo's owner

Starting with DbCharmer release 1.4 it is possible to use prefix notation for has_many and HABTM associations connection switching:

1 2	@user.on_db(:foo).posts @user.on_slave.posts

Named Scopes Support

To make it easier for DbCharmer users to use connections switching methods with named scopes, we’ve added on_* methods support on the scopes as well. All the following scope chains would do exactly the same way (the query would be executed on the :foo database connection):

1
2
3
4

Post.on_db(:foo).published.with_comments.spam_marked.count
Post.published.on_db(:foo).with_comments.spam_marked.count
Post.published.with_comments.on_db(:foo).spam_marked.count
Post.published.with_comments.spam_marked.on_db(:foo).count

And now, add this feature to our associations support and here is what we could do:

1
2
3

@user.on_db(:archive).posts.published.all
@user.posts.on_db(:olap).published.count
@user.posts.published.on_db(:foo).first

Documentation

For more information on the plugin internals, please check out the source code. All the plugin’s code is ~100% covered with a tests that were placed in a separate staging rails project located at github. The project has unit tests for all or at least the most of the parts of plugin’s code.

What Ruby and Rails implementations does it work for?

We have a continuous integration setups for this plugin on MRI 1.8.6 with Rails 2.2 and 2.3. We use the plugin in production on Scribd.com with MRI (rubyee) 1.8.6 and Rails 2.2.

Who are the authors?

This plugin has been created in Scribd.com for our internal use and then the sources were opened for other people to use. All the code in this package has been developed by Alexey Kovyrin for Scribd.com and is released under the MIT license. For more details, see the LICENSE file.

If you have any comments on this project, feel free to contact me here in comments or by email. And, of course, patches are welcome (only when covered with tests).

Advanced Squid Caching in Scribd: Hardware + Software Used

Scoundrel — Tue, 04 Aug 2009 05:23:18 +0000

After the previous post in this caching related series I’ve received many questions on hardware and software configuration of our servers so in this post I’ll describe our server’s configs and the motivation behind those configs.

Hardware Configuration

Since in our setup Squid server uses one-process model (with an asynchronous requests processing) there was no point in ordering multi-core CPUs for our boxes and since we have a lots of pages on the site and the cache is pretty huge all the servers ended up being highly I/O bound. Considering these facts we’ve decided to use the following hardware specs for the servers:

CPU: One pretty cheap dual-core Intel Xeon 5148 (no need in multiple cores or really high frequencies – even these CPUs have ~1% avg load)
RAM: 8Gb (basically to reduce I/O pressure by caching hot content in RAM)
Disks: 4 x small SAS 15k drives in JBOD mode (no RAIDS – we’ve tried all kinds of RAID configs and it did not help with the I/O performance)

So, once again: nothing is as important in a squid box as I/O throughput.

Here is a sample CPU load graph from one of the boxes:

Software Configuration

This could be a long story, but in a few words our experience with different squid versions was the following.

First, when I’ve started working on this caching project I’ve just installed squid using Debian’s apt-get install squid command. As the result we’ve got some ancient squid 2.6 release that for some reason (still unclear to me) was painfully slow in I/O operations and it had some leaking file descriptors problem so after a few hours under production load the box would simply stop processing requests.

When the first approach failed, I’ve decided to go to the squid web site, download the latest production release and install it from sources (yes, we do it all the time when OS vendor ships too old or buggy releases). Result – freaking fast and stable squid 3.0 which worked flawlessly for about 5 months.

Few months ago we’ve found out about the stale-* extensions available in squid 2.7 and I’ve started wondering if we should change our perfectly stable 3.0 setup to 2.7. And some time later I’ve decided to use Vary HTTP header in our caching architecture and then I found out that vary-caching correctly implemented only in 2.7 and since 3.0 is a complete rewrite of the 2.X branch, vary-caching is not yet implemented there (or not in a way we’d want it to be implemented).

So, the final result: at this moment in time we’re using custom-built Squid 2.7STABLE6 and really happy with it, it is stable, fast and feature-rich caching proxy server.

Caching Cluster Configuration

Apparently we have more than one squid server in scribd and this makes it a bit harder to use those servers (comparing to one box when you’d send all requests to one IP:port pair). We’ve tried to use round-robin balancing for the squid boxes + ICP-based neighbor checks but it was adding more latency to our responses and we’ve decided to put haproxy load balancer between nginx and squid farm and set up URL hash based balancing to distribute requests evenly amongst squid backends.

This scheme worked pretty nice, but we had one serious problem with this setup: if one squid box would go down, haproxy would quickly detect the problem and would remove it from the pool… And here comes the problem – removing a server from the pool completely changes hashing keys space and all cached requests become invalid. To solve this problem we’ve developed a nginx balancer module that performs consistent hashing of URLs and we’re testing this module now in production. What is really good about this module is that it removes one hop from the chain if http proxies between the site and a user.

So, this was a short description of what hardware we use for our caching cluster and why do we use it. In the next posts of this series we’ll talk about cache control and objects invalidation.

UPDATE: squid-3.0 was not a rewrite of Squid-2. It’s a continuation of Squid-2.5, with a conversion to compile under C++.

Advanced Squid Caching in Scribd: Logged In Users and Complex URLs Handling

Scoundrel — Tue, 21 Jul 2009 22:18:57 +0000

It’s been a while since I’ve posted my first post about the way we do document pages caching in Scribd and this approach has definitely proven to be really effective since then. In the second post of this series I’d like to explain how we handle our complex document URLs and logged in users in the caching architecture.

First of all, let’s take a look at a typical Scribd’s document URL: http://www.scribd.com/doc/1/Improved-Statistical-Test.

As we can see, it consists of a document-specific part (/doc/1) and a non-unique human-readable slug part (/Improved-Statistical-Test). When a user comes to the site with a wrong slug in the document URL, we need to make sure we send the user to the correct URL with a permanent HTTP 301 redirect. So, obviously we can’t simply send our requests to the squid because it’d cause few problems:

When we change document’s title, we’d create a new cached item and would not be able to redirect users from the old URL to the new one
When we change a title, we’d pollute cache with additional document page copies.

One more problem that makes the situation even worse – we have 3 different kinds of users on the site:

Logged in users – active web site users that are logged in and should see their name at the top of the page, should see all kinds of customized parts of the page, etc (especially when a page is their own document).
Anonymous users – all users that are not logged in and visit the site with a flash-enabled browser
Bots – all kinds of crawlers that can’t read flash content and need to see a plain text document version

All three kinds of users should see their own document page versions whether the page is cached or not.

So, how do we solve these two problems? Here is how.

First of all, to fix the URLs problem we’ve decided to rewrite the URL before it goes to a squid server. We change URLs to look like this: http://www.scribd.com/doc/1?__enable_docview_caching=1. This makes the document URL dependent only on a unique document ID that never change and sends an additional parameter to the backend to signal that the page could potentially be cached. The slug is sent to backend using an HTTP-header (X-Scribd-Slug) so that backend could check the slug and return a redirect if needed.

To make sure we won’t respond with a cached page to a request with an invalid URL (invalid slug basically), we use Vary: X-Scribd-Slug HTTP header which is implemented in Squid (only late 2.6 and 2.7) and makes it check specified headers in a request before responding with a cached content. If the header of the cached content is different then the header in the request, squid proxies the resuest to backend where we could handle the cache miss as we want.

Next, to resolve the users problem we’ve created a small nginx module that looking at a request headers could tell you whether the user is a bot or not and whether he’s logged in or an anonymous visitor. This module basically exposes a $scribd_user_id variable to our configs and we could use the variable to do separate configuration for different kinds of users.

At this point we do not cache document pages for logged in users so we basically have two copies of each page in the cache: flash-enabled document page and an inline document page. We do this separation by changing our cache URLs one more time: we add a scribd_user_id=$scribd_user_id variable (anonymous = 0, bot = -1) to the cache URL: http://www.scribd.com/doc/1?__enable_docview_caching=1&scribd_user_id=0.

And last, not not least, we use two really awesome Squid features called stale-while-revalidate and stale-if-error (AFAIR, they were invented in Yahoo! and then described by their squid admin).

Option stale-while-revalidate allow us to quickly serve content from the cache while doing background re-validation requests to the Rails backend. Option stale-if-error basically allows us to serve content from the squid cache when Rails backend is down/dead/slow.

All these changes allowed us to handle more traffic with less hardware and what is even more important, they helped us improve user experience with the site: response times dropped 2-3 fold and much less people see our Ouch! pages (HTTP 50x errors when backend is dead or overloaded). Here is an example of one of our servers’ hit ratio and traffic savings daily graphs:

This it with the logged in users and complex URLs handling in Scribd caching architecture. Next post in this series will explain how we do cache invalidation in Scribd. Stay tuned.

The Blog v.2.0

Scoundrel — Mon, 20 Jul 2009 02:19:51 +0000

Long time ago, in 2002 I decided to create my own point of presence in the Internet. Back then I’ve got pretty nice domain (scoundrel.kremenchug.net), hacked up a few pages on php, added a guestbook and that was it. Many years it was almost static and I did a few updates on my resume page few times a year. Later I’ve switched the site to wordpress to make it easier to manage my resume and stuff…

And 3 years ago in March 2006 I’ve decided to start my own blog. I took a standard template and started the blog on a separate domain while the domain was on its own domain name… This spring my wife made me a great birthday present – she’s created me a custom blog design that has all the stuff I wanted from my own web site for a long time. My friend Dima Shteflyuk has helped me with creating a wordpress template from Tanya’s mockups and here we are – now I’ve decided to merge my blog and my web site into a single web entity called http://kovyrin.net/. Welcome to my new blog/site/whatever!

Loops plugin for rails and merb released

Scoundrel — Tue, 17 Feb 2009 03:41:30 +0000

loops is a small and lightweight framework for Ruby on Rails and Merb created to support simple background loops in your application which are usually used to do some background data processing on your servers (queue workers, batch tasks processors, etc).

Originally loops plugin was created to make our (Scribd.com) own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin itself.

The major idea behind this small project was to create a deadly simple and yet robust framework to be able to run some tasks in background and do not think about spawning many workers, restarting them when they die, etc. So, if you need to be able to run either one or many copies of your worker or you do not want to think about re-spawning dead workers and do not want to spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your loop as a separate process controlled by monit/god/etc), then I’d recommend you to try this framework — you’ll like it.

For more information, visit the project site and, of course, read the sources

Rails Developer for a Large Startup: My Vision of an Ideal Candidate

Scoundrel — Sat, 07 Feb 2009 07:41:22 +0000

Few days ago we were chatting in our corporate Campfire room and one of the guys asked me what do I think about Rails developers hiring process, what questions I’d ask a candidate, etc… This question started really long and interesting discussion and I’d like to share my thoughts on this question in this post.

So, first of all I would like to explain what kind of interviews I really hate Ever since I was thinking of myself as of a developer (many years ago) and was going to “software developer position” interviews I really hated questions like “What is the name and possible values of the third parameter of the function some_freakin_weird_func() from some_weird.h” or “How to declare a virtual destructor and when it could be useful?”… All my life I had pretty practical thinking and never bothered to learn APIs or some really deep language concepts that are useful in 1% of the development time in really weird edge case situations. I always thought that a Real Engineer should know where to find an answer and should be able to find a way to implement something without knowing all possible ways to do it – engineering mind should be flexible!

So, if I’d interview a developer for our company, I’d never ask them this weird kind of theoretical questions because honestly speaking web development is not a rocket science and in most of the cases you don’t need to have a CS Master degree to be a good developer.

So, let’s get closer to our discussion in Scribd. During this discussion we were talking about our standard hiring practices, about questions we used to ask our candidates and such. As the result I decided to write a set of questions I’d really like an ideal candidate to be able to answer. Many of them are not rails specific, but rather high-load and high-scale specific. Let me share some of those questions here.

You have a generic web application (does not matter if it is rails or not) with a set of layers and parties involved in each request:
- Database
- Application
- Internet
- User
What security problems could occur on each level and what action should developer take to prevent those problems from happening? For example take a login action in your application and explain possible problems.
You have a table with 100 millions of records, you need to make some changes in each records and it is obvious that you won’t be able to do that using some SQL queries, so you need to do in some application-level code (one time use script). Your database is MySQL.
- Where would you put the code in the application: model, module with a bunch of methods, migration, some dedicated script?
- How would you process all those records: Model.find(:all).each {}, a loop with different offsets and a limit, a loop walking through primary key values with some step, something else?
- What would you choose for this task and why: Model.find or Model.find_by_sql? What is the major difference between those two? When would you definitely use find_by_sql?
- What would you use: update_attribute, attribute assignments + saves, Model.connection.execute or something else to update the table? When and why would you use each of those?
You have a rails action that accepts uploads and then transfers them to a pretty slow storage (S3, some slow nfs, something else). If you have a lots of uploads, pretty soon many of your mongrels (if not all) will be used for uploads and your application would become unusable. What options do you see to solve this problem?
You have a huge table in mysql, you need to select a few randomly selected records (and this is not a one-time action, it’ll be used pretty often, like on the home page), how’d you do that?
You have a document model with the following fields: id, user_id, title, body, created_at, views_count, download_count. There are 10000000 documents. You need to implement a download counters on your site so when one downloads a document, you’d increase the download_counter field. How would you implement counter management?
How’d you write an SQL query to find all users in a system that have no docs (standard User has_many docs relationship)?
Caching:
- What kinds of caching techniques do you know?
- What caching options does Rails provide, when would you use those?
- What caching options does HTTP protocol provide?
- You have a pretty popular action on your site which uses some heavy SQL query which should be cached. But the problem is that sometimes when the cache expires and too many users hit the action, all those SQL queries will start hammering the database trying to refresh your caches… That causes huge problems. How’d you try to solve the problem?

As you can see, these questions aren’t asking you about REST or new cool Rails buzzwords (ActiveResource, named_scope, etc, etc) because IMHO – you can learn those in a week of skimming through any rails-related mailing list. My questions are aimed at the most often happening problems in high-traffic startups where not all common practices could be used and you need to think before writing some code that’d be executed millions of times per day.

So, if you’ve read the questions and you think you know (or is able to find in Google/books/etc) answers and you think you know Rails I think you should contact us because we’re hiring. At this moment we need a few really great Ruby-developers that are not afraid of working on high-load web applications and who isn’t just a buzzword junkies, but a developers that could use their tools effectively.

For more information and more formal position information, please check out our jobs page or just email me with any questions related to these positions.

P. S. If you’re really good, we could work with you remotely w/o relocation to San Francisco (we have some team members in Canada and Ukraine now).

ActiveMQ Tips: Flow Control and Stalled Producers Problem

Scoundrel — Fri, 23 Jan 2009 07:23:09 +0000

It’s been a few months since we‘ve started actively using ActiveMQ queue server in our project. For some time we had pretty weird problems with it and even started thinking about switching to something else or even writing our own queue server which would comply with our requirements. The most annoying problem was the following: some time after activemq restart everything worked really well and then activemq started lagging, queue started growing and all producer processes were stalling on push() operations. We rewrote our producers from Ruby to JRuby, then to Java and still – after some time everything was in a bad shape until we restarted the queue server.

So, long story short, after a lots of docs and source code reading we’ve found really interesting thing. There is a “feature” added in the recent ActiveMQ release (5.X) called “flow control“, which is used by ActiveMQ core to slow down too fast producers when consumers lag or just aren’t as fast as ActiveMQ wants them to be. This results in unpredictable producers stalls and all kinds of weird problems.

If you’re experiencing the same problems, you could try to add the following to your activemq.xml file to make it stop using flow control and let your producers spool data to the queue as fast as it is possible (that’s exactly what I’d expect from a queue server):

1
2
3
4
5
6
7

>
>
>
queue=">" producerFlowControl="false" memoryLimit="64mb" />
>
>
>

When this is added, all your queues (if they take more than 64Mb of RAM) will start persisting their messages to the disk and freeing your RAM to consume more messages w/o any slowdowns.

Another useful feature we use in our producer is Async Sends which basically means that your producers won’t require any acknowledgments from ActiveMQ after it pushes your messages to a queue. This makes push() operations MUCH faster and your producers’ throughput will become tremendously higher. If you use Java as your producer language, you could use the following code to make your connection async:

1
2
3
4
5
6

queueConnectionFactory = new ActiveMQConnectionFactory(queue_user, queue_password, queue_url);
queueConnectionPool = new PooledConnectionFactory(queueConnectionFactory);
queueConnectionPool.setMaxConnections(200);

// This line is used to make your connection async
((ActiveMQConnectionFactory)queueConnectionFactory).setUseAsyncSend(true);

After we’ve made the changes explained above, our queue server and producers became rock stable and blazing fast so there is no point to look for another solution because looks like I was right – ActiveMQ is an amazing piece of software – you just need to learn it before using.

This is it for today, if you have any questions, feel free to ask them in the comments and I’d be glad to answer.

Using SSH tunnel connection as a SOCKS5 proxy

Scoundrel — Thu, 11 Dec 2008 23:13:51 +0000

Month ago I was on a vacation and as usual even though our hotel provided us with an internet connection on a pretty decent speeds, I wasn’t able to work there because they’ve banned all tcp ports but some major ones (like 80, 21, etc) and I needed to be able to use ssh, mysql, IMs and other non-web software.

After a short research I’ve found a pretty simple to set up and easy to use approach to such a connection problems I’d like to describe here.

First, you’ll need someone (or you can do it before leaving home) to start an ssh daemon on port 80 on one of your servers. I use one of my Slicehub slices for this to permanently have an ability to use it. You can do it like this (if it is a temporary solution):

1	# `which sshd` -p 80

Notice: this `which sshd` was used because on some OSes sshd does not want to start w/o an absolute path to its binary.

If you’d like to have it permanent, just add one line to your /etc/ssh/sshd_config file and restart your ssh server:

Port 80

OK, this step is finished and you have sshd listening on port 80. Now, let’s imagine you go somewhere and need to use ssh (or ICQ/GTalk/Jabber/MSN/AIM, or torrents) and some weird admin banned all tcp ports but port 80. Here’s what you’ll need to do:

You open terminal on your machine (most likely it’ll be your laptop)
You run the following command:
1
$ ssh -D 1080 -p 80 -v user@your-host.com "sleep 1000000"
Use 127.0.0.1 (port 1080) as your local socks4/5 proxy server (no authorization) in any software you want

When you have theses steps are finished, you can go to any place you want and wherever you have an access to web servers, you’ll have an access to anything you need (tested on: ssh, safari, adium, vuze torrent client, rubygems, macports).

P. S. On my macbook to make this thing even more useful I’ve installed tsocks port (non-mac users, check it out here).

Lighttpd Book from Packt – Great Thanksgiving Present

Scoundrel — Thu, 27 Nov 2008 07:15:30 +0000

Many people know me as a nginx web server evangelist. But as (IMHO) any professional I think that it is really rewarding to know as much as possible about all the tools available on the market so every time you need to make a decision on some technical issue, you’d consider all pros and cons based on my own knowledge.

This is why when I received an email from Packt company asking if I’d like to read and review their book on Lighttpd I decided to give it a shot (I usually do not review any books because I do not always have enough time to read a book thoroughly to be able to write a review). So, here are my impressions from this book.

First, when I received the book, I was in doubt: how such a small book could cover so flexible and multi-purpose piece of software like Lighttpd? But after reading a few first chapters I understood that the book is written as any professional book should be written – no weird long chapters explaining what is Internet, web server, HTTP or HTML. From the first pages you dive to the world of Lighttpd starting from installation and going through the most popular and useful configuration options to really interesting virtual hosting configurations.

Pretty interesting security-related chapters explain typical process of obtaining and installing SSL certificates. This always was the most painful process for me in almost any web setup I worked on, don’t know why though, this process seemed kind of messy for me, but I think those chapters could help novice web server administrators understand the process of securing a web server.

For many new lighttpd users the most interesting chapters should be ones that explain typical problems and use cases of using apache with or instead of apache servers and describe how to set up lighttpd to be used with the most popular backend web software (Ruby on Rails, wordpress, phpmyadmin and more). From my consulting experience I should say that this is the most popular lighttpd deployment scheme (lighttpd frontend + some backends) and this is why these chapters should be the most interesting for new users. As a RoR server administrator I wish authors would explain more different deployment options for Rails, but of course I understand that the book is dedicated to lighttpd, not Rails so it is OK to miss some options.

And finally, for experienced users like me who understands all the internals of the web traffic handling and needs to get more control over the traffic there is just awesome chapters on using Lua with lighttpd and writing custom lighttpd modules. I really wish I had such an explanations when I was doing nginx modules development for our projects! If I’d write some lighttpd module I’d definitely buy this book just because of those Lua and modules-related chapters – there is not so much info on these topics on the Net and such a solid explanation of the basics of web server extension could help a lot.

So, as the final words on this book, I’d like to recommend it to all web services administrators (even if they do not use lighttpd yet) – this book explains many fundamental things that you’d really like to know. As for professional developers/admins, this books could be useful if they are going to work on some advanced configurations or modules for lighttpd (yes, I loved those lua and modules sections).

ActiveMQ + Ruby Stomp Client: How to process elements one by one

Scoundrel — Thu, 30 Oct 2008 04:42:39 +0000

Few months ago I’ve switched one of our internal projects from doing synchronous database saves of analytics data to an asynchronous processing using starling + a pool of workers. This was the day when I really understood the power of specialized queue servers. I was using database (mostly, MySQL) for this kind of tasks for years and sometimes (especially under a highly concurrent load) it worked not so fast… Few times I worked with some queue servers, but those were either some small tasks or I didn’t have a time to really get the idea, that specialized queue servers were created just to do these tasks quickly and efficiently.

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) and queue start growing, the thing could kill your server and you won’t be able to do something about it – it just eats all your memory and this is it. Since then I’ve started looking for a better solution for our queuing, the technology was too cool to give up. I’ve tried 5 or 6 different popular solutions and all of them sucked… They ALL had the same problem – if your queue grows, this is your problem and not queue broker’s :-/ The last solution I’ve tested was ActiveMQ and either I wasn’t able to push it to its limits or it is really so cool, but looks like it does not have this memory problem. So, we’ve started using it recently.

In this small post I’d like to describe a few things that took me pretty long to figure out in ruby Stomp client: how to make queues persistent (really!) and how to process elements one by one with clients’ acknowledgments.

First, we need to connect to the queue server and this code is the same for clients and servers:

1
2
3

client = Stomp::Client.open "stomp://localhost:61613"
or
client = Stomp::Client.open(login, password, host, port, reliable)

Second form looks better for me because it allows you to specify if you want client to be reliable (lock on errors and try to reconnect, etc) or just want client to raise an error in case of any problems.

When you’ve connected to the server, you can push your data to a queue:

1	client.send('/queue/some_queue', "hello world", headers)

And this is where you have an ability to specify if you want your data to be persistent (survive server crashes, etc): headers is a hash that could have an element :persistent => true, which would do the thing. So, your code would looks like this (for example):

1 2	client.send('/queue/some_queue', "hello world", :persistent => true) client.close # this is needed only in your push code

Now, when you have your data submitted to the queue, you need to be able to read it and process with some script. This is as simple as the following code:

1
2
3
4
5
6

# Processing loop
client.subscribe('/queue/some_queue', headers) do |msg|
# Process your message here
# Your submitted data is in msg.body
end
client.join # Wait until listening thread dies

Again, we wanted to receive messages one by one and acknowledge successful processing in our code. This is simple too. You need:

Pass :ack => :client as an element in your headers hash
Call client.acknowledge(msg) in the loop if you’re sure that an element could be removed from the queue

This is basically it with the stuff I wanted to explain today. If you’ve never tried to work with any queue servers, try today and maybe tomorrow you won’t be able to imagine your systems architecture without such a component