asemanfar

Detecting User Automation

Tue, 14 Jul 2009 23:30:52 -0700

Over the past several months at Serious Business, we've had several incidents of users botting our games by using various Greasemonkey scripts all the way up to automated binaries that login and level you up without even launching a browser. At one point it got so bad that we doubled our requests per second due to bots. How do we fight this?

War is Bad

The first occurrence of this was when people were automating the collection of randomly appearing coins on user profile pages. We quickly thwarted this bot by placing decoy coins that humans wouldn't be able to see and using the click as an indicator of automation. We punished these users ultimately by taking away what they got by cheating, and of course they were upset. We quickly learned that this wasn't going to work against all bots and that targeted punishment pisses people off; it was a war we didn't want to fight and quickly abandoned.

New Perspective

The next logical step was to approach the problem from a context-free perspective. Ignore the individual bots and what they do, what's common about all bots? They can't pass CAPTCHAs. But we can't show all 4 million of our users a CAPTCHA when the login, it'd get annoying and be bad for business. So we have to distinguish between those who we suspect are botting and those are clearly not.

The Heuristics

In order to pick out the ~~cylons~~ bots from the humans, we needed to identify some heuristics. Among the many different things we could choose, we chose requests per second and session length for our first implementation.

Enter the fun part...

Implementation

We have several million users, hundreds of requests per second, 11 app servers, and hundreds of mongrels. We need to collect user data, analyze it, and feed it to the mongrels to act on flagged users. The main concerns are that it must not affect response times, it mustn't take up much space, and it should be fast. The solution we came up with has two parts a pandemic daemon and a mongrel handler (using pre-rack Rails).

The mongrel handler sends an asynchronous request to a cluster of daemons. Each node in the cluster is responsible for tracking and analyzing a subset of the users' activity, which enables us to easily scale it out by adding more nodes. Each node periodically analyzes the new data and publishes a list of misbehaving users, which each mongrels pulls down. All user activity is stored only in memory and thrown out once they become inactive.

Asynchronously collecting data, analyzing it offline and in-memory , and periodically syncing it back to the mongrels turned out to efficient with virtually zero impact on response times and server load.

Pandemic -- because information needs to be contagious

Fri, 10 Apr 2009 23:03:50 -0700

Pandemic is a map-reduce framework. You give it the map, process, and reduce methods and it handles the rest. It's designed to serve requests in real-time, but can also be used for offline tasks.

It's different from the typical map-reduce framework in that it doesn't have a master-worker structure. Every node can map, process, and reduce. It also doesn't have the concept of jobs, everything is a request.

The framework is designed to be as flexible as possible, there is no rigid request format, or API, you can specify it however you want. You can send it http-style headers and a body, you can send it JSON, or you can even just send it a single line and have it do whatever you want. The only requirement is that you write your handler to appropriately act on the request and return the response.

Here is how you use it:

Server require 'rubygems' require 'pandemic'



class Handler < Pandemic::ServerSide::Handler
  def process(body)

body.reverse


  end
end

pandemic_server = epidemic! pandemic_server.handler = Handler.new pandemic_server.start.join

In this example, the handler doesn't define the map or reduce methods, and the defaults are used. The default for each is as follows:

map: Send the full request body to every connected node
process: Return the body (do nothing)
reduce: Concatenate all the responses

Client require 'rubygems' require 'pandemic'



class TextFlipper
  include Pandemize
  def flip(str)

pandemic.request(str)

end end

Config

Both the server and client have config files:



pandemic_server.yml

servers: - host1:4000 - host2:4000 response_timeout: 0.5 Each value for the server list is the host:port that a node can bind to. The servers value can be a hash or an array of hashes, but I'll get to that later. The response timeout is how long to wait for responses from nodes before returning to the client.



pandemic_client.yml

servers: - host1:4000 - host2:4000 max_connections_per_server: 10 min_connections_per_server: 1 response_timeout: 1
The min/max connections refers to how many connections to each node. If you're using the client in Rails, then just use 1 for both min/max since it's single threaded.

More Config

There are three ways to start a server:

ruby server.rb -i 0
ruby server.rb -i machine1hostname
ruby server.rb -a localhost:4000

The first refers to the index in the servers array: servers: - host1:4000 # started with ruby server.rb -i 0 - host2:4000 # started with ruby server.rb -i 1
The second refers to the index in the servers hash. This can be particularly useful if you use the hostname as the key. servers: machine1: host1:4000 # started with ruby server.rb -i machine1 machine2: host2:4000 # started with ruby server.rb -i machine2
The third is to specify the host and port explicitly. Ensure that the host and port you specify is actually in the config otherwise the other nodes won't be able to communicate with it.

You can also set node-specific configuration options. servers: - host1:4000:



  database: pandemic_node_1
  host: localhost
  username: foobar
  password: f00bar

host2:4000: database: pandemic_node_2 host: localhost username: fizzbuzz password: f1zzbuzz And you can access these additional options using config.get(keys) in your handler: class Handler < Pandemic::ServerSide::Handler def initialize @dbh = Mysql.real_connect(*config.get('host', 'username', 'password', 'database')) end end

Code: github repository

Install:
sudo gem -a http://gems.github.com
sudo gem install arya-pandemic

Request Queue via Mongrel Proctitle

Wed, 11 Feb 2009 23:04:57 -0800

In our cluster, we run many mongrels on several app servers all behind a load balancer. In order to get an idea of how each app server and its mongrels are doing, we use rtomayko's mongrel_proctitle gem. This gem sets up a mongrel handler that extracts request information from the client's request and sets it as the process title so when you do a ps aux or run top you can see each mongrel's queue length and request info:

mongrel_rails [8000/1/4]: handling 127.0.0.1: GET /users



that's [port/queue length/requests handled]: handling client ip: request

Problem

Last week, we were having some misbehaving mongrels that would lock up and essentially stop serving requests. We'd look at the process titles and see something like this:

mongrel rails [8021/14/6123]: handling 127.0.0.1: GET /status

We have an action designated to be a lightweight health check to let the load balancer know what's up and according to this process title, this action was locking up the mongrel and growing to a queue length of around 14. This didn't make sense, why is the lightest weight action locking up the mongrel?

Solution

With a little digging, I found that the mongrel_proctitle gem is broken. More precisely, it is only accurate when there is a single request being handled and none queued. The handler was setting the process title as soon as it received the request, which then led that client's thread to the next handler, which happens to be the synchronized Rails handler. So the request the process title shows is the most recent received request and not the current request being handled.

With a few small modifications, we now have a more accurate process title that also shows the rest of the queue (up to a character limit) as a comma-separated list:

mongrel rails [8021/2/6123]: handling 127.0.0.1: GET /users, 127.0.0.1: GET /status

The updated version is available on github and also as a gem at gems.github.com as arya-mongrel_proctitle:

gem sources -a http://gems.github.com sudo gem install arya-mongrel_proctitle

Note, as described in the comments in the code, it's still not exact. There is at least one race case where the entire list won't be ordered accurately, but it's unlikely to occur and still gives a good idea of what's going on.

I Went Unobtrusive

Sat, 08 Nov 2008 10:44:24 -0800

So I've been cleaning up my blog's code base and decided to switch to unobtrusive javascript. I've heard many reasons advocating its use but for some reason, they never stuck. Until a couple of weeks ago when I worked on a Railsrumble project with a couple of buddies of mine, one of which was a big advocate of unobtrusive javascript using LowPro. Actually using unobtrusive javascript turned out to be much more gratifying that I anticipated. It satisfied some inner OCD tendencies of mine that a lot of programmers share and it made developing javascript'd UIs much easier to develop and much much easier to maintain.

Looking back, I don't think know why I chose to use the Rails javascript helpers or RJS; I like writing javascript, especially when there are great frameworks out there like LowPro and Prototype. The fact that the Rails javascript helpers generate inline javascript, which sometimes turns out being longer than a simple one-liner, is kind of gross. As for RJS, the fact that it's generates code that contains both the data and the logic makes OCD baby Jesus cry.

The separation of HTML (markup), CSS (presentation), Javascript (interactivity) is almost as awesome as the separation of church and state...oh wait, crap.

If you haven't already switched, do it now!