Bionic Spirit

How To Build a Naive Bayes Classifier

2012-02-09T00:00:00+02:00

Some use-cases for building a classifier:

Spam detection, for example you could build your own Akismet API
Automatic assignment of categories to a set of items
Automatic detection of the primary language (e.g. Google Translate)
Sentiment analysis, which in simple terms refers to discovering if an opinion is about love or hate about a certain topic

In general you can do a lot better with more specialized techniques, however the Naive Bayes classifier is general-purpose, simple to implement and good-enough for most applications. And while other algorithms give better accuracy, in general I discovered that having better data in combination with an algorithm that you can tweak does give better results for less effort.

In this article I'm describing the math behind it. Don't fear the math, as this is simple enough that a high-schooler understands. And even though there are a lot of libraries out there that already do this, you're far better off for understanding the concept behind it, otherwise you won't be able to tweak the implementation in response to your needs.

0. The Source Code

I published the source-code associated at github.com/alexandru/stuff-classifier. The implementation itself is at lib/bayes.rb, with the corresponding test/test_002_naive_bayes.rb.

1. Introduction to Probabilities

Let's start by refreshing forgotten knowledge. Again, this is very basic stuff, but if you can't follow the theory here, you can always go to the probabilities section on khanacademy.org.

1.1. Events and Event Types

An "event" is a set of outcomes (a subset of all possible outcomes) with a probability attached. So when flipping a coin, we can have one of these 2 events happening: tail or head. Each of them has a probability of 50%. Using a Venn diagram, this would look like this:

And another example which clearly shows the dependence between "rain" and "cloud formation", as raining can only happen if there are clouds:

The relationship between events is very important, as you'll see next:

2 events are disjoint (exclusive) if they can't happen at the same time (a single coin flip cannot yield a tail and a head at the same time). For Bayes classification, we are not concerned with disjoint events.
2 events are independent when they can happen at the same time, but the occurrence of one event does not make the occurrence of another more or less probable. For example the second coin-flip you make is not affected by the outcome of the first coin-flip.
2 events are dependent if the outcome of one affects the other. In the example above, clearly it cannot rain without a cloud formation. Also, in a horse race, some horses have better performance on rainy days.

What we are concerned here is the difference between dependent and independent events, because calculating the intersection (both happening at the same time) depends on it. So for independent events, calculating the intersection is easy:

Some examples:

if you have 2 hard-drives, each of them having a 0.3 (30%) probability of failure within the next year, that means there's a 0.09 (9%) probability of them failing both within the next year
if you flip a coin 4 times, there's a 0.0625 probability of getting a tail 4 times in a row (0.5 ^ 4)

Things are not so simple for dependent events, which is where the Bayes Theorem comes into play.

1.2. Conditional Probabilities and The Bayes Theorem

Let's take one example. So we have the following stats:

30 emails out of a total of 74 are spam messages
51 emails out of those 74 contain the word "penis"
20 emails containing the word "penis" have been marked as spam

So the question is: what is the probability that the latest received email is a spam message, given that it contains the word "penis"?

So these 2 events are clearly dependent, which is why you must use the simple form of the Bayes Theorem:

With the solution being:

This was a simple one, you could definitely see the result without complicating yourself with the Bayes formula.

1.3. The Naive Bayes Approach

Let us complicate the problem above by adding to it:

25 emails out of the total contain the word "viagra"
24 emails out of those have been marked as spam
so what's the probability that an email is spam, given that it contains both "viagra" and "penis"?

Shit just got more complicated, because now the formula is this one:

And you definitely don't want to bother with it if we keep adding words. But what if we simplified our assumptions and just say that the occurrence of penis is totally independent from the occurrence of viagra? Then the formula just got much simpler:

To classify an email as spam, you'll have to calculate the conditional probability by taking hints from the words contained. And the Naive Bayes approach is exactly what I described above: we make the assumption that the occurrence of one word is totally unrelated to the occurrence of another, to simplify the processing and complexity involved.

This does highlight the flaw of this method of classification, because clearly those 2 events we've picked (viagra and penis) are correlated and our assumption is wrong. But this just means our results will be less accurate.

2. Implementation

I mention it again, you can take a look at the source-code published at github.com/alexandru/stuff-classifier.

2.1. General Algorithm

You simply get the probability for a text to belong to each of the categories you test against. The category with the highest probability for the given text wins:

Do note that above I also eliminated the denominator from our original formula, because it is a constant that we do not need (called evidence).

2.2. Avoiding Floating Point Underflow (UPDATE Feb 27, 2012)

Because of the underlying limits of floating points, if you're working with big documents (not the case in this example), you do have to make one important optimization to the above formula:

instead of the probabilities of each word, you store the (natural) logarithms of those probabilities
instead of multiplying the numbers, you add them instead

So instead of the above formula, if you need this optimization, then use this one:

2.3. Training

Your implementation must have a training method. Here's how mine looks like:

def train(category, text)
  each_word(text) {|w| increment_word(w, category) }
  increment_cat(category)
end

And its usage:

classifier.train :spam, "Grow your penis to 20 inches in just 1 week"
classifier.train :ham,  "I'm hungry, no I don't want your penis"

For the full implementation, take a look at base.rb.

2.4. Getting Rid of Stop Words / Stemming

First of all, you must get rid of the junk. Every language has words that are so commonly used that make them meaningless for any kind of classification you may want to do. For instance in English you have words such as "the", "to", "you", "he", "only", "if", "it" that you can safely strip out from the text.

I've compiled a list of such words in this file: stop_words.rb. You can compile such a list by yourself if you're not using English for example. Head over to Project Gutenberg, download some books in the language you want, count the words in them, sort by popularity in descending order and keep the top words as words that you can safely ignore.

Also, our classifier is really dumb in the sense that it does not care about the meaning or context of a word. So there's a problem: consider the word "running". What you want is to treat this just as "run", which is the morphological root of the word. You also want to treat "parenting" and "parents" as "parent".

This process is called stemming and there are lots of libraries for it. I think currently the most up-to-date and comprehensive library for stemming is Snowball. It's a C library with lots of bindings available, including for Ruby and Python and it even has support for my native language (Romanian).

Take a look at what I'm doing in tokenizer.rb, where I'm getting rid of stop words and stemming the remainings.

each_word('Hello world! How are you?')

# => ["hello", "world"]

each_word('Lots of dogs, lots of cats! 
  This is the information highway')

# => ["lot", "dog", "lot", "cat", "inform", "highwai"]

each_word("I don't really get what you want to
  accomplish. There is a class TestEval2, you can do test_eval2 =
  TestEval2.new afterwards. And: class A ... end always yields nil, so
  your output is ok I guess ;-)")

# => ["really", "want", "accomplish", "class",
#     "testeval", "test", "eval", "testeval", "new", 
#     "class", "end", "yields", "nil", "output", 
#     "ok", "guess"]

NOTE: depending on the size of your training data, this may not be a good idea. Stemming is useful in the beginning when you don't have a lot of data. Otherwise consider "house" and "housing" ... the former is used less frequently in a spammy context then the later.

2.5. Implementation Guidelines

When classifying emails for spam, it is a good idea to be sure that a certain message is a spam message, otherwise users may get pissed by too many false positives.

Therefore it is a good idea to have thresholds. This is how my implementation looks like:

def classify(text, default=nil)
  # Find the category with the highest probability

  max_prob = 0.0
  best = nil
  
  scores = cat_scores(text)
  scores.each do |score|
    cat, prob = score
    if prob > max_prob
      max_prob = prob
      best = cat
    end
  end

  # Return the default category in case the threshold condition was
  # not met. For example, if the threshold for :spam is 1.2
  #
  #    :spam => 0.73, :ham => 0.40  (OK)
  #    :spam => 0.80, :ham => 0.70  (Fail, :ham is too close)

  return default unless best
  threshold = @thresholds[best] || 1.0

  scores.each do |score|
    cat, prob = score
    next if cat == best
    return default if prob * threshold > max_prob
  end

  return best
end

Final Words

My example involved spam classification, however this is not how modern spam classifiers work btw. Because the independence assumptions are often inaccurate, this type of classifier can be gamed by spammers to trigger a lot of false positives, which will make the user turn the feature off eventually.

But it is general purpose, being good enough not only for spam detection, but also for lots of other use-cases and it's enough to get you started.

Data Mining: Finding Similar Items and Users

2012-01-16T00:00:00+02:00

Because we want to give kick-ass product recommendations.

I'm showing you how to find related items based on a really simple formula. If you pay attention, this technique is used all over the web (like on Amazon) to personalize the user experience and increase conversion rates.

To get one question out of the way: there are already many available libraries that do this, but as you'll see there are multiple ways of skinning the cat and you won't be able to pick the right one without understanding the process, at least intuitively.

Defining the Problem

To find similar items to a certain item, you've got to first define what it means for 2 items to be similar and this depends on the problem you're trying to solve:

on a blog, you may want to suggest similar articles that share the same tags, or that have been viewed by the same people viewing the item you want to compare with
Amazon has this section called "customers that bought this item also bought", which is self-explanatory
a service like IMDB, based on your ratings, could find users similar to you, users that liked or hated approximately the same movies you did, thus giving you suggestions on movies you'd like to watch in the future

In each case you need a way to classify these items you're comparing, whether it is tags, or items purchased, or movies reviewed. We'll be using tags, as it is simpler, but the formula holds for more complicated instances.

Redefining the Problem in Terms of Geometry

We'll be using my blog as sample. Let's take some tags:

["API", "Algorithms", "Amazon", "Android", "Books", "Browser"]

That's 6 tags. Well, what if we considered these tags as dimensions in a 6-dimensional Euclidean space? Then each item you want to sort or compare becomes a point in this space, in which a coordinate (representing a tag) is either one (tagged) or zero (not tagged).

So let's say we've got one article tagged with API and Browser. Then its associated point will be:

[ 1, 0, 0, 0, 0, 1 ]

Now these coordinates could represent something else. For instance they could represent users. If say you've got a total of 6 users in your system, 2 of them rating an item with 3 and 5 stars respectively, you could have for the article in question this associated point (do note the order is very important):

[ 0, 3, 0, 0, 5, 0 ]

So now you can go ahead and calculate distances between these points. For instance you could calculate the angle between the associated vectors, or the actual euclidean distance between the 2 points. For a 2-dimensional Euclidean space, here's how it would look like:

Euclidean Distance

The mathematical formula for the Euclidean distance is really simple. Considering 2 points, A and B, with their associated coordinates, the distance is defined as:

The lower the distance between 2 points, then the higher the similarity. Here's some Ruby code:

# Returns the Euclidean distance between 2 points
#
# Params:
#  - a, b: list of coordinates (float or integer)
#
def euclidean_distance(a, b)
  sq = a.zip(b).map{|a,b| (a - b) ** 2}
  Math.sqrt(sq.inject(0) {|s,c| s + c})
end

# Returns the associated point of our tags_set, relative to our
# tags_space.
#
# Params:
#  - tags_set: list of tags
#  - tags_space: _ordered_ list of tags
def tags_to_point(tags_set, tags_space)
  tags_space.map{|c| tags_set.member?(c) ? 1 : 0}
end

# Returns other_items sorted by similarity to this_item 
# (most relevant are first in the returned list)
#
# Params:
#  - items: list of hashes that have [:tags]
#  - by_these_tags: list of tags to compare with
def sort_by_similarity(items, by_these_tags)
  tags_space = by_these_tags + items.map{|x| x[:tags]}  
  tags_space.flatten!.sort!.uniq!

  this_point = tags_to_point(by_these_tags, tags_space)
  other_points = items.map{|i| 
    [i, tags_to_point(i[:tags], tags_space)]
  }

  similarities = other_points.map{|item, that_point|
    [item, euclidean_distance(this_point, that_point)]
  }
  
  sorted = similarities.sort {|a,b| a[1] <=> b[1]}
  return sorted.map{|point,s| point}
end

And here is the test you could do, and btw you can copy the above and the bellow script and run it directly:

# SAMPLE DATA

all_articles = [
  {
   :article => "Data Mining: Finding Similar Items", 
   :tags => ["Algorithms", "Programming", "Mining", 
     "Python", "Ruby"]
  }, 
  {
   :article => "Blogging Platform for Hackers",  
   :tags => ["Publishing", "Server", "Cloud", "Heroku", 
     "Jekyll", "GAE"]
  }, 
  {
   :article => "UX Tip: Don't Hurt Me On Sign-Up", 
   :tags => ["Web", "Design", "UX"]
  }, 
  {
   :article => "Crawling the Android Marketplace", 
   :tags => ["Python", "Android", "Mining", 
     "Web", "API"]
  }
]

# SORTING these articles by similarity with an article 
# tagged with Publishing + Web + API
#
#
# The list is returned in this order:
#
# 1. article: Crawling the Android Marketplace
#    similarity: 2.0
#
# 2. article: "UX Tip: Don't Hurt Me On Sign-Up"
#    similarity: 2.0
#
# 3. article: Blogging Platform for Hackers
#    similarity: 2.645751
#
# 4. article: "Data Mining: Finding Similar Items"
#    similarity: 2.828427
#

sorted = sort_by_similarity(
    all_articles, ['Publishing', 'Web', 'API'])

require 'yaml'
puts YAML.dump(sorted)

The Problem (or Strength) of Euclidean Distance

Can you see one flaw with it for our chosen data-set and intention? I think you can - the first 2 articles have the same Euclidean distance to ["Publishing", "Web", "API"], even though the first article shares 2 tags with our chosen item, instead of just 1 tag as the rest.

To visualize why, look at the points used in calculating the distance for the first article:

[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]
[1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1]

So 4 coordinates are different. Now look at the points used for the second article:

[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]
[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1]

Again, 4 coordinates are different. So here's the deal with Euclidean distance: it measures dissimilarity. The coordinates that are the same are less important than the coordinates that are different. For my purpose here, this is not good - because articles with more tags (or less) tags than the average are going to be disadvantaged.

Cosine Similarity

This method is very similar to the one above, but does tend to give slightly different results, because this one actually measures similarity instead of dissimilarity. Here's the formula:

If you look at the visual with the 2 axis and 2 points, we need the cosine of the angle theta that's between the vectors associated with our 2 points. And for our sample it does give better results.

The values will range between -1 and 1. -1 means that 2 items are total opposites, 0 means that the 2 items are independent of each other and 1 means that the 2 items are very similar (btw, because we are only doing zeros and ones for coordinates here, this score will never get negative for our sample).

Here's the Ruby code (leaving out the wiring to our sample data, do that as an exercise):

def dot_product(a, b)
  products = a.zip(b).map{|a, b| a * b}
  products.inject(0) {|s,p| s + p}
end

def magnitude(point)
  squares = point.map{|x| x ** 2}
  Math.sqrt(squares.inject(0) {|s, c| s + c})
end

# Returns the cosine of the angle between the vectors 
#associated with 2 points
#
# Params:
#  - a, b: list of coordinates (float or integer)
#
def cosine_similarity(a, b)
  dot_product(a, b) / (magnitude(a) * magnitude(b))
end

Also, sorting the articles in the above sample gives me the following:

- article: Crawling the Android Marketplace
  similarity: 0.5163977794943222

- article: "UX Tip: Don't Hurt Me On Sign-Up"
  similarity: 0.33333333333333337

- article: Blogging Platform for Hackers
  similarity: 0.23570226039551587

- article: "Data Mining: Finding Similar Items"
  similarity: 0.0

Right, so much better for this chosen sample and usage. Ain't this fun? BUT, you guessed it, there's a problem with this too ...

The Problem with Our Sample; The Tf-Idf Weight

Our data sample is so simple that we could have simply counted the number of common tags and use that as a metric. The result would be the same without getting fancy with Cosine Similarity :-)

Clearly a tag such as "Heroku" is more specific than a general purpose tag such as "Web". Also, just because Jekyll was mentioned in an article, that doesn't make the article about Jekyll. Also an article tagged with "Android" may be twice as Android-related as another article also tagged with "Android".

So here's a solution to this: the Tf-Idf weight, a statistical measure used to evaluate how important a word is to a document in a collection or corpus. With it you can give values to your coordinates that are much more specific than simple ones and zeros. But I'll leave that for another day.

Also, related to our simple data-set here, perhaps an even simpler metric, like the Jaccard index would be better.

Pearson Correlation Coefficient

The Pearson Correlation Coefficient for finding the similarity of 2 items is slightly more sophisticated and doesn't really apply to my chosen data-set. This coefficient measures how well two samples are linearly related.

For example, on IMDB we may have 2 users. One of them, lets call him John, has given the following ratings to 5 movies: [1, 2, 3, 4, 5]. The other one, Mary, has given the following ratings to the same 5 movies: [4, 5, 6, 7, 8]. The 2 users are very similar, as there is a perfect linear correlation between them, since Mary just gives the same rankings as John plus 3. The formula itself or the theory is not very intuitive though. But it is simple to calculate:

Here's the code:

def pearson_score(a, b)
  n = a.length
  return 0 unless n > 0

  # summing the preferences
  sum1 = a.inject(0) {|sum, c| sum + c}
  sum2 = b.inject(0) {|sum, c| sum + c}
  # summing up the squares
  sum1_sq = a.inject(0) {|sum, c| sum + c ** 2}
  sum2_sq = b.inject(0) {|sum, c| sum + c ** 2}
  # summing up the product
  prod_sum = a.zip(b).inject(0) {|sum, ab| sum + ab[0] * ab[1]}
  
  # calculating the Pearson score
  num = prod_sum - (sum1 *sum2 / n)  
  den = Math.sqrt((sum1_sq - (sum1 ** 2) / n) * (sum2_sq - (sum2 ** 2) / n))

  return 0 if den == 0
  return num / den  
end


puts pearson_score([1,2,3,4,5], [4,5,6,7,8])
# => 1.0
puts pearson_score([1,2,3,4,5], [4,5,0,7,8])
# => 0.5063696835418333
puts pearson_score([1,2,3,4,5], [4,5,0,7,7])
# => 0.4338609156373132
puts pearson_score([1,2,3,4,5], [8,7,6,5,4])
# => -1

Manhattan Distance

There is no one size fits all and the formula you're going to use depends on your data and what you want out of it.

For instance the Manhattan Distance computes the distance that would be traveled to get from one data point to the other if a grid-like path is followed. I like this graphic from Wikipedia that perfectly illustrates the difference with Euclidean distance:

Red, yellow and blue lines all have the same length and the distance is bigger than the corresponding green diagonal, which is the normal Euclidean distance.

Personally I haven't found a usage for it, as it is more related to path-finding algorithms, but it's a good thing to keep in mind that it exists and may prove useful. Since it measures how many changes you have to do to your origin location to get to your destination while being limited to taking small steps in a grid-like system, it is very similar in spirit to the Levenshtein distance, which measures the minimum number of changes required to transform some text into another.

Blogging Platform for Hackers

2012-01-05T00:00:00+02:00

I'm showing you how to:

host your own static website on Heroku's free plan;
use Google's App Engine as a CDN, for better responsiveness;
keep Heroku's free dyno alive, by using a GAE cron job;
have a very responsive, scalable and secure blog, with ultimate; control and simplicity, for zero bucks per month;

You could just skip this article and browse the source code of my blog:

github.com/alexandru/bionicspirit.com

Forget about Wordpress or Blogger. Hacking your own stuff is much more fun. Also, make sure to read Blogging Like a Hacker, by Tom Preston-Werner, GitHub's cofounder and the author of Jekyll.

UPDATE: article was changed three times to better express rationale and in response to user feedback.

Jekyll and Heroku, Sitting in a Tree

I love Jekyll, the static website generator. It is pure awesomeness for me:

all content is hosted in a Git repository, the best CMS ever invented
my articles are written in Markdown, with Emacs, the most potent text editor ever created - think Textmate-snippets, macros, syntax highlighting, keyboard-driven navigation and spelling corrections
static content scales like crazy, without any special gimmicks. A small VPS can serve thousands of requests per second without a sweat
static content is also secure by default, no constant upgrades required, no SQL injections
I always make little tweaks to my design, I'm never satisfied, which is why it makes sense to make my own, but checkout Octopress in case you want a reasonable default
I've lost an entire blog when my hosting account got blocked in the past. Never again, as my content is right now saved in 2 Git repositories and on my local machine
by working with my own domain, making my own shit, Google will never make me cry ;-)

Jekyll's first hosting option you should consider is GitHub Pages, however you will need some dynamic behavior, like having configurable redirects. If you don't then ignore this post and just read Jekyll's tutorial, but you can come back to this post when its limits start bothering you.

Heroku's free plan is awesome, in spite of what I said previously. It's great for prototyping and for quickly seeing your website online. Instant gratification is awesome. Well, it does have some problems and to tell you the truth, for hosting my blog I would have rather used Google's App Engine, if only they allowed me to have naked domains. I like my domains to be naked.

One note in regards to the scalability of static content I mentioned above. In Heroku the Bamboo stack features a Varnish frontend. If you set proper expiry headers on your content, subsequent requests will not hit the Ruby server.

Hosting Static Content on Heroku

So this tutorial is about hosting a Jekyll website, which is why I'm going to make some assumptions about your directory structure. However you can modify these instructions for any static website, not just Jekyll-generated stuff.

First, the setup:

# install the heroku command-line utility
gem install heroku

# change to your website directory
cd website/

# initialize a git repo, if you haven't done so
git init
# ... and commit everything to it
git add .
git commit -m 'initial commit'

# create the heroku app
heroku create

OK, now we need a Rake-powered application to serve our content. We'll need a ./Gemfile ...

source 'http://rubygems.org'

gem 'rack'
gem 'mime-types'

group :development do
  gem 'jekyll'
  gem 'rdiscount'
  gem 'hpricot'  
end

Then install these gems with:

bundle install

You also need a Rake configuration file, ./config.ru. What follows is the configuration that I am using. You can go simpler, a lot simpler than this actually, but I like flexibility and Heroku also does something funny with files served through Rack::File, so I refrained from using it ...

# Rack configuration file for serving a Jekyll-generated static
# website from Heroku, with some nice additions:
#
# * knows how to do redirects, with settings taken from ./_config.yaml
# * sets the cache expiry for HTML files differently from other static
#   assets, with settings taken from ./_config.yaml

require 'yaml'
require 'mime/types'

# main configuration file, also used by Jekyll
CONFIG = YAML::load_file(File.join(File.dirname(__FILE__), '_config.yml'))

# points to our generated website directory
PUBLIC = File.expand_path(File.join(File.dirname(__FILE__), 
                          CONFIG['destination'] || '_site'))

# For cutting down on the boilerplate

class BaseMiddleware
  def initialize(app)
    @app = app
  end

  def each(&block)
  end
end


# Rack middleware for correcting paths:
#  
# 1. redirects from the www. version to the naked domain version
#
# 2. converts directory/paths/ to directory/paths/index.html (most
#    importantly / to /index.html)

class PathCorrections < BaseMiddleware
  def call(env)
    env['PATH_INFO'] += 'index.html' if env['PATH_INFO'].end_with? '/'
    request = Rack::Request.new(env)
    
    if request.host.start_with?("www.")
      [301, {"Location" => request.url.sub("//www.", "//")}, self]
    else
      @app.call(env)
    end    
  end
end


# Middleware that enables configurable redirects. The configuration is
# done in the standard Jekyll _config.yml file.
#
# Sample configuration in _config.yml:
#
#   redirects:
#     - from: /docs/some-document.html
#       to: /archive/some-document.html
#       type: 301
#
# The sample above will do a permanent redirect from ((*/docs/dialer.html*))
# to ((*/archive/some-document.html*))

class Redirects < BaseMiddleware
  def call(env)
    request = Rack::Request.new(env)

    path = request.path_info
    ext  = File.extname(path)
    path += '/' if ext == '' and ! path.end_with?('/')

    if redirect = CONFIG['redirects'].find{|x| path == x['from']}
      new_location = redirect['to']
      new_location = request.base_url + new_location \
        unless new_location.start_with?("http")
      [redirect['type'] || 302, {'Location' => new_location}, self]
    else
      @app.call(env)
    end
  end
end


# The 404 Not Found message should be a simple one in case the
# mimetype of a file is not HTML (like the message returned by
# Rack::File). However, in case of HTML files, then we should display
# a custom 404 message

class Fancy404NotFound < BaseMiddleware
  def call(env)
    status, headers, response = @app.call(env)
    if status == 404 
      ext = File.extname(env['PATH_INFO'])
      if ext =~ /html?$/ or ext == '' or !ext
        headers = {'Content-Type' => 'text/html'}
        response = File.open(File.join PUBLIC, 'pages', '404.html')
      end
    end

    [status, headers, response]
  end
end


# Mimicking Rack::File
#
# I couldn't work with Rack::File directly, because for some reason
# Heroku prevents me from overriding the Cache-Control header, setting
# it to 12 hours. But 12 hours is not suitable for HTML content that
# may receive fixes and other assets should have an expiry in the far 
# future, with 12 hours not being enough. 

class Application < BaseMiddleware
  class Http404 < Exception; end

  def guess_mimetype(path)
    type = MIME::Types.of(path)[0] || nil
    type ? type.to_s : nil
  end

  def call(env)
    request = Rack::Request.new(env)
    path_info = request.path_info

    # a /ping request always hits the Ruby Rake server - useful in
    # case you want to setup a cron to check if the server is still
    # online or bring it back to life in case it sleeps

    if path_info == "/ping"
      return [200, {
          'Content-Type' => 'text/plain', 
          'Cache-Control' => 'no-cache'
      }, [DateTime.now.to_s]]
    end
    
    headers = {}
    if mimetype = guess_mimetype(path_info)
      headers['Content-Type'] = mimetype
      if mimetype == 'text/html'
        headers['Content-Language'] = 'en' 
        headers['Content-Type'] += "; charset=utf-8"
      end
    end
    
    begin
      # basic validation of the path provided
      raise Http404 if path_info.include? '..'
      abs_path = File.join(PUBLIC, path_info[1..-1])
      raise Http404 unless File.exists? abs_path

      # setting Cache-Control expiry headers
      type = path_info =~ /\.html?$/ ? 'html' : 'assets'
      headers['Cache-Control']  = "public, max-age="
      headers['Cache-Control'] += CONFIG['expires'][type].to_s

      status, response = 200, File.open(abs_path, 'r')
    rescue Http404
      status, response = 404, ["404 Not Found: #{path_info}"]
    end

    [status, headers, response]
  end
end


#
# the actual Rack configuration, using 
# the middleware defined above
#

use Redirects
use PathCorrections
use Fancy404NotFound

run Application.new(PUBLIC)

This Rack configuration uses settings defined in the standard Jekyll _config.yaml file. Here are some settings needed for it to work as intended:

destination: ./_site

expires:
  html: 3600 # one hour
  assets: 1314000 # one year

redirects:
  - from: /rss/
    to: http://feeds.feedburner.com/bionicspirit
    type: 302

OK, so once done, test this configuration:

# generating the website
jekyll

# starting the server
rackup

Deployment is as easy as pie:

git push heroku master

One note: Heroku could be configured to automatically generate the website for you. However you either have to use the Cedar stack, or generate the pages on the fly. In case of the Cedar stack, you lose Varnish. Just keep your generated files in Git, it's easier.

Commenting with Disqus, Facebook or Roll Your Own

For commenting Disqus is a really good service. In case you have a very popular website amongst normal people, it may be even better to integrate Facebook's commenting widget.

Well, I had some fun a while ago and created my own: TheBuzzEngine.

Unfortunately it doesn't have all the features I want, but it does get the job done and it isn't bloated. These days I'll probably get around to adding some stuff to it, like threaded comments and email subscriptions. This is what happens when working for fun on stuff - once you're over a certain threshold, the return of investment is too low to bother with extra development.

I recommend Disqus, although rolling your own is fun and keeps you in control (which is the reason I'm using Jekyll in the first place).

Using Google App Engine as Your CDN or Cron Manager

So when using Heroku's free plan, I feel a little uncomfortable because relying on one dyno can get you in trouble. Having Varnish in front is great, but Varnish is a cache manager. For instance, if you happen to push a new version of your latest article to Heroku, then the Varnish cache gets cleared and the Ruby server can potentially get exposed to a lot of requests and one dyno on Heroku can only serve one request at a time.

So why not push all our static assets, except HTML files, to a CDN? It's best practice anyway as your website should be more responsive. If you have an Amazon AWS account, then CloudFront + S3 are great.

However, I started with the goal of hosting this for zero bucks (it's fun, so why not?). Therefore I'm going to teach you how to push your files to Google's App Engine. I don't really know how GAE works as a CDN for static files, but it seems that it does have the properties of a CDN (i.e. serving content to users from servers closer to their location).

Another problem with Heroku's free plan is that the free dyno goes to sleep, to save resources. While I advise you to just pay up for an extra dyno, you can get around this restriction by just configuring GAE to send a periodic ping to your website.

Here's my GAE configuration file, app.yaml which should sit in your root (assuming ./assets is the directory you want to serve from GAE):

application: assets-bionicspirit
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:

- url: /assets
  static_dir: assets
  expiration: "365d"

# next item is for our cron job, described below, but you can ignore
# it if you don't want a cron job ...

- url: /tasks/ping
  script: ping.app

As you can see, I'm setting the expiry of my static assets to a whooping 1 year.

I also have a real handler, at /tasks/ping configured. This will be our cron job that sends a ping to our Heroku app, every X minutes. Here's the code for ping.py ...

import webapp2
from google.appengine.api import urlfetch

class PingService(webapp2.RequestHandler):
  def get(self):
      self.response.headers['Content-Type'] = 'text/plain'

      url = "http://bionicspirit.com/ping"
      try:
          result = urlfetch.fetch(url, deadline=30)
          self.response.out.write('HTTP %d - %s' % 
              (result.status_code, (result.content or '').strip()))
      except:
          self.response.out.write('ERROR: no response')

app = webapp2.WSGIApplication([('/tasks/ping', PingService)], debug=False)

But we are not done. To configure /tasks/ping to run every X minutes, you also need a cron.yaml file ...

cron:
- description: ping bionicspirit.com to wake it up
  url: /tasks/ping
  schedule: every 4 minutes

Assuming you already have the GAE SDK installed, then run this command:

appcfg.py update .

To see it working on this blog, here are the requests:

Heroku URL getting requested: http://bionicspirit.com/ping
GAE Cron Job getting executed: http://assets-bionicspirit.appspot.com/tasks/ping

Extra Tip - CloudFlare

Luigi Montanez kindly pointed out in below's comments the availability of CloudFlare.

CloudFlare is a proxy that sits between your website and your users. It allegedly prevents DDOS attacks on your website, but it also caches static content, which helps because apparently it also has the properties of a CDN.

I activated it to see how it works. The main reason is that GAE has a 1 GB bandwidth-out daily limit - and this article generated ~ 10,000 visits in only one day, which consumed ~ 700 MB of bandwidth on GAE (for a couple of small images, I don't want to imagine what would happen for an image-rich post). So that's not good and I placed CloudFlare in front of GAE and my Heroku instance, which should save some bandwidth for me.

I don't have a conclusion on CloudFlare. If it works as advertised, then it is awesome. Although be careful about it as I've seen reports on the Internet that it may in fact add latency to your website, instead of decreasing it.

For my website however, everything seems to be fine. I am monitoring my website with Pingdom.com, a service which also reports the average responsiveness of the website, calculated by doing requests from multiple locations. The homepage, which is not cached by CloudFlare or served by GAE, has an average load time of 300ms, while cached static resources from GAE and proxied through CloudFlare are doing much better.

So we'll see.

Conclusion

The result is a really responsive, scalable and kick-ass blog, for zero bucks spent on hosting.

This very blog is hosted using the method described above. Well, I'll probably return to my trustworthy VPS instance as I'm paying for it anyway, but this was fun.

Enjoy ~

Crawling the Android Marketplace

2011-12-15T00:00:00+02:00

I had a very specific need for fetching the details for some apps in the marketplace, in an automated manner. And so I found the supermarket gem, a wrapper for the Android Market API Java implementation. However, it gives unpredictable results (it wouldn't return the details of our in-house apps or of many other examples I tried) and Google is placing hard-limits on the number of requests you can make per minute. This is an internal API, probably used by the marketplace client and the implementation mentioned above was created through reverse-engineering.

This really pissed me off, this is Google, they should grok APIs. But this info is already available from their website and so I went ahead and crawled it.

The script and the data collected are is available. Read below.

How To Do it By Yourself

The actual script that I created can be found in the AndroidMarketCrawler GitHub Repository, with the relevant files being:

crawler.py - source code with lots of comments, it's really not complicated, you should go read it
marketplace_database.json_lines.bz2 - compressed file containing the details of the crawled apps, one per each line; this is not a proper JSON file, you use it by reading it line by line, where each line represents a JSON object (personal preference, as otherwise the file is pretty big and you can run out of memory)

UPDATE: The Android Marketplace explicitly bans crawling apparently. This crawler and associated data only serves educational purposes. Don't abuse it.

for app in AndroidMarketCrawler(concurrency=10):
    # app is at this point a dictionary with the details needed, like
    #  id, name, developer name, number of installs, etc...
    fh.write(json.dumps(app) + "\n")

I used the Python programming language, along with Eventlet for fetching URLs in parallel (async I/O with epoll/libevent, providing you with coroutines support and green threads) and PyQuery for selecting DOM elements using CSS3 selectors (instead of XPath or BeautifulSoup). If you fancy Ruby instead, you could use slight equivalents such as em-http-request and Nokogiri.

So you start fetching content from a root and add application links as you encounter them in a queue. We are then using a (green) threadpool to start fetching jobs for each of the links in the queue. So it's recursive. The results are also pushed in another queue, ready to be consumed by the client.

Be careful though, don't abuse this, as it will generate a ton of traffic and your IP may end up being banned by Google. It also takes a lot of time; with good bandwidth and a VPS located in California, it still took me 5 hours for the script to finish. Don't abuse the concurrency settings either, 10 is enough.

155,200 Apps Available From the US

You have to realize that this number is only approximate. Apps are going strong in other countries, such as South Korea and Google does Geo-IP filtering, which means some of the apps were unavailable to me, depending on restrictions set by their developers.

The numbers published by Research2Guidance in October tell the story of 500,000 apps published on the Marketplace. But this gets weird, as I took the number of downloads from those 155,200 apps and it matches the number of downloads published by Google this month. See below.

An Average of 13.63 Billion Downloads

So there have been between 5,514,202,281 and 21,545,335,515 downloads for free apps, making the average 13,529,768,898 downloads.

More interesting however is that according to my data for paid apps, the number of downloads is between 42,576,311 and 164,116,615. This number seems rather low to me, making it clear that Android distribution is freemium based.

UPDATE: initially here there were some charts showing the popularity of free/paid apps per category. I deleted them as I don't like the flames these kind of charts generate!

Android Learning Resources

2011-12-12T00:00:00+02:00

Starting out learning Android development may be intimidating at first, as with any new platform of reasonable complexity you'll have a lot to learn. However the learning process is fun. So here are some learning resources that I'm currently following.

DISCLAIMER: the Amazon links in this article contain my affiliate code and I get a commission should you choose to buy from Amazon in the next 24 hours. However I'm including these links primarily because of the awesome reviews included, but you should buy straight from the publisher (publisher links also included, without an affiliate ID).

Books for Learning Android

My problem with books is that technical books get obsolete really fast. Books from 2010, while still useful, are already insufficient now with the release of Ice Cream Sandwich. The upside is that there are a lot of books out there.

For ~ $40 you can buy a 1-year subscription for CommonsWare.com. The author, Mark Murphy, is a very proficient Android developer, trainer and consultant, with a huge StackOverflow reputation :) More seriously - for $40 you get 3 books that are continuously updated, which is great.

Here are the books (but don't buy them from Amazon, as you won't get the 1-year subscription, which is the main reason I'm recommending these):

Again, don't buy these items from Amazon. Buy them with the 1-year subscription from the Author's Website (I did so myself and btw, I have no affiliation with the author). The subscription is useful because you'll get upgrades for new versions of Android and bug-fixes, for a whole year.

Another book I've been reading is Hello Android by Ed Burnette, published by the Pragmatic Programmers. It's pretty good, but it is more of an introduction (truly a Hello World).

Other books I have not tried, so my list stops here, but updates will follow.

Free Stuff Available Online

I'm the kind of developer that prefers to rely on freely available stuff, because I learn by doing and technical books on APIs are boring. Plus, I like free stuff, however if you plan on getting serious about it, then a small investment is worth it and will keep you focused (nothing will keep you more focused than spending some money).

The official beginner tutorials are not so comprehensive, however you'll get a lot of value from reading the Articles. Lots of value in there, which works best if used in conjunction with the samples provided:

Samples, by Android Developers

Seriously, you won't find a faster learning path than reading and understanding the source-code of real apps. However to not attempt doing this without going through some of the tutorials in the link above.

The videos from Google I/O are a gold mine, providing insight you won't find anywhere else. Highly recommended. All in all, the official documentation is good, although lacking beginner friendliness and a structured clear path. Plus I noticed it has holes in it, but you'll be fine.

For asking questions, or browse around for insightful gems, there's nothing better than:

The Android tag on StackOverflow.com.

At the time of this writing, I don't have many Android-related blogs to share with you and the ones I know about are of poor quality - apparently not many people blog about their Android experience. Hopefully this will change for the better.

Please help me out in identifying other resources! Thanks!

Earning Money as an Amazon Affiliate

2011-11-29T00:00:00+02:00

I published an article that I've meant to publish for a long time. I'm usually lazy to not bother writing many articles, however this time I also thought about doing an experiment - you see I'm (1) on a tight budget and (2) a cheap bastard - so what if I could get enough money to pay for my monthly hosting on Linode, while satisfying my urge to write from time to time?

As such I included Amazon Affiliate links in that post to see what happens.

UPDATE: Moral Considerations Against Amazon's Associates

The discussion on Hacker News took an interesting turn. A couple of comments are a little unbalanced, however there's a comment that I like and with which I am in agreement:

Having affiliate links creates incentives that may not align with faithfully serving your readers. It does not automatically bias your writing, but it can certainly create the appearance of bias.

A concrete example would be writing an especially glowing review of the new Kindle because you have a vested stake in people buying them. Or, perhaps, NOT writing a glowing review because you fear it will be perceived as shilling for affiliate cash.

In fact On The Media recently did a story about the Washington Post struggling with whether or not to include Amazon affiliate links in its book reviews. I think it presents both sides of the argument: www.onthemedia.org/2011/nov/11/web-links...

This comment is striking, as when I started fantasizing about where should I go from here, the one thing that crossed my mind was that I could write a review for the Kindle (I own one) - and oops, unfortunately in that instance my actual opinion would have been unfaithful - @esd nailed it.

Also, a reply puts the above in balance:

Newspapers likely disallow such practices in order to maintain journalistic integrity, but a blog author who is writing posts on purpose to sell things is probably not interested in maintaining journalistic integrity. The blog author is just interested in selling stuff. Maybe the blog posts are well-written and interesting, or maybe they are not. If they are not, then readers who care principally about content will likely avoid the blog on the lack of merit of the content itself.

(the quality of Hacker News still amazes me)

In my oppinion, putting affiliate links is not bad or evil per se - for instance you could say that customers of Apple or Google have an inherent bias because they feel the need to protect their monetary and/or emotional investment. And we aren't professional journalists, trained to watch out for such things - but a mistake is a mistake. So from now on, no more Amazon-related reviews coming from me (maybe I'll try clearly delimited boxes or something).

Some Facts and Stats

The article I'm talking about:

4 Books for Learning to Design, the Hard Way

I published this article on the 25th, The Black Friday - so it had perfect timing. Then I pushed this link to Hacker News and Reddit. I hope you will forgive me, since this was a little self-promotion and I don't deny it, however I hope you found the content therein to be worth it, as it was published with my best intentions.

The resulting traffic and the fee that Amazon gives for the orders generated took me by surprise. Here's how my traffic looks like (unique visitors):

Nov 24: 45
Nov 25: 10,263
Nov 26: 6,939
Nov 27: 1,713
Nov 28: 1,299

Wow. I never generated this much traffic with my blog. Also, here's the Amazon stats (updated for Nov 28th):

Ordered items: 333
Clicks: 6,259
Conversion: 5.32%
Total items shipped: 294
Total earnings: $367.10

So considering that not all items have been shipped and this ain't over yet (I'm still getting traffic), we're looking at:

a half-an-hour effort for a single article, earning $400, in 4 days, on a very low-traffic personal blog

My goal was achieved too - I now have enough money for ~20 months of hosting on Linode. Thank you dear readers, I am in your debt.

Amazon Associates versus Google AdSense

Google AdSense rewards are per-click and is the first option of many webmasters, because it does generate more money. However I feel that the overall user experience suffers a lot - the links served may be context-dependent, but the quality is poor.

Amazon Associates rewards are instead per-action. When the user buys something, you get a referral fee. This can work very well because the items displayed are hand-picked by you and the links add value to your content - whenever I search for reviews of individuals (which I trust more than those of experts), my first stop is on Amazon.

Why A Single Flower Doesn't Bring Spring

So whenever anybody does this successfully, the appetite only grows bigger. After all, this kind of revenue has the potential of being passive. However don't get your hopes up:

in this instance, the timing was perfect and so the circumstances can't be reproduced every month
to have constant conversions you need a big and loyal following that trusts your recommendations
to have a big and loyal following, you also need lots of traffic driven by search engines
to score well in search engines, you need lots of articles with lots of keywords and good ranking
to have loyal readers and a good ranking on Google, those articles must be high-quality
it takes lots and lots of work for the above, probably 1 or 2 years, considering you have the talent of writing content already, or the money to hire people to do it
if you can't recommend anything else other than books, then you're screwed, as you don't have the time to read so many
if the products getting recommended are not relevant to your audience, then they won't convert

So there you have it. My experiment with Amazon's Associates, while probably a short one, has brought joy in my heart :)

Thank you,

4 Books For Learning to Design, The Hard Way

2011-11-25T00:00:00+02:00

This is the path I'm taking to not suck at design anymore, as frankly, I'm getting tired of sucking. I've read the first 3 books here and I'm having progress on the last one. I highly recommend all 4.

(UPDATE - DISCLAIMER: this article contains Amazon affiliate links, as part of an experiment which I'm describing here, however the article expresses my genuine view)

The Design of Everyday Things (link)

It explains the design of common household objects that you may use daily and it is one of the best books you can read on user experience.

To me this was an eye opener that completely changed the way I think about interfaces. It explains how people interact with the objects around them and how they learn. It explains the importance of a user's mental model of how your product works. It gives you a good feeling on what it means and why it matters to design for errors. It helps you to prevent a lot of design errors that a lot of products have.

The book itself was written in 1990, so it does have here and there some references to products that are outdated, but the analysis itself will never be outdated or obsolete. Quite the contrary - it's fascinating how user centric design guidlines stay the same, even though there are a lot of people out there that repeat the same mistakes over and over again.

When it comes to software, a complement article on the subject is User Interface Design For Programmers, by Joel Spolsky.

Non-Designer's Design Book (link)

Robin Williams does an excelent job introducing you to the basic concepts of designing visuals, with clearly explained principles and techniques.

It's pretty hard for us developers to design anything pleasing to the eye. And that's not the only problem you're facing - the visuals of a site have to give hints to the user about their next actions, so you've got many constraints to worry about. Sometimes you get lucky by just copying and combining other designs you like. Sometimes you have a good idea about what you want, but one day you like the result, then the next it looks like an abomination.

The book goes into some detail about how designers think. It has plenty of visual examples, it gives you many examples of what not to do, it explains how to work around those problems, it is concise and doesn't bore you to tears - while not that useful for someone with design experience, for a developer that sucks at this game, this is a really, really good design manual.

Color: A Course in Mastering the Art of Mixing Colors (link)

One of the most surprisingly difficult problems when creating designs is picking the colors palette. This is so goddamn difficult sometimes. We do know that certain color combinations work better than others, but how do you pick them? How can you achieve harmony as to not make your users' eyes bleed? And not only that, but you also want to emphasise certain portions of your pages in a way to attract attention - did you know that according to statistics, more car accidents involve red cars than any other color?

There are web services out there, like Adobe Kuler or COLOURlovers.com which allow you to choose color palettes created by other people - the PROBLEM being that you'll never know what makes a palette work and so you'll make good changes to it only by luck or with expensive A/B testing.

This book by Betty Edwards is not your only choice for learning Color Theory. It isn't even related to web design in any way, having a whole section on mixing oil paint.

However, I believe that color theory can only be learned from people that have real experience in mixing colors. For this reason, articles or books about color theory that aren't written by painters are quite shallow - and the theory in this book transcends the tools used.

It explains notions on color harmony, on the importance of contrast and gives you valuable insight and advices on how to mix and match colors. It's a fun read too, because of the quotes from faimous people contained - however, it's not a light read because you do have to execute the exercises within for best results. But I think it is worth it.

(I'm now my wife's advisor on colors, although our bedroom ended up looking awful, but there's no substitute for experience earned by making mistakes and at least I know where I went wrong ;))

Drawing on the Right Side of the Brain (link)

I've picked up drawing as my hobby. I'm only a beginner, but it's a fun hobby to have - it's impressive, silent and inexpensive. It also boosts your creativity like nothing else, as the only real limit you have is your imagination (much like software development ;))

Ever tried drawing anything lately? You should. The results will be awful. But do you know why? It's not your hand, it's not from a lack of talent, it's your eyes that are deceiving you - in order to learn how to draw, you have to relearn how to see. That's because everything you see is right now filtered and transformed by your brain - as a cheap/fast exercise, look in a mirror at an arm's length and take a guess if your mirrored head is of the same size as your actual head (then use your hands to measure). The picture you're getting through your mind's eye is deceiving and for drawing skill and creativity to emerge you have to silence your brain.

This book of Betty Edwards has an awesome technique that works for everybody. Or so she says, but as far as I'm concerned I'm progressing in leaps and bounds and can already do drawings that I couldn't hope of doing before starting to read this book ... I also promise to publish some drawings, but only after I'll get decent (only started to do this 2 months ago, a frog doesn't transform into a prince overnight you know).

How does drawing help you in web design? Well, do you really have to ask?

Update: Suggestions Received from Readers

The Design of Design, by Frederick P. Brooks
Visual Language for Designers, by Connie Malamed
Interaction of Color, by Josef Albers (review)
Keys to Drawing, by Bert Dodson
Design for Hackers, by David Kadavy
Design and Form, by Johannes Itten
Elements of Color, by Johannes Itten
Principles of Form and Design, by Wucius Wong
Thinking with a Pencil, by Henning Nelms
The Sciences of the Artificial, by Herbert A. Simon
Designerly Ways of Knowing, by Nigel Cross
Designing Visual Interfaces, by Kevin Mullet and Darrell Sano
Grid Systems: Principles of Organizing Type, by Kimberly Elam (presentation)
101 Things I Learned in Architecture School, by Matthew Frederick

Thanks people, wow, I know have my hands full :) ... keep `em coming!

How I Use Flickr: For Backup

2011-10-29T00:00:00+03:00

I've got a growing number of personal pictures and the collection is growing since 2003, when I got my first digital camera, a shitty Sanyo that still works and that I still use whenever I forget about my Nikon.

But here's the thing with digital pictures - they are cheap to make, but also easy to lose. Digital storage is not as reliable as glossy paper. Pictures printed on paper can easily last for a 100 years. That's not the case with any digital storage medium and we will suffer for it.

Storing My Pictures In The Cloud

Pro accounts on Flickr have unlimited storage and can upload and access full-resolution pictures. This is great, although be careful about believing in "unlimited plans", as nothing is really unlimited and by abusing Flickr you may find yourself locked out of your account.

Unfortunately the tools for uploading really suck and I haven't encountered yet a graphical interface that did what I needed. So for synchronizing, I've built my own script in Ruby using the excelent Flickraw gem and exifr, another Ruby gem that reads Exif headers from Jpeg files.

One common problem is that you ALWAYS have duplicates. And you don't want to upload duplicates. What you really want is an "rsync" command for Flickr. But how do you know if a picture was already uploaded?

The approach I'm using is to add a machine tag to my pictures, which is set like a tag, but has the format "namespace:key=value". This machine tag represents the MD5 hash of the picture and if you want to see if a certain photo was already uploaded to flickr, you can always search for it. Here's how it looks on one of my pictures:

checksum:md5=5b2fa91c38a7f878088e1420b924e6d9

Besides this, I have this problem with some of the older photos taken by my Sanyo, where the taken-date is totally fucked and for personal photos the taken date is maybe more important than the actual photo quality. I use the excelent ExifTool to correct those photos. It's nice building on the hard work of other people ;)

So, currently I have 3545 pictures uploaded on Flickr in full resolution and the number will more than triple as soon as I make an inventory of my pictures stored on old hardware I've got lying around.

It is fun being a developer. I can make shit happen.

Flickr is Not A Reliable Backup

Flickr is an online service that isn't meant for being a backup. I share only a fraction of what I upload, everything else is family only. They may terminate your account at any time for whatever reason. They may also go out of business. Yahoo may sell it, etc, etc... I do think Flickr is awesome btw and one reason that I store my photos on Flickr is to be able to always have the whole archive with me. But for backup alone, that's not enough.

What you really need to do is:

your main repository should be stored locally and properly maintained - I do that on my main computer currently, but multi-TB external hard-drives are cheap
in case of cloud backup, you always need a secondary service for redundancy

Google's Picasa is a good option because you can explicitly buy storage. This means that if you pay for 80 GB of storage nobody is going to get upset that you uploaded 80 GB of private photos ... also, if your photo collection matters to you, I wouldn't put my trust in their Google+ offering (photos of up to 2048x2048 pixels do not count towards your free quota). That's because that offer is not meant for you. Just pay up.

So on Google's Picasa, I'm currently working on integrating with their API too. The desktop app is nice, but too limited for me.

Testing Different Browsers: It`s a Pain in the Ass

2011-10-25T00:00:00+03:00

I got a notice that my crossdomain-requests-js script (described in length here) does not work on IExplorer 9.

God, how much I hate dealing with browsers. Initially when I wrote that script I tested on IExplorer 6 and IExplorer 8. I already had to deal with issues regarding the IExplorer 8 compatibility mode. I already had taken the decision to completely ignore IExplorer 7, as that was a fucked up release. Either way, it is possible that I completely wrecked IExplorer support with later changes that I failed to retest.

And all of this for a 233 lines script (including comments and whitespace). How fucked up is that?

This is also why I love JQuery so much. Unfortunately if you want to publish a reusable library it's extremely annoying to bring a dependency such as JQuery with it. Even though JQuery is popular, the latest release is a whooping 90K in size. Do you know how painful that is on mobile browsers? NO, infrastructure code has to be done for the lowest common denominator: that's why people still do complex shit in C. C is portable and you can link to C from any other library.

Either way, I might do a rewrite in CoffeeScript: at least that's going to save me the pain of dealing with syntax differences, as I've been having those kinds of problems too ;-)

So what to do?

prefer CoffeeScript for development, not because it is cool but because it is sane
Microsoft was kind enough to release virtual-machine images for testing your websites in various IExplorer versions - I'm installing them in VirtualBox right now using this script here: https://github.com/xdissent/ievms (many thanks to the author, I'll let you know how it goes)
Always test with the 6 biggies: Firefox, Chrome, Opera, IExplorer 6, IExplorer 8 (watch out for the compatibility mode) and IExplorer 9
There are few differences between Chrome and Safari, both being based on WebKit and V8 is pretty compatible with Safari's JS Engine, so I don't bother with it. But it doesn't hurt if you have it around.
Mobile Safari and Android's browser are different from desktop Safari and Chrome. Do not assume that your shit will automatically work on mobiles.

Do me a small favor ...

Open this page: http://bionicspirit.com/projects/crossdomain-requests-js/

Then tell me if it works or not in your browser, including the browser name and version you are using. You can use the comments section below, or in this opened issue on GitHub.

Why I Find Heroku Suboptimal

2011-10-23T00:00:00+03:00

I love freebies. I often find myself compelled to search for the best price / convenience ratio, and from this perspective you cannot really argue against something offered for free. And yet, here I am bitching and moaning about Heroku.

Heroku provides a free-quota that's a LOT more reasonable than all the shitty PHP hosting offerings out there. And when time comes to scale, it lets you scale nicely for a price.

Normally you develop your app on your localhost (which is like this warm and cozy place for all developers, no place like 127.0.0.1 and all that), but then you want to deploy. You have to get out of your comfort zone and face the jungle and it's a true jungle out there, filled with shitty / underpowered and expensive hosting offerings. If going for a normal VPS, you'll have to configure your application server, your database server, your webserver that sits on top, maybe a reverse proxy cache, a memcached instance or two, a load balancer, a firewall, an email server and it goes on and on. And if going for a classic shared-hosting environment, then God help you.

There's a reason children with happy childhoods don't want to grow up - the world is an ugly and scary place.

git push heroku master

Heroku is great. It basically allows you to avoid growing-up. The deployment itself couldn't be simpler, and when browsing their web interface for available add-ons, I feel like a child in a candy-store.

Basically you start with a free worker, to which you can add other "free" services, like a 5MB PostgreSql database and a 5MB Memcached instance, allowing you to prototype stuff. They even have plugins from third-parties that give you freebies, like a 250MB CouchDB, or a 240MB MongoDB. Then as you grow, you start adding more and more resources as needed. This has been labeled as platform as a service and it's what the cool kids are talking about these days. Heck, there are people that are living within that free-quota without problems. One such example that I know of is http://tzigla.com ... or it was last time I talked to the authors, both acquaintances of mine, and Cristi described how he ended-up doing lots of workarounds to get around limitations and he was really excited about how everything fell into place.

But as I was sitting there admiring their determination and skill, I started wondering why the hell haven't they rented a normal VPS?

I mean really, if you end up pulling all kinds of crap to get around limitations, wouldn't it be better to just pay up? And if you're short on cash or you're the kind of entrepreneur that likes to spend frugally, then wouldn't you be better just renting a normal VPS? I asked him just that of course, and his reply was basically:

I hate to do sys-admin stuff, installing and upgrading packages and all that

But it doesn't have to be that way. It's really not that hard. The reason for these feelings is the Ubuntu I have had installed on my primary laptop for 5 years already. Once you work with Ubuntu or your favorite Linux distribution, every day, configuring a web-server for starters is something like a half-an-hour chore. Or let's say 1 hour, and then it's done. And you don't have to worry about it again.

And there are disadvantages to Heroku, lots of them: that's because you lose control and end up on top of a platform that's designed as a common denominator to appeal to all needs in an equally substandard manner.

Example 1: Nginx

Nginx is a freakishly fast web server that consumes really few resources. Its main appeal is in serving static files and you do have static files to serve. When you grow you may want to move those static files to a CDN, like CloudFront, which serves content from locations closer to the actual users, but for serving css/javascript and small images - a properly configured Nginx is all you need. And you can't really move any files served from your main domain to a CDN (like HTML content).

You can also be smart about semi-static pages in Rails - you can cache the output inside the public/ directory to be served by Nginx. And if you still want to hit your controller on every request, like when doing A/B Testing on a page, you can send an X-Accel-Redirect header in your response to Nginx and let Nginx to the actual content streaming for you. You can also instruct Nginx to serve files from different locations, based on certain variables like the domain name, thus avoiding hitting the Rails application server on every request.

There's a lot you can do with Nginx if you're on a budget, and yet this is not possible within Heroku ... which even though it may use Nginx as an http reverse proxy, it certainly doesn't use it for serving static files. All files are thus served by hitting the Rails server, unless Varnish is involved.

Example 2: Varnish

Varnish is described as being a web application accelerator and the things it can do are truly mind-blowing.

Varnish sits in front of your application servers. It can do load-balancing for you with extreme efficiency, although that's not its main purpose. Its main purpose is to cache content.

When caching content you have an extreme freedom to specify the Key for fetching cached items. You can use anything when instructing Varnish on what and how to cache, like cookies or the user's IP or any HTTP header. Do you want to also cache content for logged-in users, even though that content is slightly different from user to user? Not a problem. The configuration language is also extremely flexible, allowing you to tap in the request pipeline with any custom behavior you want. The performance of Varnish coupled with this extreme flexibility is what makes it great. It also has this uncanny ability to reload its configuration without restarting or dropping active connections.

Heroku has Varnish in its stable stack, called Bamboo. But you cannot configure it. The configuration is the same for everybody ... you basically set expiry headers on your response, Varnish caches it for you and the cache gets invalidated on every new deployment.

This is actually good and has given rise to the famous Heroku use-case: hosting mostly static websites on it. But Varnish can be much more than that, otherwise it kind of gets in your way, and surprise - Heroku is pulling Varnish out of the configuration, starting with the new Celadon Cedar stack. This is because Varnish gets in the way of their ambitious plans: to make heroku platform-agnostic, thus adding support for Node.js and long-pooling.

The now recommended alternative for serving cached static content is to use Rack::Cache in combination with their Memcached add-on. But this sucks because (1) it hits the Rails server on every request and in the free plan you only have a single process to serve those requests + (2) the free plan for Memcached is only 5MB.

Example 3: asynchronous jobs

One common-sense approach to not having a sluggish web interface is to get slow code out of your HTTP process. Lots of libraries and plugins are available for all web frameworks, like delayed_job for Rails or Celery for Django. And you can just write your own half-baked jobs queue and shove it in your cron.

You cannot have asynchronous jobs using Heroku's free plan. You must get an extra dyno for that.

Price comparison with Linode

The cheapest Linode instance is $20 per month, and for starters you can have ...

1 Nginx server
2 Passenger/Rails processes
1 worker for processing asynchronous jobs, it can even be a plain cron-job ; you do have complete flexibility in configuring cron-jobs
1 PostgreSQL database, configured for 256MB RAM usage, with 18 GB of storage. It's not much, but it isn't shared either and does just fine, trust me ... btw, the PostgreSQL magazine (first issue) has an article about configuring/optimizing PostgreSQL's memory usage
1 Postfix email server, for bug reports + sending all the spam you want (Linode lets you configure reverse DNS lookup, so you can have a cheap email server that doesn't trigger spam alerts)
ability to serve for any domain you want, including wildcard subdomains
your own SSL certificate, for free depending on provider

The equivalent Heroku configuration would cost a minimum of $114 per month.

So lets say that you're growing and you want to add Ronin, Heroku's plan for a database of 1.7 GB hot data set (whatever the fuck that means). That will cost you a whooping $200 per month extra, versus $80 for a 2GB of RAM instance on Linode, or even better, $160 for a 4GB of RAM instance.

Linode sucks too, but that's besides the point

You lose the ability to increase your dynos in response to traffic surges. On the other hand you'll be amazed at how much you can squeeze out of your rented hardware and if a properly configured setup fails to serve, then the problems you have probably can't be solved by just adding extra web servers.

Really, do some reading on why Reddit is down so often. Do some reading on why Amazon's EBS is completely unreliable for databases (btw, Heroku does use EBS and they've also had their share of downtime due to AWS experiencing problems).

Stop fearing the penguin and start configuring your own damn servers. As with everything that's actually worth it in life (like having children of your own), it's hard at first but the return of investment will be tenfold.

PS: I'm obviously advertising Linode here. Links to it contain my affiliate tracking code, and if you become a customer you'll give me $20 worth of credit, which helps me pay for this blog's hosting (what can I say, I'm a cheap bastard). On the other hand this does express my genuine view of these services.

Cross-Domain, Cross-Browser AJAX Requests

2011-03-24T00:00:00+02:00

This article describes how to make cross-browser requests, in all browsers (including IExplorer 6), without using a proxy or JSONP (which is limited and awkward) -- as long as you control the destination server, or if the destination server allows.

I'm explaining this file: crossdomain-ajax.js
Simple usage example: http://bionicspirit.com/projects/crossdomain-requests-js/

For a more serious example that works, checkout the Comments section getting loaded at the bottom of this page.

UPDATED Oct 27, 2011

Added restrictions of usage and removed functionality that doesn't work on IExplorer. So in case this doesn't work for you, please see this page: Troubleshooting

In Modern Browsers - Meet Cross-Origin Resource Sharing

Or CORS for short, or HTTP Access Control, available in recent browsers, allows you to make cross-domain HTTP requests; the only requirement being that you have must have control over the server-side implementation of the domain targeted in your XMLHttpRequest calls.

This little piece of technology is available since Firefox 3.5 / IExplorer 8 and yet when searching for answers on websites like StackOverflow, it rarely comes up.

For the purposes of this tutorial, we'll assume we want to make a request from website http://source.com to http://destination.org, and that you control the implementation to both.

var xhr = new XMLHttpRequest();
// NOPE, it doesn't work, yet
xhr.open("POST", "http://destination.org", true);

Response of destination.org

It's pretty simple really, all you need to do is to return these headers in your response:

Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://source.com
Access-Control-Allow-Headers: Content-Type, *

You can find a description of them on Mozilla Doc Center, but the most important one is Access-Control-Allow-Origin, which indicates the Origin(s) allowed to make such a request.

Note: these options allow for wildcards (like you can say that you allow for any Origin by putting a "*" in that header), but it is better to be explicit about what's allowed, otherwise your request won't work very well cross-browser.

(New) Note: In regards to Access-Control-Allow-Origin, IExplorer DOES NOT support wildcards. See Troubleshooting for details.

Client-side Implementation of Ajax Request for CORS

On browsers where XMLHttpRequest is valid, support for CORS can be validated by checking for the availability of the withCredentials property.

So we've got a tiny issue: IExplorer's implementation is different than that of Firefox's or the rest of the browsers (naturally). Instead of using the same XMLHttpRequest object, IExplorer 8 adds an XDomainRequest object.

So to initialize an async request, that will work on IExplorer 8, Firefox, Chrome and the other browsers supporting it:

try {
    var xhr = new XMLHttpRequest();
} catch(e) {}	

if (xhr && "withCredentials" in xhr){
    xhr.open(type, url, true);
} else if (typeof XDomainRequest != "undefined"){
    xhr = new XDomainRequest();
    xhr.open(type, url);
}
else
    xhr = null;

But we aren't done yet, the callbacks used by these request objects have different behavior on IExplorer. So let's say we've got 2 callbacks that we want to register, one for success, one for errors, having the following signatures (same as jQuery):

function success(responseText, XHRobj) { ... }
function error(XHRobj) { ... }

To have correct behavior cross-browser:

// 
// combines the success/error handlers into one 
// higher-order function (getting a little fancy for code-reuse)
//
var handle_load = function (event_type) {
    return function (XHRobj) {
        //
        // stupid IExplorer won't receive any param on callbacks!!!
        // thus the object used is the initial `xhr` object
        // (bound to this function because it's a closure)
        //
        var XHRobj = is_iexplorer() ? xhr : XHRobj;

        //
        // IExplorer also skips on readyState
        // Also, it's success/error based on the `event_type` used at the call-site
        // 
        if (event_type == 'load' && (is_iexplorer() || XHRobj.readyState == 4) && success)
            success(XHRobj.responseText, XHRobj);
        else if (error)
            error(XHRobj);
    }
};

try {
    // IExplorer throws an exception on this one
    //
    // Setting this to `true` is specifying to make the request with Cookies attached.
    // BUT -- it's pretty useless, as IExplorer doesn't support sending Cookies.
    //
    // Also, trying to set cookies from the response is not really possible directly 
    // (workarounds are available though -- you can return anything in the response's 
    //  body and use local javascript for persistence/propagation on next request)
    //
    xhr.withCredentials = false;
} catch(e) {};

//
// `onload` + `onerror` are actually new additions to these browsers.
//
// IExplorer doesn't actually push params on calling these callbacks.
// For every other browser, the XHRobj we want is in `e.target`, 
// where `e` is an event object.
//
xhr.onload  = function (e) { handle_load('load')(is_iexplorer() ? e : e.target) };
xhr.onerror = function (e) { handle_load('error')(is_iexplorer() ? e : e.target) };
xhr.send(data);

Also of notice, here's how to check if the browser is IExplorer:

function is_iexplorer() { 
    return navigator.userAgent.indexOf('MSIE') !=-1 
}

Well, that's it, unless you want to support the rest of desktop browsers in use.

Fallback for Older Browsers

Opera 10 doesn't have this feature, neither do IExplorer < 8, Firefox < 3.5 -- and I don't really know when Chrome/Safari added it. Fortunately there's a workaround -- Flash can do whatever you want and runs the same on ~90% of desktop browsers out there, AND it can interact with Javascript.

Not to reinvent the wheel, here's a cool plugin: flensed.flXHR.

Why bother with CORS?

Flash is not available on the iPhone
Flash loads slower than Javascript
Flash SWF files come with a lot of junk that your browser has to download
The whole experience using flXHR will be visibly slower than with CORS

flXHR Usage

//
// Does a request using flXHR (the JS-Flash alternative 
// implementation for XMLHttpRequest)
//
function _ajax_with_flxhr(options) {
    var url     = options['url'];
    var type    = options['type'] || 'GET';
    var success = options['success'];
    var error   = options['error'];
    var data    = options['data'];

    //
    // handles callbacks, just as above
    //
    function handle_load(XHRobj) {
        if (XHRobj.readyState == 4) {
            if (XHRobj.status == 200 && success)
                success(XHRobj.responseText, XHRobj);
            else
                error(XHRobj);
        }
    }

    var flproxy = new flensed.flXHR({ 
        autoUpdatePlayer:false, 
        instanceId:"myproxy1", 
        xmlResponseText:false, 
        onreadystatechange:handle_load
    });

    flproxy.open(type, url, true);
    flproxy.send(data);
}

We are NOT done. The destination server also needs a file called crossdomain.xml, which represents a Crossdomain Policy File Spec. As a requirement, this file has to be placed in the domain's root, i.e. http://destination.org/crossdomain.xml ...

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
  <!-- wildcard means 'allow all' -->
  <allow-access-from domain="*" />
  <allow-http-request-headers-from domain="*" headers="*" />
</cross-domain-policy>

Not Loading the Junk when Not Needed

Javascript is asynchronous and we should take advantage of that by not loading flensed.flXHR, unless needed and at the last moment too (no need to load it until we want to make a request).

We need a method for asynchronously loading a Javascript file and executing a callback onload. And since we may be executing this function multiple times at once, we need to take care of race-conditions. First things first:

// keeps count of files already included

var FILES_INCLUDED = {};

// keeps count of files in the processes of getting loaded
// for avoiding race conditions 

var FILES_LOADING  = {}; 

// stacks of registered callbacks, that will get executed once
// a file loads -- this to deal with multiple file inclusions at once,
// and not ignoring anything

var REGISTERED_CALLBACKS = {};

function register_callback(file, callback) {
    if(!REGISTERED_CALLBACKS[file])
        REGISTERED_CALLBACKS[file] = new Array();
    REGISTERED_CALLBACKS[file].push(callback);
}

function execute_callbacks(file) {
    while (REGISTERED_CALLBACKS[file].length > 0) {
        var callback = REGISTERED_CALLBACKS[file].pop();
        if (callback) callback();
    }
}

To asynchronously load a Javascript file, with onload callback, behold:

//
// Loads a Javascript file asynchronously, executing a `callback`
// if/when file gets loaded.
//
// Returns `true` if callback got executed immediately, `false` otherwise.
//
function async_load_javascript(file, callback) {
    // stores callback in the stack
    register_callback(file, callback);

    // dealing with race conditions

    if (FILES_INCLUDED[file]) {
        execute_callbacks(file);
        return true;
    }
    if (FILES_LOADING[file]) 
        return false;

    FILES_LOADING[file] = true;

    // dynamically adds a <script> tag to the document
    var html_doc = document.getElementsByTagName('head')[0];
    js = document.createElement('script');
    js.setAttribute('type', 'text/javascript');
    js.setAttribute('src', file);
    html_doc.appendChild(js);

    // onload, then go through the stack of callbacks, 
    // and execute all of them
    js.onreadystatechange = function () {
        if (js.readyState == 'complete' || js.readyState == 'loaded') {
            if (! FILES_INCLUDED[file]) {
                FILES_INCLUDED[file] = true;
                execute_callbacks(file);
            }
        }
    };

    // same as above, same shit for dealing with incompatibilities
    js.onload = function () {
        if (! FILES_INCLUDED[file]) {
            FILES_INCLUDED[file] = true;
            execute_callbacks(file);
        }
    };

    return false;
}

Almost there

To bind it all together we need to plug this into our main logic. So if browser does not support CORS, it fallbacks to this implementation.

// to recapitulate
if (xhr && "withCredentials" in xhr){
    xhr.open(type, url, true);
} else if (typeof XDomainRequest != "undefined"){
    xhr = new XDomainRequest();
    xhr.open(type, url);
}
else
    xhr = null;

// NOT SUPPORTED, then fallback
if (!xhr) 
    async_load_javascript(CROSSDOMAINJS_PATH + "flXHR/flXHR.js", function () {
        _ajax_with_flxhr(options);
    });

To see the final code, go here: crossdomain-ajax.js. Or to see it working, go here: bionicspirit.com/projects/crossdomain-requests-js/.
(or just leave me a comment below ;))