Whether you're a one man team with a shoestring budget, or a financed startup with some runway, managing the feature backlog is more of an art than a science – it's always full of vaguely formulated ideas and it's constantly changing. Managing these priorities is as hard as it is, and if you have a team, it gets even harder: how do you make sure that everyone is aligned with the vision and the roadmap?

Managing the backlog: countdown to zero

A great story from the startup folklore: after closing a round of financing, the CEO brought in a big jar of marbles representing the amount of money they had in the bank and proposed that each time a project manager or a developer pitched an idea, they would have to take some marbles out of the jar (depending on the size of the project). Nobody opposed it, but a few odd looks were exchanged. Then six months went by, the jar was now half-full and the dynamics started to change - everyone became much more conscious of the amount left in the jar. They could now see the inevitable: the jar was not being filled up nearly as fast as it was being depleted.

The problem is, I think we all fall into this trap more often than not. What has changed in the fundamental goals of the company in the six months since the jar was initially full? Nothing, of course. The strategy may have shifted, the product roadmap has evolved, but the economics of running a company remained constant. Six months ago, the day that the company would cease to exist was 12 months away, 6 months later, it was simply 6 months closer. Here's an acid test: if your project would cease to exist tomorrow, would you work on the feature you're contemplating doing today?

The luxury of time

Just because the deadline is further away does not buy you the luxury of indulging into technological desires. Unfortunately, as developers we tend to forget this, as we’re too easily distracted by the shiny technical aspects of the next greatest thing, or over optimizing for the day that never comes.

Stop and examine what you are working on now: does it directly contribute to the success of your project today, or is it a nice to have? If you had a deadline in the immediate feature, would you still be working on it? If not, then you probably shouldn’t be doing it.

Time & Strategy: use them to your advantage

Does that mean we should always focus on the short term goals? Not at all. The luxury of time is exactly that: a luxury and it should be treated as such. If you waste it, you can’t get it back. But having this luxury can also buy you a strategic advantage: you can trade short term goals for a higher return in the long term.

Sometimes you have to go slow to go faster. It's the lack of that regression test suite that's coming back to bite you, or perhaps a monitoring system that will save you the maintenance time – the examples are endless. The challenge is, of course, to weight the benefits of these long term goals against the possible value of shorter and quicker releases: “Great companies ship.” You have to be able to recognize that a technologically inferior solution delivered quickly can often outweigh a ‘cleaner’ solution – the real trick is, of course, to also have your team of gearheads understand why this is true.

Time can buy you the luxury of strategic objectives, but remember that today is no different than the day you’ll be taking the last pebble out of that jar, it just happens to be a few days into the future.

Help the Ruby interpreter

This is no secret: Ruby interpreter is slow. Advances on the JRuby and Maglev front are certainly showing some very appealing performance increases, but even there, a deeper understanding of the Ruby internals is bound to help you write better and faster code. The Ruby Wikipedia entry is a great place to start to learn about some of intricacies of Ruby. For a deeper look, Dr. Stefan Kaes's 'Writing Efficient Ruby Code', and David A. Blacks' 'Ruby for Rails' are must have for any Ruby developer.

1. Minimize searches in the abstract syntax tree

Ruby has a wonderful property of being highly dynamic, which in turn, allows us to create all kinds of spectacular meta-programming scenarios. However, this comes at a price of minimal runtime and compile time optimization (JRuby and some other VM's are changing this). Unlike most other languages, Ruby's MRI does not generate bytecode (Ruby 1.9 will change this), but relies solely on searching through the Abstract Syntax Tree (AST) for virtually every method call, variable, and so on. Sounds expensive? It is. Hence, next time you're saving yourself three lines of code by writing a meta function, consider inlining the logic directly.

2. Optimize for Ruby cache, avoid expensive lookups

To optimize against expensive searches through the AST, Ruby keeps a cache of local variables and method signatures. Hence, comparatively speaking, local variables are cheap - use them:

 
@var = "local variable, which is cached by Ruby, and requires a single lookup"
self.var = "requires walking the AST, and results in multiple lookups"
 

3. Interpolation over Concatenation

This came as a surprise to me the first time I stumbled across it: interpolated strings are faster than straight concatenation! Chris Blackburn recently wrote a great blog post about it, make sure to read it.

 
puts "This string embeds #{var1} and #{var2} through interpolation"  # faster
puts "This string concatenates " << var1 << " and " << var 2  # slower
 

4. When possible, use destructive operations!

Many ruby operations have a destructive (ex: gsub vs gsub!) equivalent, and which, unfortunately are not used very often largely due to the somewhat erratic behavior: sometimes these operations return nil, sometimes they don't. However, non destructive operations (ex: gsub) duplicate your object and hence incur an expensive copy operation every time. Learn the non-nil, destructive operations and use them religiously.

 
hash = {}
hash = hash.merge({1 => 2}) # duplicates the original hash
hash.merge!({1 => 2}) # equivalent to previous line, and faster
 
str = "string to gsub"
str = str.gsub(/to/, 'copy') # duplicate string and reassigns it
str.gsub!(/to/, 'copy') # same effect, but no object duplication
 

5. Symbol.to_proc fan? Use blocks!

If you're a Rails developer, you've probably used Symbol.to_proc. Well, you're likely incurring an order of magnitude speed decrease when you do! Next time, use a block:

 
@widget_ids = @widgets.map(&:id) # Symbol.to_proc method, order of magniture slower...
@widget_ids = @widgets.inject([]) {|w, a| w.push(a.id)} # same effect, not as pretty, but much faster
@widget_ids = @widgets.collect {|w| w.id } # faster, and simpler than inject
@widget_ids = @widgets.map {|w| w.id } # yet another (faster) way to tackle the problem
 

6. Benchmark everything!

Create a macro, stash it away in an easy to access area, or post the code for the benchmark skeleton on your wall. Anytime I have a question about Ruby performance, the answer is always less than 30 seconds away:

 
require 'benchmark'
 
n = 100000
Benchmark.bm do |x|
   x.report('copy') { n.times do ; h = {}; h = h.merge({1 => 2}); end }
   x.report('no copy') { n.times do ; h = {}; h.merge!({1 => 2}); end }
end
 
#          user        system      total        real
# copy     0.460000   0.180000   0.640000 (  0.640692)
# no copy  0.340000   0.120000   0.460000 (  0.463339)
 

Your favorite optimization tip missing from the list? Leave a comment, let everyone know!

Hence, when I recently stumbled on Splunk, I knew I had to try it - it, looked, awesome. The setup was a breeze and I was up and running in no time - no more fgrep / egrep for me! If you want to give it a try, take a look at Michael Wilde's video (Splunkin' with Amazon EC2), which he posted just a few days ago. Michael provides a great walkthrough, but I think he overlooked an awesome feature of Splunk, which is especially useful for the EC2 environment, and one we're planning to make heavy use of at AideRSS: distributed logging over TCP!

Managing Distributed Logs from the Cloud

Managing logs is hard as it is, and now imagine you have several (dozen) servers in EC2, the process becomes a chore, the debugging is hard, and frustration abounds. To address this problem, I've setup Splunk to listen on a TCP port for any network traffic - if all others servers log to this host, then you will have a centralized, indexed log repository for all of your services. A simple Ruby-based NetworkLogger class later, and we're in business:

> networklogger.rb

# igvita.com (Ilya Grigorik)
require 'rubygems'
require 'socket'
 
class NetworkLogger
  def initialize(host = 'localhost', port = '8888')
    @socket = TCPSocket.new(host, port)
  end
 
  def log(level, msg)
    @socket.write "#{Time.new.strftime("%H:%M:%S")} (#{level}): #{msg}\\n"
  end
 
  def method_missing(method, *args)
    if method.to_s =~ /(info|debug|warn|error)/i
      return self.log(method.to_s, args[0])
    end
  end
end
 
logger = NetworkLogger.new('localhost', 9998)
 
# Send log messages to remote Splunk server 
logger.info "This is in INFO message"
logger.debug "This is a DEBUG message"
logger.warn "This is a WARNING message"
logger.error "Uh oh! This is an ERROR message"

networklogger.rb (NetworkLogger source)

Downloads: 222 File Size: 0.7 KB

It's a simple example, but it highlights the possibilities - you can stream any set of logs to a centralized server for easy debugging, live monitoring, and alerts and notifications. Best of all, it's a free solution as long as you stay below 500mb of indexed log data, per day.

Update: Michael Wilde posted posted a video reply with a demo of what he considers the coolest feature of Splunk to be: transactions. I didn't know about, and have to agree, thanks Michael!

However, having Tumblr to save the quotes is nice, but I rarely (if ever) went back to read them. Pondering this dilemma, I spotted my Phillips digital photo frame, and a quick weekend project was born: Tumblr API, RMagick to render the images, and an SD card full of wisdom and quotes for easy consumption!

Retrieving quotes from Tumblr API

Retrieving your content from Tumblr is about as easy as it can get: one GET request, with a few query parameters. In return you get an XML file with the quotes, the source link, publication time, and much more. To parse the XML response, HPricot comes to the rescue:

> tumblr-api.rb

require 'open-uri'
require 'hpricot'
require 'cgi'
 
# read last 5 quotes
doc = Hpricot.XML(open("http://username.tumblr.com/api/read?type=quote&num=5"))
 
# iterate and clean up (unescape HTML, and remove HTML entities)
(doc/'post').each do |post|
    text = CGI::unescapeHTML((post/'quote-text').inner_html).gsub(/&[^;]*;?/,'')
    source = CGI::unescapeHTML((post/'quote-source').inner_html).gsub(/\\<.*?\\>/, '').gsub(/&[^;]*;?/,'').strip
 
    puts "Processing: #{post['id']}\\n"
end
 

Rendering wisdom with RMagick

Once the data is cleaned up, we can get to the rendering step. Installing RMagick can be a dreadful experience, but it appears that as of 2.x release, this process has been vastly improved - everything worked on the first try!

To render the final image, I've decided to adopt the simplest possible route: render each component as a separate image (quote text, background, source link, etc), and then merge the images as layers into a single file. Easier said than done! RMagick is not for the faint of heart, and to my surprise, did not have the concept of a text box (it can only render contiguous lines), unless you use the annotate method:

> rmagick-quote.rb

require 'rmagick'
include Magick
 
text = "A goal is a dream with deadline"
 
# courtesy of http://blog.choonkeat.com/weblog/2007/02/rmagick-renderi.html
# - render the text in a 500x430 box, and truncate if needed
quote = render_cropped_text(text, 500, 430) do |img|
   img.fill = "#ffffff" # font color
   img.background_color = "transparent"
   img.pointsize = 23
   img.antialias = true
   img.font = "Helvetica"
end
 
quote.write "text.png"
 

tumblr.zip (full implementation)

Downloads: 85 File Size: 135.8 KB

It's not optimized, but it got the job done: ~160 quotes on an SD card, rotating at random intervals for my enjoyment. Next iteration, automated updates of quotes on my photo frame via RSS and WiFi - any volunteers to write a Sinatra app? I can't wait until D-Link DSM-210 hits the streets!

If you're looking for Rails coverage, follow me on Twitter (my laptop decided to take a nap for a few days, hence broadcasting from my Blackberry). For more twitter news, make sure to also check out the summize stream.

A few topics that generated a lot of talk at RailsConf yesterday:

• MagLev is Gemstone/S for Ruby, Huge News
• Twitter response to MagLev presentation
• IronRuby and Rails - watch for Silverline, very cool project (ARAX).
• Drew Blas has great coverage of the conference - keep it up!
• DHH keynote @ RailsConf '08

Last but not least, if you're at RailsConf and spot me in the crowd, come say hello! Talks I'm looking forward to attending today:

• Assembling Pages Last: Edge Cache, ESI & Rails - Aaron Batalion
• Asynchronous Processing with Ruby on Rails - Jonathan Dahi
• Quick and Easy Custom Servers using EventMachine - Soctt Fleckenstein
• Custom Nginx Modules: Accelerate Rails, HTTP tricks - Adam Wiggins
• The Great Test Framework Dance-off - John Susser

The reactor design pattern is a concurrent programming pattern for handling service requests delivered concurrently to a service handler by one or more inputs. The service handler then demultiplexes the incoming requests and dispatches them synchronously to the associated request handlers.

Why Reactor to begin with?

Steeped in the tradition of forking / threaded web-servers I found myself rather surprised when I joined one of the research projects at University of Waterloo a couple of years back: we were benchmarking different web-server architectures, and top performers were all event-driven servers.

As I pestered everyone with questions, I quickly realized why - in an environment with hundreds of thousands requests a second, forking and context switching associated with thread management become prohibitively expensive (fork is worst performer, as it does a memory copy on the parent process every time). Whereas by comparison, a tight and highly optimized event-loop really shines when it comes to performance under heavy loads.

EventMachine and Reactor pattern

Listening to a couple of recent presentations on different Ruby application server alternatives, I've come across a consistent comment: "Evented servers are really good for very light requests, but if you have a long-running request, it falls down on its face." Technically, valid, but in practice, not necessarily true. Let's start with the simplest example:

> em-http.rb

require 'rubygems'
require 'eventmachine'
require 'evma_httpserver'
 
class Handler  < EventMachine::Connection
  include EventMachine::HttpServer
 
  def process_http_request
    resp = EventMachine::DelegatedHttpResponse.new( self )
 
    sleep 2 # Simulate a long running request
 
    resp.status = 200
    resp.content = "Hello World!"
    resp.send_response
  end
end
 
EventMachine::run {
  EventMachine.epoll
  EventMachine::start_server("0.0.0.0", 8080, Handler)
  puts "Listening..."
}
 
# Benchmarking results:
#
# > ab -c 5 -n 10 "http://127.0.0.1:8080/"
# > Concurrency Level:      5
# > Time taken for tests:   20.6246 seconds
# > Complete requests:      10
 
 

Here we've built the simplest possible HTTP web-server using EventMachine. To test it, we ran ab (Apache Bench), and set the concurrency to 5 (-c 5), and number of requests to 10 (-n 10). Time to process is ~20 seconds, which makes sense, because as expected the Reactor is processing each request in synchronous fashion, effectively overriding our concurrency setting to 1. Hence, 10 requests, 2 seconds each, and the math works out!

EventMachine: Reactor with lightweight concurrency?

The synchronous nature of the Reactor pattern is the bottleneck in the previous example, and that's where EventMachine deviates from the purist pattern. Specifically, it provides a mechanism by which it is also possible to dispatch the request to a pool of green Ruby threads (20 by default):

> em-http-pool.rb

require 'rubygems'
require 'eventmachine'
require 'evma_httpserver'
 
class Handler  < EventMachine::Connection
  include EventMachine::HttpServer
 
  def process_http_request
    resp = EventMachine::DelegatedHttpResponse.new( self )
 
    # Block which fulfills the request
    operation = proc do
	sleep 2 # simulate a long running request
 
        resp.status = 200
        resp.content = "Hello World!"
    end
 
    # Callback block to execute once the request is fulfilled
    callback = proc do |res|
    	resp.send_response
    end
 
    # Let the thread pool (20 Ruby threads) handle request
    EM.defer(operation, callback)
  end
end
 
EventMachine::run {
  EventMachine.epoll
  EventMachine::start_server("0.0.0.0", 8081, Handler)
  puts "Listening..."
}
 
# Benchmarking results:
#
# > ab -c 5 -n 10 "http://127.0.0.1:8081/"
# > Concurrency Level:      5
# > Time taken for tests:   4.21405 seconds
# > Complete requests:      10
 

Once again, 10 requests in total, concurrency set to 5, but this time we are done in ~4 seconds! This means our new server is processing requests in parallel, much like your typical, best of breed Mongrel. Not quite the Reactor pattern EventMachine advertises, but a very powerful feature nonetheless - now you can see where all the FUD is coming from.

Deferrable: Concurrency with no threads

Borrowing heavily from Twisted, a Python event-driven network programming framework, EventMachine also includes a Deferrable module, which in some cases allows us to get all the benefits of concurrent processing without any threading overhead!

The Deferrable pattern allows you to specify any number of Ruby code blocks (callbacks or errbacks) that will be executed at some future time when the status of the Deferrable object changes. How might that be useful? Well, imagine that you're implementing an HTTP server, but you need to make a call to some other server in order to fulfill a client request.

> em-http-deferrable.rb

require 'rubygems'
require 'eventmachine'
require 'evma_httpserver'
 
class Handler  < EventMachine::Connection
  include EventMachine::HttpServer
 
  def process_http_request
    resp = EventMachine::DelegatedHttpResponse.new( self )
 
    # query our threaded server (max concurrency: 20)
    http = EM::Protocols::HttpClient.request(
	      :host=>"localhost",
              :port=>8081,
              :request=>"/"
           )
 
    # once download is complete, send it to client
    http.callback do |r|
     	resp.status = 200
        resp.content = r[:content]
        resp.send_response
    end
 
  end
end
 
EventMachine::run {
  EventMachine.epoll
  EventMachine::start_server("0.0.0.0", 8082, Handler)
  puts "Listening..."
}
 
# Benchmarking results:
#
# > ab -c 20 -n 40 "http://127.0.0.1:8082/"
# > Concurrency Level:      20
# > Time taken for tests:   4.41321 seconds
# > Complete requests:      40
 

No threading, and yet we still finish 40 requests in ~4 seconds - the only limitation is the threaded server we built in the previous example, which maxes out at 20 concurrent requests. Magic, almost! This is the beauty of EventMachine: if you can structure your worker to defer, or block on a socket, the Reactor loop will continue processing other incoming requests. When the deferred worker is done, it generates a success message and reactor sends back the response. The sky is the limit here, no Ruby threads, no synchronous processing. For a great example of this pattern, take a close look at Dnsruby.

eventmachine.zip (Server implementations)

Downloads: 278 File Size: 1.9 KB

EventMachine is a speed demon!

For more information on EventMachine make sure to check out the documentation and a few examples. Likewise, Bruce Eckel's article on Twisted is well worth a read if you're interested in learning more about the Deferred pattern. Last but not least, kudos to Francis Cianfrocca for the great framework, and his time to help me wrap my head around it!

Next time you hear someone shoot down an event-driven server, please (politely) correct them!

A brand new Rails/Merb app you put together over a weekend, a pack of Mongrels, a reverse proxy (like Nginx), and you're up and running. Well, almost, what about that one request that tends to run forever, often forcing the user to double check their internet connection? Response time is king, and you always want to make sure that your site feel snappy to the user. Did you know that Flickr optimizes all of their pages to render in sub 250ms?

When you're fighting with response times, the worst thing you can possibly do is queue up another request behind an already long running process. Not only does the first request take forever, but everyone else must wait in line for it to finish as well! To mitigate the problem HAProxy goes beyond a simple round-robin scheduler, and implements a very handy feature: intelligent request queuing!

HAProxy Request Queuing

Usual (bad) scenario: 2 application servers, each capable of serving 5 concurrent requests, and a reverse proxy (Nginx, Apache, Pound, etc) in front, which distributes request in usual round-robin fashion (a,b,a,b, ...). A long running request hits our application, gets to the head of the app. server queue and begins processing. Alas, now everyone else queued on that server is waiting for it to finish! Usual scenario and poor user experience. (For the astute reader: yes, the concurrency on the server is usually greater than 1, but Rails in particular has some nasty mutex locks deep inside. Hence, a long running database call can very quickly render the entire app. server incapable of doing anything else.)

HAProxy (good) request queuing: instead of blindly passing requests to each app. server, in round-robin fashion, HAProxy allows us to limit the number of outstanding connections between the client and the server. For example, we can specify that each app server may only have two outstanding requests (which means that at most 4 jobs will be queued on our two application servers), and all other clients will wait in the HAProxy queue for the first available app. server. Beautiful! If one of our app. servers is taking a long time to respond, at most only one other request will be affected and everyone else will be processed by the first available server in our cluster.

Sample config to illustrate the functionality (download full config below):

> haproxy.conf

listen app_a_proxy 127.0.0.1:8100
  # - equal weights on all servers
  # - queue requests on HAPRoxy queue once maxconn limit on the appserver is reached
  # - minconn dynamically scales the connection concurrency (bound my maxconn) depending on size of HAProxy queue
  # - check health of app. server every 20 seconds
 
  server a1 127.0.0.1:8010 weight 1 minconn 3 maxconn 6 check inter 20000
  server a1 127.0.0.1:8010 weight 1 minconn 3 maxconn 6 check inter 20000
 
listen app_b_proxy 127.0.0.1:8200
  # - second cluster of servers, for another app or a long running tasks
 
  server b1 127.0.0.1:8050 weight 1 minconn 1 maxconn 3 check inter 40000
  server b2 127.0.0.1:8051 weight 1 minconn 1 maxconn 3 check inter 40000
  server b3 127.0.0.1:8052 weight 1 minconn 1 maxconn 3 check inter 40000
 

QoS with HAProxy and Nginx

Do you want to guarantee best performance to a sub section of your site? With a little extra work on your HTTP server (Nginx in this example), we can distribute the load between different Mongrel clusters, each hidden behind a HAProxy interface:

> nginx-haproxy.conf

# ... nginx configuration
 
upstream fast_mongrels  { server 127.0.0.1:8100; }
upstream slow_mongrels { server 127.0.0.1:8200; }
 
server {
  listen 80;
 
  location /main {
     proxy_pass http://fast_mongrels;
     break;
  }
 
  location /slow {
     proxy_pass http://slow_mongrels;
     break;
  }
 
}
 
# ... nginx config continues
 

Take a close look at the earlier HAProxy config: app_b_proxy cluster has lower maxconn (local request queue) of 3 requests, and also features more app. servers to process the long-running tasks. This way, anytime a user requests '/slow', they will be sent to our dedicated app_b_proxy and the load will be spread between 3 dedicated servers. Meanwhile, the rest of our site is completely unaffected, as it is served by a different cluster!

igvita-haproxy.zip (Full Nginx + HAProxy configs)

Downloads: 528 File Size: 1.3 KB

Hidden Gems in Nginx and HAProxy

Make sure to read the detailed overview of the architecture and many great examples of using HAProxy in the wild, as well as, study the many configuration options available for more hidden goodies.

Last but not least, it's worth mentioning that you can get similar functionality in Nginx directly with the 'Fair Proxy Balancer' plugin (EngineYard is rumored to be running it). Having said that, HAProxy is more robust, gives you more knobs to tweak, and features a great admin interface!

Memcached Best Practices

At its core, Memcached is a high-performance, distributed caching system. It is application neutral, and is currently used on many large scale web sites such as Facebook (2TB of cache, circa Q1 2008), LiveJournal, Mixi, Hi5, etc. However, it is also an extremely simple piece of software: all of the logic is client-side, there is no security model, failover, backup mechanisms, or persistence (albeit the last one is in the roadmap). But that hasn't stopped the developers from deploying it in all kinds of environments, and here are a few best practices suggested by Brian:

1. Don't think row-level (database) caching, think complex objects
2. Don't run memcached on your database server, give your database all the memory it can get
3. Don't obsess about TCP latency - localhost TCP/IP is optimized down to an in-memory copy
4. Think multi-get - run things in parallel whenever you can
5. Not all memcached client libraries are made equal, do some research on yours - hint, use Brians.

Slab Allocator - Heart of Memcached

The heart of Memcached is in its memory slab allocator. A little daunting at first sight, it is actually a very elegant solution once you understand the motivation and the tradeoffs of its architecture:

• Size of maximum allocated memory to Memcached is only limited by the architecture (32/64-bit)
• The key size is limited to 250 bytes, the data (value) size is limited to 1mb
• On bootup, Memcached grabs the memory and holds on to it - wasted memory at cost of performance
• Allocated memory is split into variable sized buckets by the slab allocator
• Default slab allocator will create 32-39 buckets for objects up to 1mb in size
• You can customize the page size at compile time, and slab sizes on startup
• Each stored objects gets stored in a closest-size bucket - yes, memory is wasted
• Fragmentation can be a problem - either customize slab sizes, or evict/flush your cache every so often
• If there is unused memory in a different slab class, that memory will be re-allocated, if required
• To guarantee no-paging, disable swap on your OS - practically, just keep it tiny, as to avoid disasters

Memory Management

The memory is managed via a Least Recently Used (LRU) algorithm:

• Each slab class has its own LRU - evict target depends on the size of the object
• Expiration timestamps are checked once every second - minimum lifespan is 1 second
• Objects marked for deletion are handled asynchronously - checked and evicted once every 5 seconds
• Inconsistency between the two timers listed above can result in sub-optimal eviction policy
• LRU can be completely disabled - do it at your own risk

Best Practices: Invalidations and Expiry

Memcached does not provide any mechanisms for deleting a set of associated keys (object, name, etc.). For good or for worse, you could implement this functionality yourself with the help of prepend and append commands, however, be careful with the 1mb limit! A much cleaner way to handle this situation is to forgo the invalidation process all together:

• Instead of invalidating your data, expire it whenever you can - memcached will do all the work
• Generate smart keys - ex. on update, increment a version number, which will become part of the key
• For bonus points, store the version number in memcached - call it generation
• The latter will be added to Memcached soon - as soon as Brian gets around to it

Roadmap and Future Ahead

Memcached is a force to be reckoned with already, and the roadmap is only going to solidify this position. Amongst the big objectives:

• Binary protocol is in the works
• Support for generations - as described in paragraph above
• Multi-engine support: char based, durable (persistence!), queue
• Facebook has overhauled the core to be highly-threaded, and is expected to contribute the changes

Kudos to Brian and Alan for a great presentation! Don't forget to bookmark the slides and check back every so often, as they will be updated with time to describe best practices, common mistakes, and the roadmap ahead.

Customizing color schemes in PuTTY

It appears that the best way to approach this problem is to enable ANSI color support and customize from there. Thankfully, Steve T. has documented his findings, which enabled me to easily customize a few themes of my own. From left to right, in the preview above: default theme, light, desert (with a few improvements).

igvita-light.reg (Light-er default theme)

Downloads: 2340 File Size: 14.2 KB

igvita-desert.reg (Desert theme)

Downloads: 2826 File Size: 14.3 KB

To get started, download the registry file, run it, reload your PuTTY terminal, enjoy! Tip: Vista users, make sure you're using the new 'Consolas' font as your default.

Nice upgrade: PuTTY Tray

While hunting for colors, I also came across an awesome enhancement: PuTTY Tray, which I highly recommend. Amongst many other things, it offers auto-reconnect, session persistence in files, 'always on top', URL hyperlinking, and the list goes on. And the best part is, for all intents and purposes, it's the same PuTTY terminal!

What's your favorite PuTTY tip or trick?

The underlying tenets of agile - short iterations, quick releases, faster feedback loops - create an underlying tension with many of the 'traditional' testing requirements. A quick google search for 'types of software testing' brings up several dozen concepts, many of which are nothing short of unrealistic given typical one/two week team iteration. Of course, value of any practice depends on its context, which is another way of saying: there are good practices in context, but there are no best practices. Hence, let's set the context: you're a small web development team (2-7 people), working on a live project (you have customers), with expectation of delivering a functional slice of your backlog every two weeks.

Flickr, Friendfeed, and others...

Good software testing is a challenging process. Every project follows some logical path from under the developers fingers and into a production environment - granted, sometimes this process can be as simple as editing a file live (we're all guilty of it at times). In their early days, Flickr ran with pseudo 30-minute release cycles, more recently, Friendfeed is showing several 1-2 day release iterations. Have either fully automated their testing and staging environments? I doubt it. In fact, Cal Henderson's remarks about the early days of Flickr clearly show lack of such infrastructure, and yet, Flickr has always been known for their excellent user experience.

Discovering the formula

How far do you go? Unit testing can be derived from test driven development, integration testing requires a staging environment, user acceptance is the logical conclusion in Scrum, and what about the remainder? Given the scarce resources - time, people, and money - most web startups seem to forgo, or pick and choose from the remainder. Perhaps this is a sign of immaturity and we're unconsciously hurting ourselves, or maybe, the old model simply does not recognize the modern requirements.

Intrigued by this question, I've recently engaged in a number of online and offline conversations with team leads, and fellow entrepreneurs. The end result? A full spectrum of testing environments, including a few cases of complete lack of thereof - all the while, the projects are high-profile, high-traffic websites. For obvious reasons, we'll let the names be anonymous, but this raises an interesting question:

How do you test, what do you test, when do you (or your team) stop? Phrased differently, what are your release procedures?

Would love to hear your feedback, war stories, and rumors. Once the dust settles, I'll aggregate and report the results.

P.S. Testing is not assurance of quality.