Richard Crowley’s blog

BashReduce

2009-06-27T17:18:55-07:00

I might be crazy but I’m loving Last.fm’s BashReduce tool [1]. So much so that I’ve taken time at work to make some improvements so we can use it for some internal data analysis (unrelated to the public DNS Stats I spoke about at Velocity [2]).

Changes

I had to change the argument structure a bit from Erik Frey’s original version to accommodate some of the bigger changes I made:

-h is now how you specify a list of hosts.
-? is now how you get the detailed help message.
-m is now how you specify the map program.
-M allows you to specify the merge program.
-i now accepts a file or a directory.
-f does some DFS-like things I’ll discuss in more detail later.

I think specifying your own merge step can be pretty useful, especially in cases where you want to create a re-reduce step on the end. A user still has to be careful to do as little work as possible in the merge step since it is serializing the output of map and reduce.

The new -f option causes br to distribute filenames instead of lines of data over the network (via nc). The implication here is that each node has a mirror of the directory specified with -i.

Because br can now act on a directory instead of a single file, it’s possible that you as a user would want to keep some or all of this data gzipped. br will transparently handle gzipped content when -i is specified (gzipped stdin is not supported, use zcat).

One last thing: stderr from most parts of br is saved to $tmp/br_stderr where $tmp is the -t option (which defaults to /tmp).

Performance

I’ve found performance to pretty well follow with the number of cores made available with one caveat: if your dataset is small and your merge step is significant, it will dominate and performance gains will be reduced.

Distributed Filesystem

I want to spend some time talking about what a distributed filesystem for br will look like. For OpenDNS’ immediate purposes, a filesystem mirrored using rsync is desirable from both a performance and durability point-of-view. For more general use, being limited by the size of your smallest node is not acceptable.

Growing the dataset beyond the size of the smallest node introduces two challenges. First, how to partition and replicate the data and second, how to find these partitions later. Ultimately, I realize the answer is probably to stop hating and use Hadoop. Humor me.

The simplest solution is probably the RedHat GFS [3] since it’s POSIX-compliant and designed to scale beyond one node’s capacity. The downside is that it isn’t able to take any of the locality considerations of HDFS into account.

The other interesting possibility is wiring MogileFS [4] or HDFS [5] itself underneath br as a frontend. With Hadoop Streaming, this isn’t too far fetched and may represent the best route for a br user who outgrows a single-node’s storage capacity.

Licensed by the goodwill of Erik Frey

Here’s my version: http://github.com/rcrowley/bashreduce

“Building OpenDNS Stats” at Velocity

2009-06-23T17:45:56-07:00

I just finished talking about OpenDNS’s stats system at Velocity and I think it went rather well. I got the official Allspaw and Hammond seals of approval so my life is pretty much complete.

Here are the slides: http://rcrowley.org/talks/opendns_stats.pdf

Questions I expected and wasn’t asked because I only left 8 seconds for questions:

When you catch std::bad_alloc, why do you shutdown the process instead of just freeing some memory? I chose to exit and let supervise restart the process because I might have run out of memory due to a slow memory leak. Starting fresh is safer and doesn’t have any downsides. In production, each node tends to restart every day shortly after midnight, UTC as rows for the new day start being allocated.
What’s up with the waffle at the end? Wednesdays are Waffle Wednesdays at OpenDNS. We’re hiring.

~~I’ll update later with the video link when O’Reilly posts.~~ Moving pictures with sound! http://blip.tv/file/2286154/ and embedded below.

(spacetimeshift)

2009-05-01T16:37:44-07:00

It’s almost the first Saturday in May, which means it’s almost time to ~~drink and gamble~~ celebrate my Kentucky heritage. Here’s the setup I will be using to observe the race at 6:24pm as Eastern Daylight Savings Time intends.

It’s good, ol’ fashioned Kentucky cable TV, split and sent to both the TV there and an old TiVo. The TiVo outputs to and is controlled by Mike Malone’s Slingbox, which is being displayed on my TV in San Francisco via Malone’s DVI-to-HDMI cable. Thanks, Mike!

Haystack and Whoosh notes

2009-04-26T20:52:02-07:00

Real search is always better than running LIKE queries from MySQL so today I picked up Haystack [1] and Whoosh [2]. I chose this combination for the low barrier to entry and the easy upgrade path should that be required. Both are pure Python and speak setup.py.

The first problem I ran into has actually been fixed but not committed. The Gist embedded below patches Whoosh as recommended in the bug report [3]. The bug manifests itself as “IOError: [Errno 24] Too many open files” when you try to load even modestly sized datasets all at once. I can’t make my laptop give me more than 8192 file descriptors and my Slice will only give me 1024 so I could never see just how bad things got on a 1.5 million row sample. With the patch, though, everything is golden.

The second and last problem I encountered was more of a documentation problem. Some of the official tutorial is a bit overkill, so here’s the fastest get-up-and-go tutorial I can distill:

Add 'haystack', to INSTALLED_APPS in your settings.py. Also add these two lines to let Haystack know where to keep your Whoosh index files:
```
HAYSTACK_SEARCH_ENGINE = 'whoosh'
HAYSTACK_WHOOSH_PATH = '/path/to/server/writable/directory'
```
Add two lines to your global urls.py:
```
import haystack
haystack.autodiscover()
```
Create a file called search_indexes.py next to models.py. This file will contain model-like classes defining your search schema. It is important to list every field you will want in your search results (the primary key, for example) in the search schema. The field defined with document=True and the prepare method determine the searchable data.

Update: Daniel Lindsley pointed out that Haystack reserves id for itself so I’ve changed my example to use a slug field. Same point applies, just don’t use an id field in your subclasses of SearchIndex.
```
from haystack import indexes
from haystack.sites import site
from models import Foo
class FooIndex(indexes.SearchIndex):
	text = indexes.CharField(document=True)
	slug = indexes.CharField(model_attr='slug')
	name = indexes.CharField(model_attr='name')
	city = indexes.CharField(model_attr='city')
	state = indexes.CharField(model_attr='state')
	def prepare(self, obj):
		self.prepared_data = super(FooIndex, self).prepare(obj)
		self.prepared_data['text'] = obj.name
		return self.prepared_data
site.register(Foo, FooIndex)
```
I’ve called the indexable data “text” and use the prepare method to explicitly allow searching by name only. The official documentation ask for a template file to use during preparation but I think this is overkill.

Replace your old ORM-based search view with something like this:

from haystack.views import SearchView
def search(req):
	return SearchView(template='search.html')(req)

Replace your search page’s template with something like this:

{% extends 'layout.html' %}
{% url core.views.search as base %}
{% block content %}
<form action="{{ base }}" method="get">
<h1><label for="query">{% block title %}Search{% endblock %}</label>
for <input id="query" name="query" type="text" value="{{ query }}" />
<input type="submit" value="Search" class="button" /></h1>
</form>
{% if page.object_list %}
	<ol start="{{ page.start_index }}">
	{% for o in page.object_list %}
		<li><a href="{{ base }}/{{ o.slug }}">{{ o.name }}</a></li>
	{% endfor %}
	</ol>
	<p>Page {{ page.number }} of {{ page.paginator.num_pages }}</p>
	<ul>
	{% if page.has_previous %}
		<li><a href="{{ base }}?query={{ query|urlencode }}&amp;page={{ page.previous_page_number }}">&larr; Previous</a></li>
	{% endif %}
	{% if page.has_next %}
		<li><a href="{{ base }}?query={{ query|urlencode }}&amp;page={{ page.next_page_number }}">Next &rarr;</a></li>
	{% endif %}
	</ul>
{% else %}
	{% if query %}
		<p>We couldn’t find anything named <strong>{{ query }}</strong></p>
	{% endif %}
{% endif %}
{% endblock %}

The view gets the query, a page from a regular Django Paginator and the paginator itself. A form comes along too but I prefer to ignore this. If you defined your own User model that lives at req.user, it must implement get_and_delete_messages [4] because django.contrib.auth.models.User leaks into django.core.context_preprocessors a bit.



Here’s the previously mentioned patch-and-install script
(Updated to reflect bugfixes merged into the trunk of
Whoosh!):





	http://haystacksearch.org/ ↩
	http://whoosh.ca/ ↩
	http://trac.whoosh.ca/ticket/25 ↩
	http://github.com/rcrowley/django-twitterauth/commit/83ba6a07df3e97455abd4ab2b3ffaba7096407bc ↩



College courses I should have taken
2009-04-09T17:15:56-07:00
Yesterday, I read through A List Apart’s 2008 Survey and the
Relevance of Education [1]
section tricked me into reflecting on my time at Wash U.  My professors
taught me all about performance and time complexity, data structures,
operating systems, circuits and hardware architecture (I am a hardware
designer by education).  All of this, save the direct design of new
hardware, has been immensely helpful in my first two years as a Real
Boy™.

Since yes-or-no questions are kind of boring, I reflected further on
what my college education lacked — courses I wish I’d
taken.  While I was working for Flickr and recruiting for Yahoo!, I
spent a part of each visit to Wash U encouraging the Computer Science
department to get serious about web programming.  The result was an
ostensibly senior-level course that guided students through LAMP, Rails,
databases and JavaScript plus the protocols/standards they build upon:
HTTP, HTML and CSS.  It was a great addition to the program and
I hope it’s still around.  Though I’ve not kept up
my course-list evangelism, I now have several more courses I wish I’d
taken.

Network algorithms
The algorithms course that I took focused on efficient computation on
a single node.  Wash U offered an Advanced Algorithms course that, in
retrospect, I should have taken.  I hope such a course covers
network-aware algorithms like map/reduce, gossip and P2P protocols,
queuing and distributed hash tables (Chord
[2], etc.).  These topics
become required knowledge when you start talking about “Internet
scale” anything.

Managing disk I/O
This course definitely doesn’t exist.  Academics are big fans
of pretending they have infinite I/O capacity but they do students a
disservice by hand-waving one of the largest time-sinks in all of
computing.  I have learned a lot by reading papers and blog posts
produced by my more experienced peers and even more by
trial-and-error.  Topics for such a course include write-ahead logs
(and all of the other things InnoDB does), aio and its effect
on performance plus some time to address concurrency as it pertains to
disk performance.  Engineering school should prepare people for reality
and I can’t thing of anything more real than iowait.

Lock-less data structures
I’m not certain this doesn’t exist at Wash U.  The
Drizzle mailing list [3]
made mention of lock-free data structures, a topic I haven’t given
much thought to beyond using concurrent_hash_map from the
Intel TBB [4].  Not
surprisingly, the design of these libraries is difficult and
theoretical — you have to prove correctness since so
much is left to chance.  Academics love proofs, right?

OS kernels
I took Wash U’s senior-level Operating Systems course but never
got around to taking the graduate-level version.  The former focused
on libc, system calls, threading and writing programs that
behave like UNIX programs.  The latter dives into the Linux kernel
itself.  I don’t anticipate becoming a kernel hacker anytime
soon but having deep knowledge on the design of the Linux kernel would
definitely make me a better software architect.

More econ
I took up through intermediate microeconomics and wish I had learned
more.  I don’t have the solutions to the world’s problems
and I still wouldn’t even if I knew more but I would understand
the expert’s opinions more thoroughly.

Anything I’m missing?  I’m curious what others wish
they’d taken in college, especially those who have been at it
professionally for more than two years.



	http://aneventapart.com/alasurvey2008/02.html ↩
	http://en.wikipedia.org/wiki/Chord_(DHT) ↩
	https://lists.launchpad.net/drizzle-discuss/msg03337.html ↩
	http://software.intel.com/en-us/intel-tbb/ ↩


 



How-to
2009-04-02T09:17:12-07:00
My absolute favorite kind of how-to post involves me downloading,
reading and running a shell script that automates an otherwise
tedious and non-obvious process.  If it were straightforward, I
wouldn’t be googling for it in the first place.  The Gist
tool on GitHub is perfect for how-to scripts, so here we go.

So, since I’ve already chugged the promises of the Drizzle
Kool-Aid, it is time I started using it in hopes I can soon contribute
code myself.  Step one is getting the bugger to install.  Step
two is writing a little program using the very Brian Aker-ified
libdrizzle C library.  Today is a step one kind of
day.

I’m on Ubuntu Intrepid myself but from the package lists, this
looks like it'll work almost out-of-the-box for all Ubuntu flavors. 
The hands-on part will be the Bazaar PPA but a visit to
Launchpad will get
you the right /etc/apt/sources.list lines and the appropriate
key.

I started from a system that was already full of compilers and libraries
so, for completeness, I’ve included a modification to the
official Ubuntu
dependency list.  The modifications get you newer Bazaar, newer
libevent and what I believe to be a more generally build-ready
environment.

This installs libprotobuf and libevent
manually so make sure to comment those sections if you already have a
version you like.



 



Distributed version control
2009-03-16T21:32:17-07:00
First Wistow [1] and
now Malcolm [2]! 
Wistow wrote a couple of weeks back about Git being awesome but
encouraging the kind of patch hide-and-seek he had to endure when
building NCSA httpd.  In the comments we discussed whether the
tools offered enough benefits to outweigh the inherent encouragement
to fork your forks of your mother’s fork.  I still maintain
the tools offer significant benefits even in the face of Malcolm’s
concern that rampant forking can splinter communities before they really
form.  Being a Django juggernaut, his word carries a lot of weight
but I feel obligated to point out that the pile of forks that every
open-source project must endure is not a new problem.

I bet every open-source program ever released has been tweaked by
savvy users to do something extra, something different or something more
correct.  Distributed version control and sites like GitHub have
brought the forks front-and-center, which I think is a much nicer place
to keep them.  Malcolm’s got a point — it shouldn’t
be up to the user to decide which fork or set of forks is best; a successful
project needs a release to serve as the official starting point. 
Merging patches or branches is a job for the fearless leader and one which
should not be taken lightly.

Distributed version control gives the fearless leader a fighting chance
to make sense of the parallel thought processes of any project’s
multiple developers.  An open-source project living in Subversion is
stuck with patch files, emails and tickets; a project using Git gets branches,
history, false starts and good ideas in as much or as little detail as the
fearless leader needs to decide the fate of the patch.

Version control is a tool, not a form of government.  It is the
fearless leader’s responsibility to merge the best patches into the
master copy, get second opinions and rewrites of iffy changes or flat-out
reject feature creep and poor implementation.

There will always be both selfish users and philanthropic users. 
In this awesome new world that has so many people coding in public
(thanks GitHub, Launchpad and Bitbucket!), it is actually becoming easier
to be philanthropic than selfish.  It is easier to fork
$PROJECT on GitHub than to sequester changes behind your
Virtual Paranoid Network and GitHub keeps the fearless leader
informed.  The loop is closed.  We’ve emerged from patch
hell.  We have the commit messages to prove it.



	http://deflatermouse.livejournal.com/148975.html ↩
	http://www.pointy-stick.com/blog/2009/03/16/dark-side-distributed-version-control/ ↩


 



Drizzle
2009-03-03T08:09:49-08:00
The SF MySQL meetup last night [1] about Drizzle [2] was a home run by my limited research [3].  It
won’t be a get-out-of-jail-free card for very write-heavy
applications but I bet it will do wonders for heavily replicated, heavily
federated, read-heavy architectures (you know, normal stuff).

The common conception of Drizzle is a bit off center so I’d
like to offer my 2-sentence version.  First and foremost, Drizzle
is a refactor of MySQL for today’s hardware and today’s
architectures.  Second, Drizzle is pluggable — everything
from the storage engines (which default to InnoDB) to the pluggable
PAM authentication and replication — is replaceable or removable.

I intimately understand the pros and cons of global mutexes. 
Much of my performance testing at OpenDNS has to do with how long our
stats pipeline spends waiting for mutex locks.  I have to commend
Brian Aker and Drizzle’s commitment to their removal.  I
suspect read-heavy loads will see a major performance increase from
this alone (Brian was quoting something like 11-13% of time was spent
in the heavily-mutexed authentication code Update: that time
is spent in parsing, not in auth, still, the mutexes are
bad).  The last big mutex
is the one that enables MyISAM and it sounded like (it could have been
a convincing planned monologue) Brian decided during his talk to remove
it, meaning you’d not be able to use MyISAM in Drizzle. 
That’s fine by me — I only use MyISAM in places where
table-level locks are okay.

That time spent in authentication is usually a complete waste. 
Motivated by the countless MySQL instances running as “root”
with no password, authentication is now a removable plugin that can
also use PAM auth like your OS can.  I hope my operator will let
me just take it out.

Most folks in the room didn’t seem to pick up on this one but
there was a hand-wavey mention of a “shard bit” in the wire
protocol which would open the doors to software (or even hardware!)
load-balancers that could understand how you federate your data.

The last thing I found compelling enough to write down is that Drizzle
is making heavy use of Google’s Protocol Buffers [4] as part of their query planning. 
Serialized, they’re used as an alternative wire format and as the
replication protocol.  Crazy things like replicating some Drizzle
tables through a Protocol Buffer proxy into a different database engine
become downright easy.  For example, keeping your main table in a
very indexed Drizzle database while replicating a distilled version into
MemcacheDB for fast primary-key lookups of the essential data.

(This reads like a press release, I’m sorry.  I wrote
all of this down as a cheatsheet for talking to the rest of OpenDNS about
when and if this will make sense for parts of our architecture.)


	SF MySQL Meetup — Why Drizzle ↩
	Drizzle ↩
	mjmalone, mihasya & ericflo ↩
	Google Protocol Buffers ↩


 



Diversion
2009-02-27T07:56:32-08:00
All I think about at work is disk I/O.  That’s an
overstatement but the point is to motivate my late night and early
morning batch of small changes to Bashpress.

First up, I made the homepage a 302 redirect to the most recent
month’s posts.  It has long bothered me that blogs are
singluarly obsessed with permalinks and yet will place rotating
content on the homepage without a second thought.

Second and equally inconsequential is that I added a flag to
Bashpress to globally enable or disable comments, which I’ve
disabled.  Al3x’s reasoning sits well with me but for someone without
his noteriety, I need a crutch.  delicious.js pulls
in every save and note from del.icio.us for the post, at least
giving folks willing to dive deeper a map of sorts.  It
isn’t perfect but hopefully I can work out trackbacks or
something to complete the puzzle.

Small pieces loosely joined, etc.

 



django-twitterauth
2009-01-24T19:30:36-08:00
For a new little side project I’m working on, I need a way to
interact with Twitter on a user’s behalf.  You know, like
what OAuth is for.  Until that blessed day, I’ve got django-twitterauth
to keep from stealing Twitter passwords.

Your app’s Twitter user first follows the user that is trying
to authenticate.  The user is then directed, by way of a hidden
<iframe>, to send a direct message to the app’s
Twitter user containing a SHA1 hex digest.  Once the app verifies that
it received the direct message, the user is authenticated.

It isn’t the best flow in the world but it isn’t
the worst, either.  I hope my Django n00b-iness isn’t shining
too bright.

Once again, django-twitterauth on GitHub.

 



Out of paper
2009-01-13T11:04:45-08:00
Last night at the laundromat it took me three attempts to add money to
my stored-value card because I, like everyone else, don't read the messages
in modal dialogs.  I know that ATMs, gas pumps, laundromats and other
credit card readers ask if you’d like a receipt before they complete
the transaction.  Sometimes, though, they tell you they can’t
print a receipt and ask for your permission to continue.  The problem
is that the options available are the same in both cases: “yes”
or “no.”  It looks like this:

Would you like a receipt?
    YES             NO

Unable to print receipt.  Continue?
    YES             NO

Because no one reads the actual prompt, the options need to indicate
that something unexpected is going on.  This is doubly true when the
machine is so low that I can’t even see the top line of the
display.  The special case of the printer being out of paper should
read:

Unable to print receipt.
    OK              CANCEL

By breaking the expected pattern of put in my card, press “no”,
select $20 and wait, I can do the right thing the first time and avoid being
turned away by the washing machine.

 



Django
2008-12-15T11:56:19-08:00
I asked Malone the other week why
Django made me go all return render_to_response('foo.html')
in my view named foo instead of assuming some defaults.
To be explicit, candidates for those defaults might go something along
the lines of:


	If a view function returns None (which is,
	it seems what happens when you don't return anything), render the
	template that shares its name with the view.
	If a view function returns a dict, render the
	template that shares its name with the view, passing this
	dict wrapped up in a Context.
	If the view function returns a string, take the string as the
	template name.
	If the view function returns a 2-tuple containing
	a string and a dict, take the string as the template
	name and make the dict into a Context.


On top of that, I asked him why Django didn't figure out what response
format makes sense, given the URL and headers in the request and the
formats available for the template chosen.  His response was basically
that Django hates magic.  If it did all of these things, it'd favor
convention over configuration and be called Rails.  Or Pails.

After playing around with Django for a few days and now starting to
use it for a real project, I'm seeing the wisdom in Django's (and indeed,
Python's) choice to avoid magic.  I've already used the word "explicit"
once in this post and that's what this is really all about.

Additionally, Ruby seems to skew towards implicitness in
the language, while Python skews toward explicitness. I like explicitness.
— Joe
Stump

I'm not here to start a Ruby-versus-Python or Rails-versus-Django fight.
I'm here to talk about being explicit with code.

My efforts a while back hacking
the Ruby interpreter were a bit misguided.  I was heavily and
prematurely optimizing my desired use case at the expense of code clarity
and at least seven other traits generally considered to be positive.  In
the resulting Frankenruby interpreter, @foo could show
up in a file outside of a class with no warning, meaning that file could
only ever be used by my very streamlined URL-to-code routing script.  In
retrospect, it's a good thing I never got it to work just right.

The desire to remove as many declarative, housekeeping-esque lines of
code is seemingly noble but it is at odds with another principle of mine,
learned from years in the trenches writing C(++): always
#include everything a file uses.  This makes files instantly
more portable and less magical.  Despite Django's mascot being a magical
pony, I gather that it, too, avoids magic in most scenarios.

Being explicit allows Django to be loosely-coupled.  Imagine trying to
remove the Django ORM or templating system if they were omnipresent in
every Django app from start to finish.  Tedious.  As it is, a single
"#" sets you free.  Even the concept of having many apps
within a project enables further explicit use or disuse of each package.

Being explicit also lowers the barriers-to-entry for would-be framework
hackers like myself.  My first serious afternoon saw me trying to create
a fairly involved template tag.  It would have made page titles work just
about like the excellent Headliner plugin for Rails.  I discovered along
the way that the {% block foo %} template tag, if placed within
another block, would both override the named block and output its contents
in place.  Observe, here's how I decided to do page titles in Django:

In templates/layout.html:
<title>Foo — {% block title %}{% endblock %}</title>

In other templates:
{% extends 'layout.html' %}
{% block title %}Only shows up in the title tag{% endblock %}
{% block content %}
<p>foo</p>
{% endblock %}
{% extends 'layout.html' %}
{% block content %}
<h1>{% block title %}Shows up here and in the title tag{% endblock %}</h1>
{% endblock %}

The ORM looks similarly easy to figure out though I haven't taken the
plunge yet.  I already know what I'm going to build to learn it, though:
a command-line SQL tool that works through a combination of Python and
hard links.  The python script will find the closest Django settings module
and use the database connection info there.  The command will be invoked by
typing out a SQL statement on the command line.  The SQL statement will be
' '.join(sys.argv) and the commands SELECT,
INSERT, etc. will be routed to the Python script using hard
links.  I don't know if this is a good idea but I know it'll be relatively
easy to do thanks to how easy it seems to be to dive into the Django
internals.

 



Hacking the Ruby interpreter
2008-10-26T18:11:16-07:00
I like Ruby.  Ruby the language anyway.  I have started to have violent
reactions to PHP's lack of closures.  However, Ruby has one deficiency that
makes me almost as mad as PHP's lack of closures.  Assume for this example
that the file foo.rb looks like this:

@foo = "I'm getting ready to abuse the instance variable syntax."

As is, Ruby lets me do the following, which is functionally desirable
but makes the good programmer part of me twitch:

class Foo
    def initialize(path)
        self.instance_eval(File.read(path))
    end
end
f = Foo.new('load_me.rb')
f.instance_variables # => ['@foo']

Replacing the bold line with the following is what I'm after:

        load_in_scope path

In my dream, load_in_scope is a function defined in the C
code that compiles a file using the Bison parser to read through the file.
This saves a whole round trip to and from memory, making the program more
efficient and me less twitchy.

I'm having major trouble getting the Ruby interpreter to do this.
Starting with a trunk checkout and working backwards from the
Kernel#load method, I've been able to segfault at least
two-dozen different ways.  I think there are only two changes that are
necessary.  First, after rb_load_file returns a NODE
*, rb_iseq_new must turn that into an instruction
sequence with a type other than ISEQ_TYPE_TOP (perhaps
ISEQ_TYPE_CLASS or ISEQ_TYPE_EVAL?).  Second,
the instruction sequence can't be passed to rb_iseq_eval
because that calls vm_set_top_stack, so
load_in_scope must call vm_set_eval_stack and
vm_exec itself.

Am I insane?

 



Allspaw's playbook
2008-10-23T20:07:46-07:00
I just finished the proper book part (appendices tomorrow) of Allspaw's The Art of
Capacity Planning.  His playbook is now my playbook.  I've got more
than my fair share of charts and graphs (as all good engineers do) but
am to some degree at a loss in using them.  On one hand, the book presents
nothing but common sense and on the other, it has saved me endless
trial-and-error.  I've been reading it on the morning bus and have put
something into practice each day.

Most of his advice on growing a service applies equally to launching a
service, albeit with a nasty chicken-and-egg problem that needs solving
before your boss will let you buy hardware.  By piping production traffic
into my dev environment I've been able, in stages at least, to put every
bit of my architecture under real production load (empirical, as
per the Allspaw doctrine), piling on more and more until my work queues
go hockey stick.  (That'd be the bad kind of hockey stick.)  The result speaks for
itself.

Thank you, John Allspaw, for giving me a script to follow for the past
week and a half.

 



Updates to my street sweeping app
2008-09-21T19:51:56-07:00
It's been a long time coming but I've finally updated Where's my car? to take recent updates
over at SF
GIS into account.  They've augmented the street sweeping schedule in
most places to include something to the effect of "Effective 8/25/2008,
1st & 3rd Tuesday only" which will reduce the "false positive" problem with
the site.  Now the site will be very explicit about the date and time you
should move your car instead of relying on "Mon 9AM" to get the point
across.