Recent Blog Posts

Django at Scale slides

2011-06-24T00:00:00Z

Last night I gave a talk on scale at AWPUG. I'll admit I had no real idea what I was going to talk about until about two days before, so... your milage with these slides may vary. Most things worth discussing in a group are applicable to more than yourself and by definition reusable. Thankfully Disqus has released a lot of code related to problems we've had a scale, so I pointed out a lot of those in my presentation. Aside from apps you should investigate there are a few quick do's and don'ts, but nothing earth shattering.

I included my notes in the slides. If you click through to SlideShare you can download a plain ol' PDF.

Django at Scale

View more presentations from bretthoerner

Austin

2011-05-30T00:00:00Z

Austin is a relatively liberal (or libertarian, depending who you ask) city in the heart of Texas, you might think of it as "everything you've heard about Texas, but the opposite." It may be the capitol, but it's the third smallest city of Texas' big six (Houston, Dallas, San Antonio, Austin, Fort Worth, El Paso). It's an affluent city with a relatively low cost of living and is within 1% the per capita degrees of Seattle and San Francisco. If you've never been here you might think of it like Texas' Portland or Boulder.

Indoors

Downtown Austin full of museums. Tastes tend to differ too much here, I'll let you decide which sound interesting.

You should check out Waterloo Records and Book People which are right next to each other west of downtown. They're also both by the original (and flagship) Whole Foods, if you're into that kind of thing.

The state capitol and University of Texas campus (38,000 undergras and 12,000 postgrads) are both interesting to see (and easy to find), but are a mix of indoor/outdoor.

The Alamo Drafthouse is an awesome local chain of theaters that serve food and alcohol to you while you watch. Not super unique these days, right? Well, check out their signature event list, first. They do quote-a-longs, sing-a-longs, shoot-a-longs (get a cap gun, help shoot bad guys), dance-a-longs, and more. Most events are adult only so you aren't surrounded by small children.

Outdoors

Texas is hot as shit in the summer. This shouldn't be news to you. There's a reason many outdoor activities revolve around water.

Zilker Park: A huge park just Southwest of Downtown, lots of various activies and events depending on time of year.
Barton Springs Pool: Located in Zilker Park, fed by spring water.
Mount Bonnell: A short drive from downtown brings you up to a great view of downtown and the hill country.
Kayaking on Lady Bird: There are many locations along the river that borders downtown where you can rent a kayak (or paddle boat, etc) to play with on the relatively still water.
Tube San Marcos: This one is a bit of a drive, but you basically float a slow river (preferablly with your cooler floating next to you).
See the Congress Bridge bats: In Summer a colony of bats live under the Congress Bridge (just South of downtown). People lay on the hill under the bridge and wait for them to fly out sundown.

Austin is located in Texas' Hill Country. There are many popular hiking and bike trails, but I'm not your man for information on either. A quick search should turn up lots of results.

Food

Austinites spend the most money dining out per capita. We're bound to have some good eats.

Uchi: Some of the best Japanese / Sushi in the nation. Really. ($$$)
Home Slice: The best NYC style pizza I've ever had (I've been to NYC four times, for what it's worth).
Franklin Barbecue: It's Texas. This is the place (inside Austin) to go.
Frank: Hot dogs, cold beer.
Moonshine: Your typical "new twist" on American food, very good though. ($$)
Magnolia Cafe: Local 24 hour diner with multiple locations. As seen on TV.
Curra's Grill: Interior Mexican food, get some pork.
Bouldin Creek: Vegan / Vegetarian cafe, all day breakfast.

There are many, many more. Yelp is a pretty good resource here. One thing I've mostly left out is Austin's huge trailer food "culture." Hit South Congress or East 6th for whole "trailer parks" full of amazing food. No joke.

We also have tons of great coffee shops, I could name a few but I don't drink the stuff.

Music and Nightlife

Austin is the self proclaimed "live music capital of the world". Did you roll your eyes? It's cool. But if we're far from it, I wonder why nobody else has taken the crown? If someone else is really #1, and we're #2 or #3 in the middle of Texas with a population of 800,000, so be it.

The point stands, though: we have lots of great live music.

Venues and shows change far too often to list here, check out Show List Austin.

As for non-music nightlife, you have to see 6th Street on a weekend night, preferably 10PM-2AM. It's one of the most densly packed areas of bars you'll ever find. The streets around 6th also have bars and are often better for actually lounging in with friends.

East 6th ("East" in Austin terms almost always means "cross under I-35") is also a fun place to go find a bar. It's much more quiet and much more hipster than downtown. You have to go to Rio Rita and have some habenero infused vodka.

Redis at Disqus

2011-02-21T00:00:00Z

We've been running Redis in production for a bit over three months now on projects ranging from critical (paying premium users) to experimental (in-house). Just as Will at Bump was inspired by someone else, I was inspired by Will to do a write up on what we're up to.

Analytics

Our entire premium analytics service runs on Redis. These stats are, as you'd expect, time-based. In order to squeeze this problem into a key/value store, we just append date-string suffixes to the keys: "comments_by_day:disqus:disqus:2011-01-01". If we split on ":" we have 4 pieces: the stat name, the account, the site and the date. (I'll go into the purpose of the account piece shortly.) Having the stats split by day means it's easy to calculate totals (or averages, unions, etc.) over arbitrary user-defined time periods. For example, a user could select the time range "Jan 1, 2011 - Feb 4, 2011", and we can give them accurate stats simply by summing the results of a mget of each day in the range.

Why Redis?

Now, that's all nice and easy, but it's not much of a selling point for Redis over any other key/value store (or even a good RDBMS). The real advantage Redis offers is powerful support for data structures and the operations you'd expect to use on them. The big selling point for us was the availability of sets (sorted and not). We need to track sets of unique users that do certain activities in certain time periods. An example of this would be the set of (unique) users who commented in a single day. Can you store sets pretty easily in other key/value stores? Sure you can. The difference for us is that our sets can easily contain tens of thousands of elements each. The benefit of a store like Redis is that sets are "first class", and you can ask the server to do operations like unions for you. Imagine if a user were to request stats for an arbitrary 17-day period: We would just do a zunionstore of all 17-day sets and get the cardinality of that new set, which is the total number of unique users who posted in that time period. If Redis didn't support sets or set operations, we'd have to fetch all 17-day sets ourselves then calculate it locally. The network I/O alone would kill performance.

Sharding

So what was the "account" bit in the key all about? Sharding is one of the pains of planning ahead when using Redis. Since we use (and depend on) things like set operations, we can't just arbitrarily split our data across different nodes. What if Jan 1 was on Node1 and Jan 2 on Node2? Doing a set union when the data isn't in memory takes us right back to doing network I/O of entire sets, killing performance. The first obvious step is to shard on sites (so my blog and Disqus' blog might be on different Redis nodes). No harm there, an easy win. But what if we're talking about Big Newspaper Inc., who have one Disqus account but 20 distinct Disqus sites? We could easily provide them stats per site with the current sharding scheme, but that's no fun. We wanted to offer "network-wide" stats. That is, the ability to see things like total comments, upvotes, replies, and even unique users across all of their sites at once, and for any arbitrary time period. That means all of a user's sites need to be hosted on the same Redis node. This is actually pretty trivial once you decide to do it (we just take the modulo of the owning user's ID against the number of nodes we have to decide which node to read/write from/to), and the bit in the key isn't strictly necessary, but we like to have it in there just incase we need to reason about them later.

Sessions

In short, we attempted to migrate our sessions from PostgreSQL to Redis but failed. Redis started as an in-memory database but added a virtual memory backend in (if I recall correctly) the 2.0 branch. While the VM backend helped, we found that it still wouldn't stay within the bounds we set, and would continually grow no matter what we set. We did report the issue but never came to a good solution in time. For example, we could give Redis an entire 12GB server and set the VM to 4GB, and given enough time (under high load, mind you) it would climb well above 12GB and start to swap, more or less killing our site. Since we didn't have a need for the powerful Redis datastructures here, we eventually decided to go with Membase because its main purpose is a simple disk-backed key/value store.

We look forward for Redis 2.4's "diskstore" which aims to solve the issues the VM backend has had.

Aggregators

We have many other counters and sets for all sites and users in Disqus network, outside of the complex and rich premium analytics data. There's nothing too fancy here, but again Redis has been our choice (often migrating away from PostgreSQL) when we need a high write capacity. As most (all?) large sites eventually find, our master database can be a bottleneck, and using SQL to increment rows that are already essentially key/value stores can be a waste. When (and only when!) it's more natural, we move denormalized counters and other data to Redis.

Experimentation

Lately we've been pushing the limits further with some in-house (maybe soon to be user-facing) projects that do all of the above at much larger and faster rates. I've been using Redis as storage for output from Hadoop jobs and we've been pushing data to other nodes to collect statistics on reads. If you imagine the full network size of Disqus being rather large in writes, you'd have to see a few services crumble when you try to do a write-per-request. ;)

Conclusion

All in all, I'd say we're big fans. Aside from the VM issues (which should only affect a very small percentage of users, and should be resolved in 2.4) it's been a pleasure to administrate even for our very small team. I wouldn't call us technically conservative by any means, but we have a decent amount of PostgreSQL expertise in house and we don't throw in new moving parts for no reason. When it's the right tool for the job, it's great. Huge props to Salvatore Sanfilippo, Pieter Noordhuis and others.

Two months using Android

2010-06-29T00:00:00Z

I've had the HTC Incredible for about two months now. It's been a great experience overall. Verizon's network has been very good (especially compared to AT&T) in Austin. I even had a week to use it in San Francisco and didn't experience any problems (a first!).

From what I can tell, the only iPhone 4G hardware advantages that I care about are the polished aluminum build and the display. The 3GS has 163 ppi and the 4G has 326, that's clearly a huge and much needed boost. The Incredible has 252 ppi, so it's not exactly like I'm living with the weak 3GS display, but more is always better (when a resolution independent UI is used, as both systems do).

Software-wise I'm still sold on Android. Even when I was using iOS I hardly used any apps. A browser, a twitter client, a music player, a password manager, and a maps application cover 99% of my use cases. I'm curious what else people are actually using (aside from games, which I don't care about). Just like the desktop, the web is where it's at for me.

Given the apps I use, I find the Android OS vastly superior. In my previous entry I explained that I prefer Android's pinch to zoom in the browser, and that definitely still stands. I've also learned that applications can extend the "Share Page" button, so I have an app ("plugin"?) to quickly and easily mark things in Read It Later. The fast and available network doesn't hurt my web experience, either.

My music and Twitter apps are basically the same between platforms. I don't really ask much of them and I'm not sure what they could do to really stand out for me.

Maps on Android, as I said before, is amazing. It was a huge selling point among friends that are using iPhones but up for a new phone soon. Sadly it sounds like they didn't update the maps app in iOS4 (via same disappointed friends).

Finally, I'm still loving the Android notification system and exposed filesystem (mostly for Dropbox syncing files to use in other apps).

On Emacs

2010-06-11T00:00:00Z

Or "that other OS."

When I was visiting the DISQUS office a few weeks ago my coworkers and I made it a point to tease each other about our awful tool choices (especially editors). It's also come up among some other friends recently, so I think now is as good a time as any to try and explain why I use Emacs (and think you should, too).

A little history: I've been using Emacs now for about a year and a half. Before that I used TextMate for about 2 years, and before that it was Vim for about 2 years. I should note that when I say "used" I mean as my day-to-day, minute-to-minute editor. I still make use of all 3 pretty regularly.

The most important aspect of Emacs is that it can (often easily) do pretty much whatever you want (or need) it to do. Some people take this to extremes, with browsers and more inside of Emacs. I won't defend that here because I believe in the right tool for the job. What does matter is that you can often recreate any other editor feature within Emacs, and more often than not someone has already done it for you.

Where people often go "wrong" is that they assume customization should be done after you master the basics. That's the first step toward the anti-Emacs club. I think the tutorial should go into init.el and basic customization as soon as opening and saving are out of the way. If you use Emacs without customizing it, you're doin' it wrong. If you do something that pains you two days in a row, you're doin' it wrong.

Here's an example: Emacs teaches you commands like C-n and C-p to move between lines. If you don't like that you are free (no, encouraged!) to not use those and bind different keys (in this case the arrow keys are enabled by default). The same goes for saving, copying, pasting, closing a buffer, etc. If you really can't stand the key combinations then change them! I've set my Emacs up with a very OS X muscle-memory friendly configuration, Command-S,C,X,Z,F,G,W,Q all do what you'd expect. This isn't a sin against Emacs, this is Emacs.

Once you've given yourself a pretty basic TextMate like editor that you can at least use, you start the slow day-over-day growth of your own config (and thus your own editor). You can see my config for some examples. When you miss something from another editor, add it. When you see something new in another editor, you can probably find it and add it. When something (anything) annoys the hell out of you, change it.

Can you do all of this in Vim? Sure, kinda. I just couldn't get very far with Vimscript before banging my head against a desk. I feel like the Emacs "add-on" (they really aren't add-ons) community is larger, faster, and more engrained in the culture. Elisp is the editor.

I'd like to point out that I'm pretty bad with Elisp, you don't need to become an expert either. Elisp is also a pretty bad Lisp in my opinion, but it's hard to point out another popular editor written entirely with customizability in mind in a language that's usable.

If you're a programmer I think you'll enjoy being able to program your own tools. And what do you use more than your text editor? You're missing out.

Note: Since this was mostly about the ideology of Emacs I hope and plan to do a little screencast comparing my current setup to another editor (probably TextMate) to explain what I've done with my own setup and the crazy features Emacs has that nothing else seems to.

HTC Incredible

2010-05-12T00:00:00Z

I recently made the switch from an iPhone 3G to the HTC Incredible (on Verizon). The following is simply a list of what I like and don't like, in no specific order.

The Bad

Screenshots

They aren't built in to Android yet. You have to plug in via USB and use dev tools. I didn't do this often, and haven't actually needed it yet. This is scheduled for an upcoming release.
Bookmarklets

They don't seem to work yet, also scheduled for an upcoming release.
Updates

I'm worried about the speed at which HTC will update my version of Android OS. With the iPhone you're guaranteed everyone will be updated on day 1 of a new release. Only time will tell, but it's been a sore point of Android in the past.
Podcasts

One of the few things I think iTunes does right is Podcast syncing. I haven't found an alternative that's on par, though I haven't looked hard. This is less of a pain since I moved because I'm always at home. Mobile Podcasts are best for a commute.

The Neutral

Flash

Flash Lite came installed, I was able to watch embedded YouTube clips. I disabled it. "Real" Flash 10 is coming to Android, I highly doubt I'll care, but it's a choice I wouldn't have otherwise.
Cost

$200 for the phone, and I believe my service plan costs are exactly the same as the iPhone on AT&T.
Big name apps

Which is to say, the ones I care about. They're all there. Google (everything), Bank of America, Amazon, Pandora, Dropbox, Evernote, Meebo, PayPal, Twitter, Yelp.
Multitasking

Coming to the iPhone, so I'll call it even. Really, really great for Pandora, IM, etc.
Multitouch

Yeah, it has it. It may be the first Android phone that gets it right. Again, it's on iPhone so we'll call it even.

The Good

Verizon

My calls don't drop. 3G is faster and with more, better coverage. Fuckin' A.
Fast

Network aside, the hardware is fast as hell. 1Ghz processor, 512MB RAM (that's 2x the 3GS, I assume the 4G will bump up to this).
Open Platform

No section 3.3.1. I develop because I enjoy it, and I have no interest someone telling me what editor I'm allowed to use. Another benefit is not being forced into one choice for much of anything. For example, you can swap out the default keyboard and many other pieces. The idea of competing keyboard software is just awesome to me.
Syncing

Built-in over-the-wire syncing of Google, Facebook, Twitter and Flickr accounts. I have not hand entered or computer-synced a single contact to this phone and yet it already has everyone I care about in it. The same goes for my calendar. Change the cloud and it changes the phone.
Notifications

I prefer the Android way. They stay in the top bar until you clear them (or respond), rather than a pop-up that is instantly lost. I can keep my "new @reply" notification up for hours until I'm ready to check it.
Speech-to-text

It's everywhere. I haven't used it much, but it seems handy for driving.
Exposed filesystem

Being able to sync Dropbox to a directory just like on a computer and then use other apps to easily access those files is just awesome.
iTunes-less sync

Related to the exposed filesystem, when I wanted some music on my phone I plugged it in and it mounted the 16GB MicroSD card. I dragged a bunch of files (gasp) to the volume, unmounted, and the Music app found it all automatically.
Hardware back button

And other buttons. I'm surprised I like these, actually. But the back button is especially handy, it brings some of the best of the web to every app on the phone.
Google Voice

Automatically use it for all outgoing calls, or have it ask you each time.
Browser zooming

This is hard to explain, but basically the browser will squish text content when you double tap to zoom. I always liked how the iPhone would snap to content, but this seems like a step forward. Snapping to desktop-sized content still usually means it's unreadable, and this extra squishing to bring it into readable font-size has been really, really handy.
Improved Google apps

Google is able (allowed?) to give extra love to their apps on Android. Maps is improved in a lot of ways, which is great considering how much I use it. The same goes for most others (and more of their apps are available, period).

A sudden interest in Java (the platform)

2008-11-23T00:00:00Z

I'll admit that I didn't have much interest in Java until Sun opened it up (thus creating the OpenJDK). Like many Python developers I've been pretty "spoiled", and reading Java code has generally left a bad taste in my mouth. In short: it's way too verbose for me, and I definitely prefer dynamic to static. I'm not saying I think anything less of Java developers, Java libraries, or Java programs - I just personally would prefer to use another language.

Java (the platform) has a lot of goodies, though. Instead of dreaming up a new language and writing your own rinky-dink VM, if you choose to target the JVM you start out with one of the largest (the largest?) sets of libraries available (assuming you make it easy to call Java from your language). You also get one of the world's best virtual machines, and thus you share a whole eco-system of tools like debuggers (I hear JSwat is great), DTrace, and more.

As of today you can already run a bunch of well known languages on the JVM - Jython (Python), JRuby (Ruby), Rhino) (JavaScript), Kahlua (Lua) in addition to "new" JVM-targeted languages like Clojure (a Lisp), Groovy (similar to Ruby) and Scala (a better Java).

It looks like there will be a lot of new features in Java 7, some of which were specifically added to support more / better / faster languages (other than Java) on the JVM. The most important one (for dynamic languages) is JSR 292 - support for a new bytecode called invokedynamic which, in short, seems to allow languages implementations to stop pretending like they have static type information when they really don't. From my loose understanding that means part of the implementation should be easier to write and the code should run faster.

Wouldn't it be cool to write an app in Jython but be able to call directly into a JRuby library that did what you needed in the best way? Or for all of the different parts in your larger app to be able to use the same DB interface or memcached library, so you don't have to worry about differences between the one you use for Python and the one you use for Ruby? At that point it seems like you really can use the best tool for the job, without a huge context switch of learning tons of new libraries, debug techniques, etc.

I know I'll be keeping my eye on all of this. For now I'm playing around with Clojure and from there I'll probably see what Django is like on Jython - seeing as that's what I'm most familiar with.

Using the Nginx memcached module with Django

2008-10-27T00:00:00Z

I decided to play around with the Nginx memcached module recently. It's a very interesting and simple mod that checks memcached for pages before falling back to your backend (or whatever you tell it to do, actually).

The simplicity does have a couple of drawbacks,

Cached bits are pulled out of memcached and nginx doesn't magically know their content-type. You can (should) provide a default content-type that will be used for your typical dynamic requests such as /foo and /bar/. If you want to cache non-[X]HTML content, you have to do some redirection hacking so that nginx thinks it knows what it's serving - I don't go into that here as I'm only caching my dynamic HTML requests.

The second issue is that you have to handle caching (and invalidation) inside of your app. This brings with it all of the normal cache invalidation issues. Since this is a very simple blog I was able to get away with a quick hack, and I haven't figured out how to best handle it on larger sites. On the other hand, you control the memcached timeouts, meaning even complex sites could use this with only time-based invalidation if they decided X seconds of dirty cache were alright for Y URLs.

nginx.conf

I think this part is pretty self explanatory. I only check memcached on GET requests, and I have chosen a key prefix of NG: so that I don't pollute the hash table (as the URL is used for the rest of the key).

user www-data www-data;
worker_processes 4;
pid /var/run/nginx.pid;

error_log /var/log/nginx/error.log;

events {
  worker_connections 1024;
}

http {
  upstream backend {
    server 127.0.0.1:8000 weight=1;
  }

  # pass Host header on to the backend
  proxy_set_header Host $host;
  # set X-Forwarded-For header so the backend knows who
  # actually made the request
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  # don't send version in response headers
  server_tokens off;

  include /usr/local/conf/mime.types;

  # fallback if none matches
  default_type application/octet-stream;

  # about: http://www.baus.net/on-tcp_cork
  tcp_nopush on;
  sendfile on;

  gzip on;
  gzip_min_length 1000;
  gzip_proxied any;
  gzip_types text/css text/plain application/atom+xml application/x-javascript;
  gzip_vary on;

  server {
    listen 80;
    server_name bretthoerner.com;
    root /a/bretthoerner.com/root;

    location ~ [^/]$ {
        # catches paths that don't end in / (typically media)
        # if there is no matching static file in the root
        # proxy the request to the backend
        if (!-f $request_filename) {
          proxy_pass http://backend;
          break;
        }
    }

    location / {
      # don't check memcached on POST, etc
      if ($request_method != GET) {
        proxy_pass http://backend;
        break;
      }

      # pages fetched from memcached don't know their
      # own mime type
      default_type "text/html; charset=utf-8";

      # use same prefix as backend does
      set $memcached_key "NG:$uri";

      # memcached location
      memcached_pass localhost:11211;

      # 404 for cache miss
      # 502 for memcached down
      error_page 404 502 = @cache_miss;
    }

    location @cache_miss {
        proxy_pass http://backend;
    }
  }

  server {
      listen 80;
      server_name media.bretthoerner.com;
      root /a/bretthoerner.com/media;
  }
}

Django middleware for setting cache

This is where the cache is set. On each Django response I check that,

The request is a GET
The request is not to /admin
The request is a success

Then you simply set the cache where nginx will look,

class NginxMemcacheMiddleWare:
    def process_response(self, request, response):
        path = request.get_full_path()

        if request.method != "GET" \
           or (path.startswith('/admin') and not request.user.is_anonymous()) \
           or response.status_code != 200:
            return response

        # settings.NGINX_CACHE_PREFIX == 'NG', just like nginx.conf
        key = "%s:%s" % (settings.NGINX_CACHE_PREFIX, path)
        cache.set(key, response.content)

        return response

Django signal for invalidating cache

Cache invalidation is freakin' hard, especially on a more complicated site. As I said previously, this is mostly a hack, it isn't the sort of thing that could scale (not in performance, but code and development). I'm trying to think of a better way to handle it, but describing and maintaining dependencies in a code-base is tough.

Anyway, the following should also be pretty self explanatory. When an object is saved, the URL for the object itself is invalidated (that's the easy part). Now the tough question is - what else needs to change? For my simple blog I've just hardcoded these values because I only have one model I really care about (Entry). I invalidate the homepage and all archive pages, so that the next request to any will have the latest title / body for all entries.

def delete_memcache_keys(sender, instance, created, **kwargs):
    keys = []

    if hasattr(instance, "get_absolute_url"):
        keys.append(instance.get_absolute_url())

    if isinstance(instance, Entry):
        date = instance.pub_date

        keys.extend(["/",
                     "/blog/",
                     "/blog/%s/" % date_format(date, "Y"),
                     "/blog/%s/" % date_format(date, "Y/b"),
                     "/blog/%s/" % date_format(date, "Y/b/d")
                    ])

    for key in keys:
        cache.delete("%s:%s" % (settings.NGINX_CACHE_PREFIX, key))


signals.post_save.connect(delete_memcache_keys)

Benchmarks and conclusion

So how much faster is it? Well, there's a little hitch when you use a basic blog to test this stuff out - it's already fast as hell. I'm using sqlite and am pretty confident the entire 150K database is in memory. A typical request does maybe 1 simple query and returns the pre-rendered entry field from the database. What that all means is that the largest cost of requesting a page (on this site) is traffic over the wire, e.g. waiting for the bits.

A more complex app would be a much different story, although I don't have one to try it on. I will say that if I added a simple time.sleep(.1) (100ms) to my index view - in order to simulate more work that would have to be done on a real site - I went from ~13 req/s to ~27 req/s. (Keep in mind I'm on a small VPS and I was only doing 3 concurrent requests at a time because I don't have many pre-forked Apache children created, and I wanted to keep both tests 1:1) Still, a pretty big deal, no?

Kickstarting a better Ubuntu server environment

2008-10-27T00:00:00Z

I decided to clean up and post my notes on how I setup my Ubuntu server environment (I happen to use Slicehost, but it shouldn't matter).

I began with an "empty" Ubuntu 8.04 machine and the root password, at the end I have:

Disabled root login via ssh
Created your own user who can login via ssh public-key authentication
Updated your system with the latest Ubuntu packages
Installed various packages that most people expect on a Linux server
Setup simple mail forwarding (I forward bretthoerner.com to my Gmail account)

This assumes you already have a domain pointed at your server and that you have ssh keys stored locally in ~/.ssh.

Replace /bin/sh symlink with a more fully featured shell

$ ln -sf /bin/bash /bin/sh

Create your user

# Add USER (using bash, with a home, in groups USER, users, and sudo)
$ useradd -G users,sudo -m -c "FULL NAME" -s /bin/bash USER
# Configure sudo to be password-less for users in group sudo
$ visudo
# Uncomment %sudo ALL=NOPASSWD: ALL

Setup ssh public-key login

# Login as user
$ su - USER

# Configure `ssh` public key login
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ vim ~/.ssh/authorized_keys
# Paste in your public key (from local ~/.ssh/id_[rd]sa.pub)

# Exit out of user
$ exit
# Exit out of root
$ exit

# Test login from your local machine
(local)$ ssh USER@DOMAIN # should work without a password

Disable root login

# Test sudo from your user before you disable root login
$ sudo whoami # should print 'root'

# Disable root ssh and password-based logins
$ sudo vim /etc/ssh/sshd_config
# Change "PermitRootLogin yes" to "PermitRootLogin no"
# Change "#PasswordAuthentication yes" to "PasswordAuthentication no"

# Reload the ssh config
$ sudo /etc/init.d/ssh reload

# Remove root's password
$ sudo passwd -d root

Update the local package repository and packages on your system

$ sudo apt-get update
$ sudo apt-get dist-upgrade -y

Install packages you'd expect to have on Linux

# Favorite editor and shell niceness
$ sudo apt-get install -y bash-completion command-not-found \
                          emacs-snapshot-nox exuberant-ctags vim-nox

# Postfix
$ sudo apt-get install -y postfix procmail
# Just hit enter through the setup for now

# Linux basics
$ sudo apt-get install -y dnsutils file info logrotate lsof \
                          mailx mlocate openssl rsync screen unzip

# Developer basics
$ sudo apt-get install -y autoconf build-essential cdecl colordiff \
                          git-core git-svn libtool make patch subversion

Re-configure Postfix

$ sudo dpkg-reconfigure postfix

General Type: Internet Site
System mail name: DOMAIN
Root receipt: USER
Destinations: localhost
Force synchronous updates: no
Local networks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
Use procmail for local delivery: yes
Mailbox size limit: 0
Local address extension character: +
Internet protocols to use: all

Edit the Postfix config

$ sudo vim /etc/postfix/main.cf
# Add the following
virtual_alias_domains = DOMAIN
virtual_alias_maps = hash:/etc/postfix/virtual
smtpd_helo_required = yes
strict_rfc821_envelopes = yes
disable_vrfy_command = yes

Create virtual users and aliases that forward to you

$ sudo vim /etc/postfix/virtual
# Add one USER@VIRTUAL_DOMAIN EMAIL_TO_FORWARD_TO per line, e.g.:
user@example.com user@gmail.com

$ sudo postmap /etc/postfix/virtual

$ sudo vim /etc/aliases
# Append the following line so that root mail goes to USER@DOMAIN
root: USER

$ sudo newaliases

# Restart postfix for changes to take effect
$ sudo /etc/init.d/postfix restart

That's it. I'll follow this up with more information on how I setup Apache, mod_wsgi, Django, nginx, etc.

Configs for Nginx and Apache with mod_wsgi

2008-10-09T00:00:00Z

I cleaned up my configuration files for nginx and Apache (running mod_wsgi and Django). It's pretty straight forward stuff. I use nginx to serve static files and proxy all other requests (like this page) back to Apache.

nginx.conf

user www-data www-data;
worker_processes 4;
pid /var/run/nginx.pid;

error_log /var/log/nginx/error.log;

events {
  worker_connections 1024;
}

http {
  # don't send version in response headers
  server_tokens off;

  include /etc/nginx/mime.types;

  # fallback if none matches
  default_type application/octet-stream;

  # about: http://www.baus.net/on-tcp_cork
  tcp_nopush on;
  sendfile on;

  gzip on;
  gzip_min_length 1000;
  gzip_proxied any;
  gzip_types text/css text/plain application/atom+xml application/x-javascript;
  gzip_vary on;

  server {
    listen 80;
    server_name DOMAIN;
    root /a/DOMAIN/root;

    location / {
      # pass Host header on to the backend
      proxy_set_header Host $host;
      # set X-Forwarded-For header so the backend knows who
      # actually made the request
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # if there is no matching static file in the root
      # proxy the request to the backend
      if (!-f $request_filename) {
        proxy_pass http://localhost:8000;
      }
    }
  }

  server {
      listen 80;
      server_name media.DOMAIN;
      root /a/DOMAIN/media;
  }
}

apache2.conf

Listen 127.0.0.1:8000

# should be name clients would use reach this host
# and not just the address you happen to bind to
ServerName DOMAIN:80
ServerRoot "/etc/apache2"

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
PidFile ${APACHE_PID_FILE}
LockFile /var/lock/apache2/accept.lock

# nginx doesn't support backend keep-alive yet
KeepAlive Off

# prefork-mpm settings
StartServers          5
MinSpareServers       5
MaxSpareServers       5
MaxClients            5
MaxRequestsPerChild   5000

DefaultType application/octet-stream

ErrorLog /var/log/apache2/error.log
LogLevel warn

# log x-forwarded-for because the request will always
# come from nginx, not the actual user
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog /var/log/apache2/access.log combined

LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so

# BASELINE is an empty virtual environment that doesn't
# fallback to the global site-packages directory.  Using it
# as the Python home ensures that all WSGI apps run out of
# their own virtualenv, so nothing pollutes anything else.
WSGIPythonHome /a/BASELINE

NameVirtualHost 127.0.0.1:8000

# default virtualhost so that _real_ sites are only
# served when specifically requested
<VirtualHost 127.0.0.1:8000>
    ServerName 127.0.0.1
    DocumentRoot /var/www
</VirtualHost>

<VirtualHost 127.0.0.1:8000>
    ServerName DOMAIN
    DocumentRoot /a/DOMAIN/root
    WSGIScriptAlias / /a/DOMAIN/etc/wsgi.py
</VirtualHost>

wsgi.py

import os
import site

site.addsitedir('/a/DOMAIN/lib/python2.5/site-packages/')
os.environ['DJANGO_SETTINGS_MODULE'] = 'PROJECT.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

Recent Blog Posts

Django at Scale slides

Austin

Indoors

Outdoors

Food

Music and Nightlife

Redis at Disqus

Analytics

Why Redis?

Sharding

Sessions

Aggregators

Experimentation

Conclusion

Two months using Android

On Emacs

HTC Incredible

The Bad

The Neutral

The Good

A sudden interest in Java (the platform)

Using the Nginx memcached module with Django

nginx.conf

Django middleware for setting cache

Django signal for invalidating cache

Benchmarks and conclusion

Kickstarting a better Ubuntu server environment

Initial login

Replace /bin/sh symlink with a more fully featured shell

Create your user

Setup ssh public-key login

Disable root login

Update the local package repository and packages on your system

Install packages you'd expect to have on Linux

Re-configure Postfix

Edit the Postfix config

Create virtual users and aliases that forward to you

Configs for Nginx and Apache with mod_wsgi

nginx.conf

apache2.conf

wsgi.py