If you’re managing your own servers, and you don’t use a tool like Chef, you’re crazy. It’s just that simple.

We’ve been using Chef here at Yipit for about 6 months, and when I think about provisioning a new server with our old load book, I cringe.

There were some pretty high upfront costs to learning Chef, especially since no one here had any real Ruby experience (Chef is written in Ruby), but the time we invested into getting set up with Chef has been 100% worth it.

I’m hoping this post will help people get up and running with Django and Chef as quickly as possible. Opscode has their own django-quick-start repository, but I think it’s too complex if you’re not familiar with Chef. This tutorial will cover:

• Getting set up with Opscode, a hosted chef-server
• Installing Ruby, Chef, and some knife plugins
• Setting up your chef-repo and installing a Django/github quickstart cookbook
• Using a Python script to deploy to ec2

If you don’t understand what any of those things mean, Opscode maintains a pretty good wiki.

Before We Start

This tutorial makes the following assumptions about your systems:

• Locally, you run a Mac.
• Remotely, you want to use Ubuntu on AWS
• Your AWS account has security groups named “web” and “linux” that allow access on ports 22 and 80
• The private key for the AWS ssh key you define in add_server.py (we’ll get to that later) is located in ~/.ssh/ on your local machine
• Your Django app contains a pip requirements file at conf/external_apps.txt (and that file contains gunicorn)
• Your Django app contains a gunicorn configuration file at conf/gunicorn/gunicorn.conf
• You’re interested in learning about Chef and managing your own servers. Seriously. If you’re just looking for a simple deployment solution, use Heroku. If you want to manage and automate your own infrastructure, use Chef

If you’re not familiar with AWS security groups and key pairs, this tutorial should help. I’ve also provided a simple “Hello World” Django application that I use as an example.

Get the Gems

I like using homebrew for everything, so even though your mac comes with ruby…

brew install ruby
gem install -n /usr/local/bin/ chef
gem install knife-ec2
gem install knife-github-cookbooks

Set up a chef-repo

We’re going to be adding some python scripts to help us manage our servers, so I like to set up my chef-repo in a virtualenv. From wherever you keep your virtualenvs…

virtualenv chef-env
cd chef-env
git clone https://github.com/opscode/chef-repo.git
cd chef-repo

Now you have the barebones Chef repository distributed by opscode.

Get started with opscode and your knife.rb file

Next you’ll want to get setup with Opscode to manage your chef server. You can create an account at opscode.com.

After you’ve signed up, download your private key. You can download it by clicking on your username on the top right of the console, and then following the link to “get private key”.

Next you’ll need to set up an organization and download the organization validator key. You’ll also want to to download the knife.rb file that opscode will generate for you.

Lastly, throw it all in a .chef directory in your chef-reop.

You should also append this code to your knife.rb:

This changes the amount of time knife will wait for ec2 to spawn a new server for you. I’ve found that sometimes the default wait time isn’t long enough for ebs-backed instances.

Setting up your project

Next we need to set up chef to actually do something. Let’s start by installing a Django quick start cookbook I put together. This cookbook will install a django application behind nginx and gunicorn, which will be managed by supervisor. You should read through the commented code for an explanation of how it works.

knife cookbook github install Yipit/djangoquickstart-cookbook

This cookbook has a dependency on the python cookbook, so we’ll want to install that too.

knife cookbook site install python

Note that “site” version of the install command will handle dependencies automatically, while the github command will not. When you use “site”, however, you’re limited to cookbooks distributed by opscode.

The next thing we’ll want to do is define a role. In your chef-repo, put something like this in roles/web.rb:

Get Some Python In there

Knife has a pretty good plugin system, but it requires you to write Ruby. Writing Ruby is worth it in the cookbooks because it’s the only option, but for helper scripts, I just use python. Aside from being more comfortable with the language, familiarity with the libraries is a huge plus. Here’s a script similar to the one we use to add servers at Yipit:

Fill in your AWS keys and put this code into a file called add_server.py. This script requires boto and pychef, so from within your virtualenv:

pip install boto pychef

I keep this file in right in the chef repo.

The last thing we need to do before we get started is upload our roles and cookbooks to the chef server.

knife cookbook upload --all
knife role from file roles/web.rb

Now, if you run the add_server.py script, you should be able to bootstrap your own web server running your django application through gunicorn behind nginx. The add server script does restart the machine, so you may need to wait a minute or two.

Hopefully you can visit your new web server and see your application.

1. There is a clear and predefined relationship between the two columns - One columns is the parent, and the other is the child. This relationship limits the amount of content available in the lesser of the two panels. Most implementations limit the height of the child column to the height of the browser window and use an overflow for additional content.
2. Navigation is inconsistant – The page will react differently depending on which browser element is in focus. Because the child column usually uses overflow for extra content, scrolling up or down does not guarantee getting any additional content located in that direction in both columns.

I’ve built a proof of concept demo of a two column layout with no hierarchy between the columns.

You can check it out here.

Last week we rolled out a new design for yipit, which I’m really excited about. It looks awesome. Sam Brown did an amazing job. In a few different places we decided to use modal forms that would submit via Ajax. I think technique provides great UX, but  if the forms require server-side validation,  displaying the appropriate HTML can be a bit of a headache.

First Attempts

The initial strategy was to have the server respond with JSON containing any errors and then show them appropriately to the user. This is a totally valid approach, but every form is different, which meant writing form specific code both server and client-side. I also find injecting HTML into an existing page to be pretty tedious, so having to write JavaScript to do exactly that for every form was not a solution I was happy with.

In addition, if the form did validate, it was unclear where the success HTML should live. We could have the server return it, but this made for an awkward transaction since we also needed to pass back whether or not the form had validated. The HTML could be part of the JavaScript, but this would break MVC and not allow us to leverage Django’s templating system. Either way, the django views we were using looked very different than the views for standard pages, and having to switch gears slows down development.

I wanted to find a solution that met these criteria:

1. Minimal custom JavaScript
2. Familiar server-side code
3. Adhere to MVC

Change of Plans

Fancybox makes it really easy to draw an overlay with HTML returned via Ajax. Fancybox also only allows one overlay at a time. This is a limitation, but it does make it really easy to replace one overlay with another. Leveraging both these behaviors, we realized we could have every call to the server result in a new fancybox that would replace one currently being displayed. This way we cold write traditional Django views to handle the Ajax request and take full advantage of the Django templating system, including the display of form errors.

The first iteration of this involved every fancybox containing the javascript necessary to launch the next fancybox. This worked really well on Firefox, Safari, and Chrome, but IE was no so cooperative. In IE, we could show a second fancybox, if that fancybox contained JavaScript, the browser would crash. Also, it didn’t satisfy the requirement of not having to write custom JavaScript for every form. Using a little client-side recursion (who knew there were practical applications?) and a couple of guidelines, we were able to launch new fancyboxes without new scripts.

Guidelines

1. Every <a> tag that launches a fancybox form must share common class.
2. Every fancybox that contains a form must share a common class.
3. Every form must have an explicit action containing the destination URL for the Ajax request.
4. Every response contains the html for a new fancybox (and must adhere to guidelines 2 and 3 to allow for chaining)

Sample Code

Django View:
Django Template:
Javascript (assumes jQuery 1.4 and fancybox 1.3.1):

Any server-side implementation will work if you adhere to the guidelines. The real meat is in the JavaScript .

We can run this query to see what Authors are available in a particular bookstore:

What’s The Problem?

That QuerySet produces the following SQL:

Look at the subquery. There’s an unnecessary join against the table book. The information returned by that subquery is the ids of books associated with a particular bookstore. This information is available directly from M2M table bookstore_book.

So What Can We Do?

We’re really looking at two problems here:

1. Django doesn’t realize we only need the ids of the books and can obtain them without accessing the book table. (I’m not going to dive into this one)
2. Even if the developer realizes, there is no easy way to extract that information using the ORM.

This second issue can be solved by subclassing two Django classes and creating an attribute that collects the data we need in a more efficicent way. Here’s the code I came up with:

This NewManyToManyField will give the ManyRelatedManager accessed through the attribute books the attribute id_list, which is a method that returns a QuerySet containing the list of the ids of the target model that have a connection to the current object. In this case it returns a list of book ids in the B&N book store. If we declare our models using the NewManyToManyField we can take advantage of this new attribute:

Now we can use the following code:

Which produces this SQL

Check out the subquery. We’ve saved a JOIN against the book table. This may seem like an insignificant optimization, but if there are a lot of people on your site  visting pages that require this type of query, losing that JOIN can really speed things up.

How does it work?

During class creation of a model, a method called contribute_to_class is called for every database field. This method gives the class an attribute that is the name of the field and gives an instance access to the value of that field. For ManyToManyFields this attribute is a ReverseManyRelatedObjectsDescriptor object, which returns an instance of ManyRelatedManager for accessing the related rows of the related table. The code subclasses ReverseManyRelatedObjectsDescriptor, catches the ManyToManyRelated manager on the way out, and then assigns it the attribute id_list, which is method that returns a QuerySet that only accesses the M2M table and contains the ids of the related objects.

There is still a lot of room for improvement, but this gets the job done. For starters, the ManyRelatedManager gets created behind the scenes here and is then overwritten. Also, id_list isn’t a bound method of the ManyRelatedManager. I know both this issues could be solved by jumping down one more level and also subclassing ManyRelatedManager, but I think this is good enough for now.

One way to evaluate  intelligence is to consider one’s ability to think. It’s not a great measurement for people since everyone (or so they say) can think, but it is an interesting way to evaluate the intelligence of machines. Because thinking is inherently human, Alan Turing proposed we could evaluate a machine’s ability to think through its ability to act as a human. If the machine could successfully impersonate a human, the machine would demonstrate its ability to think, and thus its intelligence. This is the test he proposed:

The Turing test is a proposal for a test of a machine’s ability to demonstrate intelligence. It proceeds as follows: a human judge engages in a natural language conversation with one human and one machine, each of which tries to appear human. All participants are placed in isolated locations. If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test. In order to test the machine’s intelligence rather than its ability to render words into audio, the conversation is limited to a text-only channel such as a computer keyboard and screen. (Wikipedia)

No machine has ever passed the Turing Test.

And it’s certainly not for lack of trying. If you were on AIM in the late 90′s, you most likely participated in a Turing Test. Remember SmarterChild (or any of the other bots that popped on on instant messaging services)? If you do, then  I’m sure you’ll agree with me that they were not very convincing human representations.

But what if we modernized the rules a little bit, and conducted a Twitter Turing Test — Given two twitter users, one human and one machine, could a third twitter user, who follows and is followed by both, determine which one is human and which one is not? The twitter bot would have access to the twitter api, and I imagine the algorithm would employ some of these strategies:

• Copy tweets that originiate from close social proximity to the targeted user, but wouldn’t show up in the user’s stream.
• Copy tweets that originiate from from a certain geographical area to give the appearance of living there
• Participate in trending topics, both global and local
• Share links found through StumbleUpon, Digg, or other social bookmarking services.
• Retweet popular tweets

I believe that someone could build a machine to pass the Twitter Turing Test today . Sure, it’s  less intimate (and therefore less challenging) than the original Turing Test, but it’s also  more analogous to our current forms of virtual communication. Social media has created a type of communication that is less synchronous, less personal, and (if you agree with Alan Turing about what the Turing Test proves) involves less thinking.

If social media is what will allow machines to pass the Turing Test, and the Turing Test is an indication of a machine’s intelligence, then what does that say about social media’s impact on our intelligence?

1. Teenagers have never used email as their primary mode of social communication, or any form of communication for that matter. Before the explosion of social media, most teenagers opted to use instant messaging to communicate online. Only after people move into a situation where they are forced to coordinate with others who they don’t know very well or don’t have very much face time with (think college professor or business contact) does email become the preferred way to correspond. Once we adjust to email, we start to use it socially. 
   
   When old fashioned snail mail was the standard, parents got a lot of mail; kids did not. Teenagers didn’t really need mail then, and they don’t really need email now. The only reason teenagers have email addresses is to sign up for online services, which brings me to my next point….




2. We need email as the greatest common divisor of the web. Now that oAuth is widely adopted, email appears to be less important. You can register for almost every new service through twitter, or facebook, or foursqaure, but when push comes to shove, every account you have can be traced back to your email address. It is your unique identifier. While I agree people will start to use email less often, it will never cease to exist. It’s like social security. We’re probably all going to start getting less of it; we may get none of it; but the system isn’t going anywhere because too many other processes would fail if we couldn’t be identified by it.

I’m sure Sheryl Sandberg hopes email goes away; it would only drive more traffic to facebook. In reality, it isn’t going anywhere.

def has_changed(instance, field):
    if not instance.pk:
        return False
    old_value = instance.__class__._default_manager.\
             filter(pk=instance.pk).values(field).get()[field]
    return not getattr(instance, field) == old_value

Here’s how could be used:

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)
    succumbs_to_peer_pressure = models.BooleanField(default=False)

    def save(self, *args, **kwargs):
        if has_changed(self, 'has_star'):
            self.succumbs_to_peer_pressure = True
        super(Sneetch, self).save(*args, **kwargs)

Let me know if you improve it…

Most companies put themselves at an even greater disadvantage in recruiting by writing awful job listings. If you include any of these 3 things, I promise to not be interested:

1. We’re in Stealth Mode — Listen, I understand first-mover advantage. I took Management 101 freshman year of college too. Your company being in stealth mode signals a couple of things to me:
   • There is going to be way too much back and forth before I even find out what your company does. First we’re going to send a couple of emails – You want to make sure I’m actually a developer and not just trying to steal your idea. Then we’re going to have a phone interview – You need to make sure I’d be a good fit. Next we need to do an in person interview – It’s very important that this interview is in person so that I can sign an NDA. Finally, you’ll share your mediocre idea with me.
   • You don’t really understand how startups work. I used to believe in stealth mode. I had all these ideas; all I had to do was pick one of them, and I’d be a millionaire. Guess what? Plenty of other people also had these ideas, but they actually made something happen. Now they’re millionaires. Success is about execution. In order to execute,  you’ll also need traction. If you are in stealth mode, you cannot acquire users.

   Note: Saying that “This product is going to change the way people [ambiguous verb]” or “We’re the next (Google|Twitter|Facebook)” does not count as telling me what your product doesn’t do yet.

2. Absurd, Hyperbolic Nouns — Unless you’re trying to recruit a 16 year-old out of his mother’s basement, I do not anticipate you having success looking for JavaScript Ninjas, Ruby Gurus, of Python Rockstars. Despite the apparent industry acceptance of the word, I’m also less likely to take your company seriously if you’re looking for a “hacker”.  I’m not impressed by your misunderstanding of developer speak.
3. No Equity Offer — Anyone who wants to work at a start-up for solely financial benefiet is not a good candidate. On the other hand, anyone who is willing to work for a start-up and not get a piece of the upside is a sucker. There’s a saying I love: “Equity is like shit; if you spread it around, beautiful things grow, but if you keep it all in one place, it just sits in a pile and stinks.” I have no desire to help you increase the value of your stinking pile of shit.

Here’s my make nice:

There are some non-technical founders with really good ideas trying to launch companies. Unfortunately, many of them don’t know how to attract the technical support they need. Maybe this list will help.

There’s a delicious pasta place on 22nd and 3rd called lamarca. There are two entrances: one into a cheese shop and one into a sit down restaurant. The first time I went, my group walked in and tried to figure out where we should be going. There was no menu in the cheese shop, so we decided on the restaurant. The food was great, but the pasta lunch ended up costing us each around $15, which was surprising since we had heard it was very inexpensive. Since then I’ve learned they have a 3 tiered pricing scheme:

• If you order form a waitress, it costs about $15 and you get a roll, a small side salad, a fancy bowl and a metal fork.
• If you order from the cheese shop and sit down in the restaurant, it costs about $8, but you get a different bowl and a plastic fork
• If you order to-go from the cheese shop, it costs about $7; no bowl, but you still get the fork

I’ve been back to lamarca because the food is great and affordable (now that I’m on the inside), but I never feel good about giving them my money. I’d go there a lot more often if they had been straightforward with their pricing the first time I went.

I hate when businesses don’t make pricing information available to their customers and then try to take advantage of the fact that their customers and uninformed. While these tactics do promote price discrimination and, therefore, short term profits, they are myopic and bad for business.

When Ben and I ran Wash-U-Wash, we  prided ourselves on customer satisfaction. One year many of our customers gave us more pounds of laundry then they had paid for. While most customers never asked us about the status of their account, we were also irresponsible about reaching out to our customers when they went over. Contractually, we had the right to collect a significant amount of money; however, we felt that we weren’t transparent enough about account balances. We decided to inform the relevant customers that they had gone over their poundage, but that we would forgive the balance on their account. This proved to be a good decision as over 70% of our non graduating customer base returned the next year.

Being ambiguous about pricing is a great way to irritate and ultimately drive away your customer base.

For our example, we’ll use Sneetches. Let’s start with a simple model:

from django.db import models

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)

Now Sneetches are bigots. Obviously having a star is either unequivocally better or worse than not having a star, but preference can change quickly. We want to to be able to change our code just as rapidly. To achieve this we can build a custom manager to keep our logic in one place. It will look something like this:

class SneetchManager(models.Manager):
    def get_query_set(self):
        return self.filter(has_star=True)

We’ll also need to assign this manager to the model, so here’s what we’ll end up with:

from django.db import models

class SuperiorSneetchManager(models.Manager):
    def get_query_set(self):
        return self.filter(has_star=True)

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)

    superior_sneetches = SuperiorSneetchManager()

This is what django recommends in the documentation for writing managers, and it’s pretty good. I can make this call wherever I want:

    Sneetch.superior_sneetches.all()

And because of the way we organized the code, we can change our mind about whether Sneetches with stars on their bellies are better than Sneetches without stars on thars.

Overriding the get_query_set method is great, but it’s static. We can’t pass in any arguments. Maybe we want to work with either the superior or the inferior Sneetches. To do this, we can make a method that is an attribute of a manager. So let’s rewrite the manager, and this time we’ll just assign it to the the objects attribute of our class. Since models, by default, have an attribute objects which is the class models.Manager, we can override this attribute and since we will be replacing it with a class that extends models.Manager, we won’t lose any functionality:

class SneetchManager(models.Manager):
    def segregate(self, superior=True):
        if superior:
            return self.filter(has_star=True)
        else:
            return self.filter(has_star=False)

class Sneetch(models.Model):
    has_star = models.BooleanField(default=False)

    objects = SneetchManager()

Now we have some choice. We can make either of these calls:

    Sneetch.objects.segregate(superior=True)
    Sneetch.objects.segregate(superior=False)

We almost have something really useful here, but we have a glaring limitation, segregate is not a chainable attribute. Only Sneetch.objects knows about segregate. Take a look at this code:

    Sneetch.objects.all().segregate(superior=True)

This raises an AtrributeError because all returns a QuerySet, which has no attribute segregate. We can overcome this by defining our own QuerySet that does have an attribute segregate.

    from django.db.models.query import QuerySet

    class SneetchQuerySet(QuerySet):
        def segregate(self, superior=True):
            if superior:
                return self.filter(has_star=True)
            else:
                return self.filter(has_star=False)

We know from the first example, that managers always call get_query_set, so let’s leverage that and write a manager that gets our SneetchQuerySet into the action.

class SneetchManager(models.Manager):
    def get_query_set(self):
        return SneetchQuerySet(self.model)

Now when Sneetch.objects calls get_query_set it will return a SneetchQuerySet which has an attribute segregate. We have on final step before we put it all together.

Sneetch.objects still doesn’t have an attribute segregate so our original call of Sneetch.objects.segregate(superior=False) will fail. If you read the Django source, you will see that models.Manager defines all the documented methods of QuerySet. Here is an example (copied directly from the source of v1.1):

    def filter(self, *args, **kwargs):
        return self.get_query_set().filter(*args, **kwargs)

We could mimic this behavior and allow our manager to call segregate by defining a segregate attribute for the manager. That is perfectly legal, but it would also force us to define any new methods we add to SneetchQuerySet in two places. Instead, we can use the magic python method __getattr__. I’ll save an explanation of __getattr__ for another post, but you can check the docs.

Here’s our final product:

    from django.db import models
    from django.db.models.query import QuerySet

    class SneetchQuerySet(QuerySet):
        def segregate(self, superior=True):
            if superior:
                return self.filter(has_star=True)
            else:
                return self.filter(has_star=False)

    class SneetchManager(models.Manager):
        def get_query_set(self):
            return SneetchQuerySet(self.model)
        def __getattr__(self, name):
            return getattr(self.get_query_set(), name)

    class Sneetch(models.Model):
        has_star = models.BooleanField(default=False)

        objects = SneetchManager()

    #Both These are Valid

    Sneetch.objects.segregate()
    Sneetch.objects.filter(**kwargs).segregate()

And that’s it. For any model that has repetitive/complex queries, I almost always write a custom QuerySet and Manager.