Surfing in Kansas posts category feed

Why Read the Docs matters

Documenting projects is hard, hosting them shouldn't be. Read the Docs was created to make hosting documentation simple. I think that we have solved this problem well, but now we need to start thinking about the larger picture.

Along with hosting, Read the Docs was created with 2 other main goals. One was to encourage people to write documentation, by removing the barrier of entry of hosting. The other was to create a central platform for people to find documentation. Having a shared platform for all documentation allows for innovation at the platform level, allowing work to be done once and benefit everyone. Having run the site for over a year now, I think there is a third thing that we should be striving for. That is to make the quality of documentation better.

I think that we can help a documentation culture flourish within the open source world. Django is a shining example of what a project with great documentation can do, and it has a community that values docs more than the norm. I think we can help spread this culture throughout the Python world, and beyond. This has already started, and I want to think about how something like RTD can help.

What we can do to help

I think that having a guide for writing useful documentation would be a great step towards helping people along the path of documentation enlightenment. Jacob Kaplan-Moss has started down this road with his blog series and Pycon 2011 talk on this subject. I think that we could start by collecting these into a section of the site.

We could build on top of that great start with simple guides for how to get started with Sphinx, best practices for documentation, and providing a general place to learn more about how to write good documentation. Since we host a lot of documentation, we could point to live examples of techniques, and provide helpers for people to enable the techniques.

I have started a reStructedText Philosophy document that is meant to help people understand the ideas behind how reST works, so that it isn't as mystifying. This reST cheatsteet also appears to have similar goals. These are a very basic start, and I think some more along these lines would really help a lot of people get over the barrier to starting and continuing to write good documentation.

I think that we could also help create contributors to projects, if we could find an easy way to provide patches to documentation. If you could go to the project documentation, and fix small typos, or help add a paragraph in the tutorial, it would lower the bar to helping.

However, it isn't a wiki. These changes would be represented to the project author as pull requests in their VCS, and they would still be responsible for tending the garden. This gets rid of the "Just Edit The Wiki" solution of documentation, and also helps new contributors get started in an easier fashion.

The Plone community has built a proof of concept, linking to Github's edit pages for the current document. I think we can integrate this at the platform level, and make it available to everyone.

Want to help?

Read the Docs is open source. You can help by writing docs for the site, writing code for the site, or just writing documentation in general. People can also help just by using the site, and reporting bugs. Telling us how to make the site better helps everyone in the long run. Come join us on Freenode in the #readthedocs channel as well.

Another area that we're hurting is in the design front. We have been adding features over time, and the design of the site is getting a bit strained. Having someone with a good sense of design help re-think and re-architect some of the features and ideas that we've been working on I think would help a lot.

A lot of the RTD contributors will be at Pycon 2012, where we will be having a sprint on the site. If you want to get started contributing, that is a great place to come and get started.

Read the Docs Update

It's been a while since I last talked about Read the Docs, and there has been a lot of activity. This is an update on the latest and greatest new features.

PSF Funding

The biggest news that has happened is that we have been given a grant from the Python Software Foundation to help host the site. Thanks PSF! They have blogged about it, and I am grateful that they have given us support. With the funds they have offered, we have been able to make Read the Docs a lot faster, and more robust. I will outline some of the changes below.

This also means we won't be going away any time soon!

New Theme

We have a fancy new theme for documentation on Read the Docs! If you have the 'default' theme for your project, it will show up on the build of your docs on the site. I think it is pretty great, thanks to the designers who spent their time making it awesome. A really great feature is that the new theme is mobile ready. Go ahead and view a project using it on your phone, or make your browser smaller and you will see the fanciness. Having a custom theme will give us a base to build lots of other neat features on top of.

Better architecture

We had some connectivity trouble in between our servers a while back, and this prompted me to make the site respond better to these conditions. Every time you view documentation on a subdomain of readthedocs.org, your request will never hit Django. So all of these requests will work without a database. We have also added a second application server with a load balance in front, which means that one of the app servers could go away and your documentation would still get served.

That leaves our load balancer as the main single point of failure at the moment. We're using Varnish for the load balancer, and we've implemented strong caching of data. Varnish will cache your docs for up to a week, and it will be actively purged when you rebuild your docs. This means that your docs will usually be served out of memory, and without dependence on any other server but that one. We have plans to elininate Varnish as a single POF, and then it would only be our hosting provider that would be a single point of failure (famous last words).

Intersphinx support

Intersphinx is an awesome feature of Sphinx that allows you to reference remote sphinx documentation easily. RTD now supports it for every project that we host.

Improved rtfd.org

I've always had big ideas for rtfd.org, since it can act as a short-url for things. projectname.rtfd.org has always redirects to the projects docs, but now we have something a lot better. Inspired by Jacob Kaplan-Moss and his work on django.me, we now support human-edited deep-linking within documentation hosted on RTD.

Taking another page from Jacob's book, we seeded the index of our projects with their Intersphinx data, so a lot of references will automatically work. This works best with API reference docs, but anything people have put links to in their documentation should have been picked up. A couple of examples:

If you go to a non-existent link on rtfd.org, you will be prompted to enter a suggested URL. This will help build the data, and make it more useful for everyone.

rtd command line utility

RTD has had an API for a while now, and with the addition of the support for rtfd.org, I thought it would be neat to make it easier to access docs from the command line. With a simple pip install rtd, you will get an rtd utility that will open docs on RTD. It supports 2 arguments, the first being a project name, and the second being a slug to append for the rtfd.org functionality. So like the example above:

-> rtd pip
Pip Installs Packages. 
Opening browser to http://pip.rtfd.org/
-> rtd celery Task
Distributed task queue
Opening browser to http://celery.rtfd.org/Task

It hits the RTD API to see if the project is on the site, and only opens your browser if it doesn't exist. I hope that in the future we'll make it easy to upload a project from the shell, and more.

More docs

Since we are a documentation site, we've always had documentation. I've been adding more as time has gone on, and most of the features I'll be talking about today are already documented. I also broke the documentation up into sections for users of the site, and developers on the codebase, so it should be easier to find for everyone to find what they are looking for.

Conclusion

I think that RTD can be doing a lot more to help out the community with regards to documentation. I'll write another post about that soon. But if you are interested in helping out with the effort, all of the code is open source and we love people to contribute. Feel free to jump in #readthedocs on Freenode as well, if you have any questions or thoughts.

Read the Docs Updates

Documentation writing will always be hard work. It's a much different mind-set than programming, and people that write good code might not necessarily write good docs. However, this is a known issue, and something that can't really be solved.

What you can do is make it easier to write documentation. Every step along the way that you can give yourself an excuse to not write documentation is another undocumented open source project.

Luke Plant has a great post up about how important documentation is, and I completely agree. I imagine a lot of the people using Django are using it because of the documentation. I think as members of the Django community, we need to build a culture of documentation within the greater Python world. Python does tend to have better documentation than a lot of languages, but it's still not nearly what it could be.

Read the Docs exists to make it easier to host your Sphinx documentation. Over the weekend, Bobby, Jonas, and I added a bunch of new features to the site. I think it's getting to the point where there isn't an easier or better way to host the documentation for your Django project, and we're only going to keep improving it!

A different Eric added a really nice Getting Started guide for RTD, that shows how easy it is to get your projects hosted with us.

Anyway, on to the new features that we added.

New Features

Versions

Versions of projects are easily one of the biggest requested features on the site. For a long time we just supported building the latest versions of your documentation. Now we support versions of your documentation that are tagged in your VCS (hg/git only).

A lot of larger projects need versioning because they support one or two versions, as well as developing in the trunk. Django was the main project we were thinking of, but some other projects have put this to good use. A couple of examples are:

Django's latest stable build
Fabric 0.9.3,
django-admin-tools's awesome integration that has all it's versions hosted with us.

PDF Support

Sphinx has interesting support for PDF generation through Latex. In my testing it was pretty unreliable, but I was able to rangle it into working well enough to expose in the UI. So now almost every project will have a "Download PDF" button. This code has version support as well, so we can offer PDFs of certain versions.

Another interesting part of this feature is that this building code has been abstracted out, so we can support epub, plain text, and all the other Sphinx output formats that people want.

Badges on the project pages

We killed the RTD header on hosted documentation pages in favor of a Badge in the lower right hand corner. The header clashed with a lot of the themes, and the badge is nice because it gives us a place to put functionality that is always visible, but is obviously not part of the hosted documentation. We want to build some more functionality into the badge, like switching between versions and linking back to the project's RTD page, once we build a good UI for it.

Sponsorship

Revsys has agreed to sponsor the hosting costs for RTD. Jacob Kaplan-Moss has always been a big proponent of documentation, and I'm glad that he and Frank Wiles are helping us keep Read the Docs around and get better. We tried to make the sponsorship subtle and not intrusive, so please let me know if it bothers you and we can try and figure something out.

Conclusion

I think that these features are really starting to make RTD a compelling platform for hosting your documentation. We are planning more awesome features that will make RTD even better. I'm really excited about the project and I hope that you either host your docs with us, or find docs that we host useful.

Handling Django Settings Files

I have seen a lot of talk over the past couple years about how to handle different settings files and databases, synced between production and development. I have happened onto a way of doing it that makes me happy, and figured I would share it with the world.

File structure

I use a file structure that looks like this:

project/
    settings/
        __init__.py (empty)
        base.py
        sqlite.py
        postgres.py

The base.py contains all of the configuration options that are shared among the databases. INSTALLED_APPS, etc. All of the DATABASE settings should be specified in the more-specific files. As well as things that differ by environment, like remote servers, cache settings, cookie domains, and other things.

This allows you to run the sqlite settings file, and have it be set to localhost, or whatever your development settings are. Then in production you just run against the postgres settings.

A good example of this being used in practice is on Read the Docs.

But wait, there's more!

manage.py for dev

./manage.py is great for development. It is the easiest way to get started, and it automatically sets up your paths and stuff. With my setup, I actually explicitly set manage.py's settings file to the sqlite file.

This means that whenever you are using manage.py, you are in a development context. So, what do you do about production?

django-admin.py is for production

In production, you set your DJANGO_SETTINGS_MODULE to the postgres settings file. So whenever you use django-admin.py, you will be running against the production database.

I really like this scheme, because it gives you a logical distinction between production and development in your code, and in your interface on the CLI. When you are developing, you are using manage.py and editing the sqlite settings file. The reverse for production.

Using Haystack to index non-database content

Over on ReadTheDocs, I wanted to build search around the documentation that we're hosting. I chose Haystack and Solr for this, because it's the best way to do search in Django these days. However, I've only ever used Haystack to index content that is in the database. I thought about trying to add all the rendered HTML from the documentation into the database, but that was a non-starter.

I ended up adding a ImportedFile model to the database, which would contain the metadata for the HTML file:

class ImportedFile(models.Model):
    project = models.ForeignKey(Project, related_name='imported_files')
    name = models.CharField(max_length=255)
    slug = models.SlugField()
    path = models.CharField(max_length=255)
    md5 = models.CharField(max_length=255)

This allows me to link the SearchIndex in haystack to a model. Then the interesting part is in the Haystack SearchIndex, where I override the prepare_text method, allowing me to read the data in from the filesystem instead of from the database.

class ImportedFileIndex(SearchIndex):
    text = CharField(document=True)
    author = CharField(model_attr='project__user')
    project = CharField(model_attr='project__name')
    title = CharField(model_attr='name')

    def prepare_text(self, obj):
        full_path = obj.project.full_html_path
        to_read = os.path.join(full_path, obj.path.lstrip('/'))
        try:
            content = codecs.open(to_read, encoding="utf-8", mode='r').read()
            return content
        except IOError:
            print "%s not found" % full_path

site.register(ImportedFile, ImportedFileIndex)

This means that I don't have to bloat my database with all my rendered HTML, but have the full HTML stored in Solr which works for querying.

Building a Django App Server with Chef: Part 4

Alternate title: There's no place like home!

This is Part 4, the final part, of my Chef tutorial. Today we're talking about the odds and ends left over to make the server nice to use. You can check out the first 3 parts of the series:

Today's code will be in the git repo under the tag blog-post-4.

What we'll need

So we have our app server up and running, and ready for traffic. Now we just need to add some other bits around the outside for it to be fully functioning and nice to use.

Monitoring with Munin
Background tasks with Celery
A firewall for security
A /etc/hosts file for talking with other nodes
A .bash_profile file so that when you shell in you'll have a nice environment

Let's get started.

Monitoring with Munin

For doing monitoring with munin, we're going to need to learn our final Chef concept, which is Templates. You should be pretty familiar with them already, except they use Erb, which is a template language that lets you embed Ruby.

We're only going to be configuring the Munin node here. This assumes that you have a munin server running on another machine that you want to give access to monitor your new app server. These configs depend on you putting an entry like this in your node.json, which points at the IP of the master server:

"munin_servers": ["10.177.243.34"],

Then here is how you would write the Recipe.

cookbooks/main/recipes/munin.rb

package "munin-node" do
    :upgrade
end

service "munin-node" do
  enabled true
  running true
  supports :status => true, :restart => true, :reload => true
  action [:enable, :start]
end

if node.attribute?("munin_servers")
  template "/etc/munin/munin-node.conf" do
    source "munin-node.conf"
    mode 0640
    owner "root"
    group "root"
    variables :munin_servers => node[:munin_servers] || []
    notifies :restart, resources(:service => "munin-node")
  end
end

The template Resource here is the interesting part. We're surrounding it with a conditional, that makes sure that we're defined a 'munin_servers' key in our node.json. Then we're saying that we're going to render the munin-node.conf file with the source template 'munin-node.conf'. This template will be given the extra varibale 'munin_servers', which is passed in using the variables attribute.

Template are placed inside the cookbook in a similar place to files.

cookbooks/main/templates/default/munin-node.conf

<% @munin_servers.each do |server| -%>
allow ^<%= server.to_s.gsub(/\./, '\.') %>$
<% end -%>
allow ^127\.0\.0\.1$

host *
port 4949

log_level 4
log_file /var/log/munin/munin-node.log
pid_file /var/run/munin/munin-node.pid
background 1
setsid 1
user root
group root

ignore_file ~$
ignore_file DEADJOE$
ignore_file \.bak$
ignore_file %$
ignore_file \.dpkg-(tmp|new|old|dist)$
ignore_file \.rpm(save|new)$
ignore_file \.pod$

The interesting part here is the iteration over the munin_servers list. It's just doing a simple ruby loop, and then outputting the IP address that it contains into the format that munin's configuration file expects.

Note: This data-driven template rendering is a really powerful idiom, and one of my favorite parts about Chef. This allows you to add a new server to your pool, and have all of your configuration files updated automatically across all your server. This is hugely powerful, and one of the primary wins of Configuration Management. This will be shown to better effect in the /etc/hosts file later.

Installing Celery

Installing celery is much akin to Gunicorn that was discussed yesterady. The dependencies were installed from our pip requirements file, and we just need to make it run in upstart. We'll be doing that with the following setup.

Additions to cookbooks/main/recipes/readthedocs.rb

cookbook_file "/etc/init/readthedocs-celery.conf" do
    source "celery.conf"
    owner "root"
    group "root"
    mode 0644
    notifies :restart, resources(:service => "readthedocs-celery")
end

service "readthedocs-celery" do
    provider Chef::Provider::Service::Upstart
    enabled true
    running true
    supports :restart => true, :reload => true, :status => true
    action [:enable, :start]
end

cookbooks/main/files/celery.conf

description "Celery for ReadTheDocs"

start on runlevel [2345]
stop on runlevel [!2345]
#Send KILL after 20 seconds
kill timeout 20

script
exec sudo -i -u docs django-admin.py celeryd -f /home/docs/sites/readthedocs.org/run/celery.log -c 2 -E -B
end script

respawn

There isn't anything new or interesting here. Just more of the same as before, to get another piece of infrastructure up and running.

A ghetto firewall install

I'm a big fan of not enabling services that aren't running as a fundamental security practice, but having a basic firewall to make sure that those are the only ports open isn't a bad idea either. I'm not a great expert, so this is probably the weakest part of my knowledge in this series, so take it with a grain of salt.

My favorite firewall utility is ufw. It makes managing your firewall really simple. Here is my super basic way to configure my firewall, it pretty much sucks :)

cookbooks/main/recipes/security.rb

package "ufw" do
    :upgrade

service "ufw" do
  enabled true
  running true
  supports :status => true, :restart => true, :reload => true
  action [:enable, :start]
end

bash "Enable UFW" do
user "root"
  code <<-EOH
  ufw allow 22 #SSH
  ufw allow 80 #Nginx
  ufw allow 4949 #Munin
  EOH
end

As you can see, we're just enabling SSH, Nginx, and Munin. If we need to install any more packages, we'll need to expicitly open a port, which is usually good to remind me that I'm doing it.

/etc/hosts

Whenever I'm in the cloud, I find keeping track of my other servers to be a pain. You generally want to use the internal backplane to communicate between your servers, so I use the /etc/hosts file to make that simple.

We're going to depend on an entry in your node.json that looks something like this:

  "all_servers": {"Golem": ["10.177.234.234", "173.203.234.234"],
                   "Chimera": ["10.177.234.234", "204.232.234.234"],
                   "Hydra": ["10.177.234.234", "173.203.234.234"] }

Which is a mapping of all your servers, with their internal and external IPs. This will be useful to have for lots of different recipes, and it would be nice to autogenerate this, but when you only have a few servers it isn't so bad.

The rest of out hosts configuration looks like this:

Addition to cookbooks/main/recipes/default.rb

if node.attribute?("all_servers")
  template "/etc/hosts" do
    source "hosts"
    mode 644
    variables :all_servers => node[:all_servers] || {}
  end
end

cookbooks/main/templates/default/hosts

127.0.0.1     localhost localhost.localdomain

<% @all_servers.each_pair do |name, ips| -%>
<%= ips[0] %> <%= ips[1] %> <%= name %>
<% end -%>

As you can see, when we add a server to the all_servers hash, it will propogate out to the /etc/hosts file of our app server. This makes me really happy, and showcases some of the more advanced use cases of Chef.

Customizing your shell

Now that we have the server all set up, it won't be much good if it isn't nice to use when we shell in. So here is how I go ahead and add in some nicities to bash for when you log in.

Addition to cookbooks/main/recipes/readthedocs.rb

cookbook_file "/home/docs/.bash_profile" do
    source "bash_profile"
    owner "docs"
    group "docs"
    mode 0755
end

cookbooks/main/files/default/bash_profile

. .bashrc

export PIP_DOWNLOAD_CACHE=/tmp/pip
export DJANGO_SETTINGS_MODULE=settings
export PYTHONPATH=$PYTHONPATH:~/sites/readthedocs.org/checkouts/readthedocs.org
export EDITOR=vim

. sites/readthedocs.org/bin/activate

cd ~/sites/readthedocs.org/

alias chk='cd /home/docs/sites/readthedocs.org/checkouts/readthedocs.org'
alias run='cd /home/docs/sites/readthedocs.org/run'

First off, we're sourcing the .bashrc file, so that we get all the nice things it provides, like a colored PS1. Then we're setting some environment variables so that django-admin.py and pip work nicely. Then we activate our virtualenv and switch into it's base directory, so we're always starting where we want to be on login. Then we just have a couple of aliases for easy navigation around.

I like how this makes the user experience of shelling into the server a lot nicer, and makes the manual workflow that you'll eventually have to fiddle with really nice.

Conclusion

So that's the end of this tutorial. I hope that it was instructive in learning Chef, as well as providing some insights into the deployment of a Django application. Tomorrow (or if I'm too tired, next week), I'll be providing some thoughts on how I think chef treated me, and how I feel about the build out.

Building a Django App Server with Chef: Part 3

Alternate title: Show the world what you've got.

This is Part 3 of my Chef tutorial. Today we're talking about deployment. You can check out the first 2 parts of the series:

Today's code will be in the git repo under the tag blog-post-3.

What we'll need

We'll be taking the Django application that we have on the server and actually deploying it. Let's make a list of what we'll need:

A web server to sit in front and proxy requests
A WSGI server
A way to keep both of these things running
A caching layer

We'll be using Nginx, Memcached, Upstart, and Gunicorn. This is my preferred deployment stack as of late, mainly because of the simple setup.

Let's get started

A web server

Getting Nginx up and running should be old hat by this point. We're going to need the package and service Resources, which will tell Chef to install and run it.

cookbooks/main/recipes/nginx.rb

package "nginx" do
    :upgrade
end

service "nginx" do
  enabled true
  running true
  supports :status => true, :restart => true, :reload => true
  action [:start, :enable]
end

cookbook_file "/etc/nginx/sites-enabled/readthedocs" do
  source "nginx/readthedocs"
  mode 0640
  owner "root"
  group "root"
  notifies :restart, resources(:service => "nginx")
end

cookbook_file "/etc/nginx/nginx.conf" do
  source "nginx/nginx.conf"
  mode 0640
  owner "root"
  group "root"
  notifies :restart, resources(:service => "nginx")
end

As you can see, we're providing our own nginx.conf and a readthedocs site configuration. I'm not going to paste these in, as they are pretty application specific, but you can look at them on Github if you're curious. I also wrote about it a while back.

The only new part here is the notifies command, which is pretty nifty. It basically means that whenever you change the nginx.conf file, it should restart Nginx, which is a really nice feature.

A WSGI Server

Yesterday, when we installed the deploy_requirements.txt with pip, it pulled in gunicorn. So we actually have Gunicorn already installed in our virtualenv, waiting for us to use it. The only difference is I actually committed a change to the ReadTheDocs source so that it will pull Gunicorn from the git master, which I'll explain below.

Upstart

Note: I use upstart because it ships with Ubuntu, so you don't need to install a separate package. However, it has pretty horrible documentation, with the Stanzas doc probably the best clue as to what it supports.

Here is where things get interesting. I spent a bunch of time trying to get gunicorn and upstart to play nicely yesterday night, but it wasn't working. I went on the #gunicorn IRC channel on Freenode today, and talked with benoitc. He was awesome and patched gunicorn to work with Upstart for me.

Here is the upstart script that we're using to keep gunicorn running:

cookbooks/main/files/default/gunicorn.conf

description "Gunicorn for ReadTheDocs"

start on runlevel [2345]
stop on runlevel [!2345]
#Send KILL after 5 seconds
kill timeout 5
respawn

env VENV="/home/docs/sites/readthedocs.org"

#Serve Gunicorn on the internal rackspace IP.
script
exec sudo -u docs $VENV/bin/gunicorn_django --preload -w 2 --log-level debug --log-file $VENV/run/gunicorn.log -p $VENV/run/gunicorn.pid -b 10.177.69.207:8888 $VENV/checkouts/readthedocs.org/settings/postgres.py
end script

As you can see, an Upstart script is a pretty clean way to do this. If you've ever tried to write an old SysV-style init script, this will look beautiful. You'll notice that we aren't passing the --daemon parameter to gunicorn, this is because upstart will background the process for us, and keep track of everything, so we don't need gunicorn's daemonizing behavior.

It should be pointed out how awesome it is that we can run a production ready WSGI server with a single line of bash. If you've ever set up a mod_wsgi install, needing to fuddle with your apache.conf and a WSGI file and everything makes it a chore. This is quite simply the easiest way to deploy a WSGI application.

Then we need some additions to cookbooks/main/recipes/readthedocs.rb:

cookbook_file "/etc/init/readthedocs-gunicorn.conf" do
    source "gunicorn.conf"
    owner "root"
    group "root"
    mode 0644
end

service "readthedocs-gunicorn" do
    provider Chef::Provider::Service::Upstart
    enabled true
    running true
    supports :restart => true, :reload => true, :status => true
    action [:enable, :start]
end

Here you can see we're doing a similar thing to the other service declarations. We however need to tell Chef to use Upstart for this service, instead of defaulting to init.d. Other than that, everything here should look similar to the other files and services we've set up.

Memcached

As you would expect, installing memcached is just like nginx:

cookbooks/main/recipes/memcached.rb

package "memcached" do
    :upgrade
end

service "memcached" do
  enabled true
  running true
  supports :status => true, :restart => true
  action [:enable, :start]
end

cookbook_file "/etc/memcached.conf" do
  source "memcached.conf"
  mode 0640
  owner "root"
  group "root"
  notifies :restart, resources(:service => "memcached")
end

The memcached.conf is so short, I might as well include it here:

-d
logfile /var/log/memcached.log
-m 64
-p 11211 
-u nobody
-l 127.0.0.1

Memcache's config file is pretty neat, because it's basically just a list of arguments to pass to the daemon when it's started. A little bit like a pip requirements file is just commands to pass to pip install when it's run.

Wrapping up

Now that you have these awesome new recipes, and additions to old ones, we need to make sure they're actually being run. Your run_list in your node.json file should now look something like this:

"run_list": [ "main::default", "main::python", "main::readthedocs", "main::memcached", "main::nginx"],

At this point, it's pretty neat. I can run a fab install_chef update, wait about 5 minutes, and go from a freshly paved server to a fully functioning app server.

Tomorrow we'll be adding some monitoring and auxiliary niceties. This includes setting up Munin, Celery, generating the /etc/hosts file, and throwing in a little .bashrc magic to make the user experience nicer.

There were a couple of questions yesterday about databases and other things. My current problem is running an application server, which is what I've accomplished. However, with my new-found love affair for chef, I will definitely be making my Database/Utility box into a chef configuration really soon. I might not write it up in so much detail, but hopefully you've learned enough from this series that I can just publish the code.

Building a Django App Server with Chef: Part 2

Alternate title: Actually doing something useful.

Yesterday we covered the basics to getting started with Chef. You should have a remote server configured with chef, and have curl installed! Now lets go ahead and get some useful bits for your Django application.

What we'll need

So this is going to be based around the way that I set up my servers, so if this is different than you, I'm sorry. However, I think it is a pretty solid way of managing them. A lot of the ideas here are stolen from Travis when he set up the server for Pypants.

So lets assemble a list of things we're going to want in order to get a super basic Django configuration running:

A user to run our code as and who's home directory we'll store the data.
A basic global python ecosystem, including setuptools and pip
A virtualenv to store all the project-specific packages and code in
A copy of the project that we'll be running

Let's get started.

The finished code for today is located on github, with the tag blog-post-2. It is a copy of the completed steps, so feel free to peek through that and come back here for clarification (or to ask questions).

Setting up our user

For RTD, I run everything under the user docs. So we'll go ahead and set up that user so that we can get our site set up. We're going to go ahead and replace our "default" recipe, because right now it isn't doing anything much useful. The relevant part is below:

cookbooks/main/recipes/default.rb

node[:base_packages].each do |pkg|
    package pkg do
        :upgrade
    end
end

node[:users].each_pair do |username, info|
    group username do
       gid info[:id] 
    end

    user username do 
        comment info[:full_name]
        uid info[:id]
        gid info[:id]
        shell info[:disabled] ? "/sbin/nologin" : "/bin/bash"
        supports :manage_home => true
        home "/home/#{username}"
    end

    directory "/home/#{username}/.ssh" do
        owner username
        group username
        mode 0700
    end

    file "/home/#{username}/.ssh/authorized_keys" do
        owner username
        group username
        mode 0600
        content info[:key]
    end
end

node[:groups].each_pair do |name, info|
    group name do
        gid info[:gid]
        members info[:members]
    end
end

There's a lot of stuff going on here, so lets go over it. First you might notice that there's this node variable, the node data structure is the JSON that you have in your node.json file. It is looping over the keys and values with ruby's each_pair and pair functions.

The base_packages bit is a cool example of the power of the chef configuration. We have a list of packages that we want to install in our Attributes, and we're looping over them and setting using the package Resource.

I realize I skipped over the run_list part yesterday, but it basically is just a list of recipes to run. Each of the resources in the default.rb file should be pretty self explanatory. The Chef Resource Documentation is really comprehensive, and will probably be the most referenced document that you use. The main resource's that we used were group, user, file, directory, let's take a look at the User declaration in particular.

Everything there should be pretty obvious, as it's the information that goes into /etc/passwd for the user. However, the supports keyword isn't obvious at first. This is part of the Common Attributes that can be set on all Resources. It's a way of passing along configuration options to the Resource. manage_home actually just makes it so that the users home directory is created when the user is created.

So we're going to have to go ahead and put some data in there for it to work with. Our node.json will now look like this:

node.json

{
  "run_list": [ "main::default", "main::python", "main::readthedocs" ],
  "base_packages": ["git-core", "bash-completion"],

  "users": {
    "docs": {
      "id": 1001,
      "full_name": "Docs User",
      "key": "ssh-rsa key-goes-here eric@Bahamut"
    }
  },

  "groups": {
    "docs": {
      "gid": 201,
      "members": ["docs"]
    }
  }
}

Adding a Basic Python Environment

Now lets go ahead and add a python recipe to build out some basic python stuff that we'll be needing.

cookbooks/main/recipes/python.rb

node[:ubuntu_python_packages].each do |pkg|
    package pkg do
        :upgrade
    end
end

# System-wide packages installed by pip.
# Careful here: most Python stuff should be in a virtualenv.

node[:pip_python_packages].each_pair do |pkg, version|
    execute "install-#{pkg}" do
        command "pip install #{pkg}==#{version}"
        not_if "[ `pip freeze | grep #{pkg} | cut -d'=' -f3` = '#{version}' ]"
    end
end

Additions to node.json

  "ubuntu_python_packages": ["python-setuptools", "python-pip", "python-dev", "libpq-dev"],
  "pip_python_packages": {"virtualenv": "1.5.1", "mercurial": "1.7"},

Here we're adding some global packages that we need. We're going to install setuptools and pip so that we can install further python packages. python-dev and libpq-dev are so that we have the headers for libraries that need to compile against postgres and python. We'll also be installing virtualenv and mercurial globally so that we can create our virtualenv and install packages from mercurial.

Creating a virtualenv

We're going to introduce the first new Chef concept here, which is called a Definition.

Definition (cookbooks/*/definitions/*.rb)

A definition is a custom Resource that you build to abstract a set of operations. Pretty simple

This is a definition that Jacob published and then I updated to make the permissions correct. It allows you to set up a virtualenv:

cookbooks/main/definitions/virtualenv.rb

define :virtualenv, :action => :create, :owner => "root", :group => "root", :mode => 0755, :packages => {} do
    path = params[:path] ? params[:path] : params[:name]
    if params[:action] == :create
        # Manage the directory.
        directory path do
            owner params[:owner]
            group params[:group]
            mode params[:mode]
        end
        execute "create-virtualenv-#{path}" do
            user params[:owner]
            group params[:group]
            command "virtualenv #{path}"
            not_if "test -f #{path}/bin/python"
        end
        params[:packages].each_pair do |package, version|
            pip = "#{path}/bin/pip"
            execute "install-#{package}-#{path}" do
                user params[:owner]
                group params[:group]
                command "#{pip} install #{package}==#{version}"
                not_if "[ `#{pip} freeze | grep #{package} | cut -d'=' -f3` = '#{version}' ]"
            end
        end
    elsif params[:action] == :delete
        directory path do
            action :delete
            recursive true
        end
    end
end

As you can see, it takes a bunch of arguments, then just wraps up a bunch of Resource definitions in a nice little package. There is a little bit of magic with the pip freezing things, but it's basically just how we're checking to make sure that a package isn't instead before we install it. We are using only using the directory and execute Resources here.

Now we're going to use this virtualenv Definition, and create the home virtualenv for our site. I like to keep my virtualenv's in ~/sites/, so this will go into /home/docs/sites/readthedocs.org/. Since this is becoming specific to the site we're building, it's going to go into a readthedocs recipe:

cookbooks/main/recipes/readthedocs.rb

directory "/home/docs/sites/" do
    owner "docs"
    group "docs"
    mode 0775
end

virtualenv "/home/docs/sites/readthedocs.org" do
    owner "docs"
    group "docs"
    mode 0775
end

This will set up a basic virtualenv in our directory.

Getting our site set up

To get our site set up, we need to pull in the source code, and make sure our virtualenv has all the requirements. This code is a little bit hacky, and could probably be abstracted out a bit, but it will work for now. We're going to go ahead and add some things to our readthedocs Recipe.

Additions to cookbooks/main/recipes/readthedocs.rb

directory "/home/docs/sites/readthedocs.org/run" do
    owner "docs"
    group "docs"
    mode 0775
end

git "/home/docs/sites/readthedocs.org/checkouts/readthedocs.org" do
  repository "git://github.com/rtfd/readthedocs.org.git"
  reference "HEAD"
  user "docs"
  group "docs"
  action :sync
end

script "Install Requirements" do
  interpreter "bash"
  user "docs"
  group "docs"
  code <<-EOH
  /home/docs/sites/readthedocs.org/bin/pip install -r /home/docs/sites/readthedocs.org/checkouts/readthedocs.o
rg/deploy_requirements.txt
  EOH
end

I like to have my runtime files in the venv/run directory, so we'll go ahead and create that directory. Then comes the fun part.

We are checking the Readthedocs source out of github with the git Resource. Chef only supports git and svn as far as I can tell, so luckily I'm using git.

Then we're going to install from the pip requirements file. This is using the script Resource, which allows you to inline a bash, ruby, python, or more script inside your Recipe. This is using a hard coded bash script to install the requirements, which sucks, but will work for now.

Note: Chef appears to buffer output and not show itself as doing anything when running the script Resource here, so it will look like your build will hang while it installs your pip requirements file for the first time.

Done for now

Alright, this post has gotten long enough, so we're done for today. But we're in a pretty awesome spot, I think. We now have our app server set up with a runnable version of our code. You can go ssh in and play around, you should be able to run simple manage.py commands inside the virtualenv and whatnot (after a syncdb).

Tomorrow we'll talk about deploying our code with Nginx and Gunicorn. I've been having trouble with Upstart, so we might switch our deployment to Supervisord, but we'll see how it goes.

Don't forget to check out the finished code on Github to see the actual running examples.

Building a Django App Server with Chef: Part 1

Alternate title: Fucking Chef, How does it work?

When I started looking at Chef, I found it incredibly complex and lacking in fundamental documentation. This is going to be my experience understanding Chef while setting up a single server. This strategy can be used across multiple servers, with a little tweaking.

I'd like to thank Jacob for the ideas and some of the inspiration behind the code and ideas. The code for this blog post can be found on github. It will be expanded as I write updates.

Today's code will be using tag blog-post-1 in the repo.

Goal

I'm hoping to write a blog series that goes from explaining what Chef is, to having a working Django server, and to release all this code so that you can tweak and use it with your own servers. I'll be doing this to set up a new and configuration managed server for ReadTheDocs.

There are a couple of other good chef intros available. However, they only get you to the super basic first step of setting up the server. Hopefully this series will be more in depth and useful to actually getting a real server running under Chef.

Basic terminology

Chef has some basic terminology that you need to understand before we get into things. I'm going to purposefully leave out a ton of stuff, because it isn't really important for me now.

Cookbook (cookbooks/*)

A cookbook, not surprisingly, contains recipes. With my configuration, we're only going to use one cookbook, that has multiple recipes. This is probably wrong and horrible, but it's simple, which is the goal for now.

Recipe (cookbooks/*/recipes/*.rb)

A recipe is the basic building block of Chef. It is what does the meat of the work that you want done.

Resources

A resources is an abstraction that defines a specific piece of your configuration. It can be a file, a user, or just about anything you want to talk about on your system.

Attributes (node.json)

Attributes are just a blob of JSON that Chef uses to pass around data. It will be (slightly) different for each server that you want to set up. I really like this approach, because I think a data-driven approach is the correct way to solve this problem.

Bootstrapping chef

We'll be running what is called chef solo. This means that chef will be run independently of a server, and just execute on our one host. This seems to be the easiest way to get things running.

When you first run chef-solo, it will look for a solo.rb file. The documentation about configuring chef-solo does a decent job of explaining this. By default it looks in /etc/chef/solo.rb, so lets go ahead and use that. It has 2 required fields, a cookbook_path and json_attribs. cookbook_path tells chef where to find the "cookbooks" it uses to run your code, and json_attribs tells it where to load in your data dictionary.

Note: The docs say that file_cache_path is required, but it just defaults to /var/chef/cache.

For simplicity, we're going to keep our cookbooks and json data file in /etc/chef. On my local machine I keep everything in a ~/projects/chef directory, and then sync that to the remote box.

In production you'll probably want to set up your remote server to pull from a repository somewhere, so that you can have a stable deployment base, but again, syncing from the local filesystem is simple and works.

Bootstrapping your new server

I'll be using a fabric script to run the commands on a remote server, which also allows me to run them again later on a new machine in an automated fashion. So this is the first simple script that we'll use to bootstrap our new server, fabfile.py:

from fabric.api import env, local, sudo
env.user = 'root'
env.hosts = ['204.232.205.196']

env.chef_executable = '/var/lib/gems/1.8/bin/chef-solo'

def install_chef():
    sudo('apt-get update', pty=True)
    sudo('apt-get install -y git-core rubygems ruby ruby-dev', pty=True)
    sudo('gem install chef', pty=True)

def sync_config():
    local('rsync -av . %s@%s:/etc/chef' % (env.user, env.hosts[0]))

def update():
    sync_config()
    sudo('cd /etc/chef && %s' % env.chef_executable, pty=True)

A couple notes: the chef executable is version-dependent, but that's because I don't know enough ruby to query it dynamically. You will also need to change the value of the env.hosts to a server that you actually control.

This will install the ruby dependencies, and get chef up and running for you. You'll need to install fabric (pip install fabric, presumably in a virtualenv), and then run fab install_chef to get it up and running. Then you can go ahead and run fab sync_config to get your chef configuration onto the remote server.

Now we need to go ahead and figure out how to make chef actually do something. You'll see in the cookbooks/main/recipes/default.rb file we have a simple package declaration. This simply means to make sure that the package is installed on the remote system. This is as simple as it gets, so lets go ahead and run it. With the fab update command, it will sync your local directory, and run chef on the remote server.

-> fab update
[localhost] run: rsync -av . root@204.232.205.196:/etc/chef
[204.232.205.196] sudo: cd /etc/chef && /var/lib/gems/1.8/gems/chef-0.9.12/bin/chef-solo
[204.232.205.196] err: stdin: is not a tty
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:01 +0000] INFO: Setting the run_list to ["main::default"] from JSON
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:01 +0000] INFO: Starting Chef Run (Version 0.9.12)
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:01 +0000] INFO: Installing package[curl] version 7.21.0-1ubuntu1
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:04 +0000] INFO: Chef Run complete in 2.574963 seconds
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:04 +0000] INFO: cleaning the checksum cache
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:04 +0000] INFO: Running report handlers
[204.232.205.196] out: [Tue, 09 Nov 2010 01:42:04 +0000] INFO: Report handlers complete

You now have Chef running on your server. That was pretty easy, eh? For tomorrow's lesson, we'll be making it actually do something, like installed nginx and gunicorn, and keeping track of config files.

Celery Tips

Following on yesterday's post about Virtualenv Tips, I will be talking about celery tips. Yesterday I talked about how to run celery with upstart easily, and today I'll be expanding on that below as well as talking about how to set it up using supervisord.

Note: Also interesting, I wrote a Big list of django tips back in 2008, that still has a lot of good information.

Running celery in development

When you run celery in production, you should be using a queue on the backend. However, when you're running celery in development, it's nice to execute the code paths, but not actually need a queue. This is where the CELERY_ALWAYS_EAGER setting comes in handy. It makes celery run the code in process, but will make sure your code paths work correctly.

I talk about this and more in my djangocon talk.

Killing long running tasks

On ReadTheDocs I would run into problems with celery tasks never returning. Luckily, celery has a way to handle this. The CELERYD_TASK_TIME_LIMIT setting lets you set the number of seconds that a task can run until it is killed. This is nice to make sure that a run-away task won't take down all your backend processing.

Use the JSON serializer for interoperability

I was talking on IRC to Eric Florenzano and he mentioned that you should use the json serializer if you want to be able to add celery tasks from other languages.

This allows you to use another language to put a message that looks like a celery task in the queue, and it should just work.

Explictly set the number of clients

When you run celery, it defaults to having the number of workers equal to the number of cores the machine has. If you are running multiple queue workers on the same machine, it is a good idea to use less. You can set this with the CELERYD_CONCURRENCY setting, or passing -c on the command line.

Running against multiple databases with supervisord

At work we run a bunch of different sites on multiple databases. When we were figuring out how to deploy celery, we needed a good way to make sure that celeryd was always running, and we needed multiple celery daemons for each of our databases. We have written our tasks to run against multiple sites on the same database in order to reduce the number of daemons we had to use.

Celery ships with a couple of daemon configurations out of the box, support for init.d style init scripts, and support for supervisord. We first looked at the init.d approach, but there didn't seem to be a good way to have it run multiple settings files without creating multiple scripts, which seemed hacky. So we went with superisord for the task. Below is our configuration, if you are curious.

/etc/supervisord.conf

By default, the conf file is in the top-level /etc/ directory. We kept it this way, but I kind of wish it was in it's own directory. This is basically the exact script that celery ships with

unix_http_server]
file=/tmp/supervisor.sock ; path to your socket file

[supervisord]
logfile=/var/log/supervisord/supervisord.log ; supervisord log file
logfile_maxbytes=50MB ; maximum size of logfile before rotation
logfile_backups=10 ; number of backed up logfiles
loglevel=info ; info, debug, warn, trace
pidfile=/var/run/supervisord.pid ; pidfile location
nodaemon=false ; run supervisord as a daemon
minfds=1024 ; number of startup file descriptors
minprocs=200 ; number of process descriptors
user=root ; default user
childlogdir=/var/log/supervisord/ ; where child log files will live

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface ;

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use unix:// schem for a unix sockets.

[include]
files = supervisord/celeryd.conf

Then we created a supervisord directory which we included in the above file (in the last line) that contains the celery specific configuration. On this machine the only thing that supervisord is watching is celery, so that has kept our configuration simple.

/etc/supervisord/celeryd.conf

Inside of our celeryd specific configuration we went with mostly stock options except how we are setting up the DJANGO_SETTINGS_MODULE. We need to change the environment in which we are deploying, so that the celery daemon runs against the correct database.

    [program:celery-cms]
    environment = PYTHONPATH='/home/code',DJANGO_SETTINGS_MODULE='ljworld.standard'
    command=/home/code/django/bin/django-admin.py celeryd --loglevel DEBUG -c2
    user=nobody
    numprocs=1
    stdout_logfile=/var/log/celery/cms_supervisord.log
    stderr_logfile=/var/log/celery/cms_supervisord.err
    autostart=true
    autorestart=true
    startsecs=10

    [program:celery-weeklies]
    environment = PYTHONPATH='/home/code',DJANGO_SETTINGS_MODULE='desotoexplorer.settings'
    command=/home/code/django/bin/django-admin.py celeryd --loglevel DEBUG -c2
    user=nobody
    numprocs=1
    stdout_logfile=/var/log/celery/weeklies_supervisord.log
    stderr_logfile=/var/log/celery/weeklies_supervisord.err
    autostart=true
    autorestart=true
    startsecs=10

The really nice part about using supervisord is that our fabric script for deploying changes to celery is just deploying the code and then running supervisorctl restart celery-cms.

I hope today's post was useful, and I'm again curious for any other awesome celery tips!