Mark Liu: Technical Blog Posts

How to find a good freelance developer

Mark Liu — Tue, 08 Jan 2013 18:33:06 -0600

A bunch of people have asked me recently how to find good contract developers. I’ve done this quite a few times now and feel like I now have a pretty good system for selecting quality freelancers at good fixed prices.

I can tell that this system works because on my last two contract projects I received an average of 100 bids per posting (spread across 3 different sites) and both hires ended up blowing me away with the quality, speed, and price of their work. I have viewed many friends’ postings that have received 10 or fewer bids, many of which were low quality. I’ll take you through what I do to give myself better developer options.

Step 1

Write a great specification of the work you want done. There are a lot of reasons why this is important.

First, the best freelance developers are more likely to place bids if they know the scope of the work. When they don’t know the scope of the work it becomes dangerous for them to place a fixed price bid because they might end up doing far more work than they had hoped. These developers care a lot about their aggregate ratings on these freelance sites and will make sure they do everything in their power to make their customer happy and get a top rating. If you lay out a detailed specification, they know exactly what it will take and are no longer scared that you have a list of features up your sleeve that you want them to implement as well.

Basically a great specification will tell the potential hire that you know what you’re doing and that you will be pleasant to work with.

Second, you will need to define all of these features at some point anyway so you might as well do it right up front in order to gain the benefit of finding the best possible developer. It’s a good exercise to fully define your product as you will find tricky parts you hadn’t thought of while simply thinking about your product at a high level. The earlier you think about and address these tricky bits, the less costly it will be to implement well.

Here’s an example of a spec I wrote recently that received 110 bids, at least 8 of which were very high quality: https://docs.google.com/document/d/15c6n6HAUoa5SDP7TlGU_sljpdIA_8doe3h2Kbi9VcoM/edit

Step 2

Post your spec to elance.com without setting a price. They allow you to not specify your budget or expected price. By posting it without a price you can wait for the bids to roll in and you’ll start to get a good idea of how much this job should cost.

Step 3

Take the expected cost determined in step two and post the same spec to odesk and freelancer.com. There are plenty of other freelance sites out there so you are free to post to those as well. I have had great luck with those three sites (elance, odesk, and freelancer) and can confidently say there are some very high quality developers on each of these sites who will come out of the woodwork if your spec is good.

Step 4

Wait. It’s tempting to hire the first competent person who submits a good bid, but you will get plenty more if you wait. Be patient and wait for the bids to start slowing down. 2-3 days should be enough.

Step 5

Narrow down the bids to find the right developer. This can seem overwhelming at first but if you do it systematically you can do it in about 2-4 hours. Here’s what I do:

Narrow down the field based on their bid price, their quantitative ratings, and the amount of work they’ve done in the past. After keeping only people with 4.9+ ratings, a significant amount of work, and a very good bid price the field usually narrows to about 10.
Briefly glance at each of the top 10 applicants’ portfolios and take a look at their communication skills in their messages with you or on their profile page. Eliminate all but the very best. Hopefully you’ll be down to your last 2-4 people now.
Choosing from these last few is the toughest step and for this I tend to choose the person I think would make the best long term hire. English speaking ability and the general vibe I get from the person play a big role here. Of course this is just a contract project, but while working with this person you will build up a relationship and if they produce high quality work the chances are you will want to hire them for work in the future as well.

Final Note

If your project is big you may want to break it up into small pieces. For instance you can write the spec for just a small part of your application so if you end up hiring someone who produces low quality work, you won't lose much because you haven't invested much. I view the first job I award someone as an interview to some extent. If you like that person you can always hire him/her for the rest of your project.

Good luck!

Python syntax highlighting in Django templates using Pygments

Mark Liu — Sun, 05 Feb 2012 13:14:57 -0600

Highlighting code to be placed on your site is a pretty common need that has been addressed nicely by a number of open source projects. Pygments is great, but on it's own would take you a while to get set up. This guide is meant to get you set up with using pygments in your django templates in under 5 minutes.

Choose your syntax highlighting style

First, decide which pygments style you want to use by trying them out on the pygments site.

Install django-pygments

There's a nice project on github that gives you a couple template tags for using pygments.

pip install -e git+git://github.com/odeoncg/django-pygments.git

Create a pygments css file

Using whichever style you liked from the pygments site, you can use the pygmentize command in your terminal to generate a CSS file for marking up your code.

pygmentize -S vs -f html > vs.css

Mark up your django templates

You're now ready to make the changes to your templates. Just import your css file, load the template tags, wrap your pre-formatted code with the pygment template tag, and then label the code as whatever language you use.

<link href="/media/css/pygments/vs.css" media="screen" rel="stylesheet" type="text/css" />

{% load pygmentify %}
{% pygment %}
<pre lang="python">
import simplejson
import urllib2

</pre>
{% endpygment %}

Reducing CrashPlan's Disk Space Usage on OSX

Mark Liu — Fri, 04 Nov 2011 13:48:35 -0500

I have CrashPlan backing up almost every file on my Hackintosh, and when CrashPlan runs, it caches a ton of data so it can run faster. However, they don't let you configure where that data is stored. In my case, I'm backing up around 1TB of data, and thus the cache is a whopping 8GB. 8GB might not seem like that much these days, but when it's sitting on your small SSD, it's quite annoying. Fixing this problem was pretty easy, and since plenty of people were complaining about this on their forums in different threads, I thought I'd post my simple solution here.

First, you need to have a second drive connected to your system that you'd rather have hold the cache. If you have this, then this solution should work for you:

1. Close CrashPlan, as you normally would.

2. Stop CrashPlan's background service by typing:

sudo launchctl unload /Library/LaunchDaemons/com.crashplan.engine.plist

3. Remove the Cache:

rm -r /Library/Caches/CrashPlan

4. Create a simlink:

ln -s /Volumes/OtherDrive/path/to/new/cache/ /Library/Caches/CrashPlan

5. Start CrashPlan's background service:

sudo launchctl load /Library/LaunchDaemons/com.crashplan.engine.plist

6. Restart CrashPlan

This should free up a whole bunch of space on your startup disk. Enjoy!

Django-Celery on Webfaction using RabbitMQ

Mark Liu — Thu, 29 Sep 2011 10:15:23 -0500

This tutorial is meant to get you up and running from scratch with django-celery on Webfaction. Each of the steps is a bit of a hassle since you typically need to find different install steps for each individual part, so I just lumped up the whole experience in this guide.

Install Erlang

Erlang is needed for installing RabbitMQ which is the preferred message broker for Celery. Webfaction doesn't come with this installed, so you'll need to do it manually:

Go to the webfaction control panel and create a new app -> Custom App, Listening on Port
Download the latest version of Erlang. You can just use the command: wget http://www.erlang.org/download/otp_src_R14B03.tar.gz
Unzip it: gunzip -c otp_src_R14B03.tar.gz | tar xf -
cd into the directory
Configure the build: ./configure --prefix=/home/your_webfaction_username/
Make it: make
Install it: make install
Run it on the port given to you when you created the new Erlang app: epmd -port 12345 -daemon

Install RabbitMQ

Go to the webfaction control panel and create a new app -> Custom App, Listening on Port
Download the latest version of the RabbitMQ server generic package. You can just use the command: wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.6.1/rabbitmq-server-generic-unix-2.6.1.tar.gz
Unzip it: gunzip -c rabbitmq-server-generic-unix-2.6.1.tar.gz | tar xf -
Simlink rabbitmq to the erlang lib directory: cd ~/lib/erlang/lib/; ln -s ../src/rabbitmq_server-2.6.1 rabbitmq_server-2.6.1

Next, you need to change the file ~/lib/rabbitmq/rabbitmq-server. I found some information about this on the webfaction community forums. Open your text editor and change three lines to:

CONFIG_FILE=~/src/rabbitmq_server-2.6.1/sbin/

LOG_BASE=~/logs/user/rabbitmq

MNESIA_BASE=~/src/rabbitmq_server-2.6.1/sbin/

Added these lines to the rabbitmq-env file and use the ports you reserved for epmd and rabbitmq in your earlier steps:

export ERL_EPMD_PORT=12708

export RABBITMQ_NODE_PORT=35478

export ERL_INETRC=$HOME/.erl_inetrc

Added the file $HOME/hosts which looks like:

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

127.0.0.1 web160 web160.webfaction.com

Added the file $HOME/.erl_inetrc which looks like:

{hosts_file, "/home/<your_user_name>/hosts"}.

{lookup, [file,native]}.

Run Rabbitmq and check that it is working:

./rabbitmq-server -detached

./rabbitmqctl status

Finally, add a new user and vhost, and configure it so only your app will have access to it.

./rabbitmqctl add_user <username> <password>

./rabbitmqctl set_user_tags <username> administrator

./rabbitmqctl add_vhost <vhostpath>

./rabbitmqctl set_permissions -p <vhostpath> <username> ".*" ".*" ".*"

./rabbitmqctl clear_permissions -p <vhostpath> guest

Install Celery and Django-Celery

pip install django-celery

In your settings file, you will then need to add the lines:

BROKER_HOST = "localhost"

BROKER_PORT = 36784

BROKER_USER = "username"

BROKER_PASSWORD = "password"

BROKER_VHOST = "vhostpath"

CELERYD_CONCURRENCY = 1

CELERYD_NODES="w1"

CELERY_RESULT_BACKEND="amqp"

The reason we set the concurrency so low is because Celery takes up a good amount of memory, and you are likely limited with your memory consumption on webfaction. The minimum amount of memory Celery can take will be however much it needs to run the main process (consuming messages, sending tasks to workers, etc), and a worker tasks that actually does stuff. Each of these will take up about 20-30MB of memory depending on the size of your Django app.

Add 'djcelery' to your installed apps.

Follow any other steps listed in their installation guide that are relevant to your app. If you are using mod_wsgi, add the following to your .wsgi module:

import os

os.environ["CELERY_LOADER"] = "django"

Install a tool to create a Daemon

Celery does not daemonize itself, and thus you need to do this yourself. Creating a daemon is not exactly the same as simply running it in the background, so you should install a tool that can help you do this. Celery recommends a couple options. One of the easiest ways is to use a simple tool called django-supervisor. To install this, just type:

pip install django-supervisor

Add the file supervisord.conf in the same directory as manage.py, and add the content:

[program:celeryd]

command={{ PYTHON }} {{ PROJECT_DIR }}/manage.py celeryd -l info

[program:autoreload]

exclude=true

[program:runserver]

exclude=true

[program:celerybeat]

exclude=true

Every time you restart your webserver, you can restart celery by issuing the following commands:

python manage.py supervisor --daemonize

python manage.py supervisor stop all

python manage.py supervisor start all

However, there is a downside with using django-supervisor in that it will run in the background and take up another 20-30MB of memory. A more memory efficient way would be to install a tool called daemonize. This page has very easy installation instructions. Once you install it, just add an alias to it in your .bashrc or .profile and then run:

daemonize /<full_path_to_django_directory>/manage.py celeryd

Everything should then be up and running. Good luck!

Migrating a Django Postgres DB from Concrete Inheritance to Abstract Inheritance

Mark Liu — Tue, 23 Aug 2011 12:59:36 -0500

Django comes with several ways of implementing model inheritance, and specifying which one you would like to use takes only a line or two of code. Setting up your database for the very first time is extremely easy, but migrating between types after you have existing data in infinitely harder. I have a live site whose needs have changed since I built the database and the concrete inheritance I set up is no longer needed. In my case, the site should be using Abstract Inheritance.

There are a number of gotchas in this process, and I will outline exactly how I navigated my way through this process. First, I should note that I use django-south in my project and the migrations rely heavily on using this. I asked for advice on Stackoverflow and I'll use a similar example here. This article will walk you through the process of migrating from the before to after schema while keeping all your data intact.

Before:

app1/models.py:

class Model1(base_app.models.BaseModel):

field1 = models.CharField(max_length=1000)

field2 = models.CharField(max_length=1000)

app2/models.py:

class Model2(base_app.models.BaseModel):

field1 = models.CharField(max_length=1000)

field2 = models.CharField(max_length=1000)

base_app/models.py:

class BaseModel(models.Model):
user1 = models.ForeignKey(User, related_name="user1")

user2 = models.ForeignKey(User, related_name="user2")

another_field = models.CharField(max_length=1000)

objects = CustomManager()

After:

app1/models.py:

class Model1(base_app.models.BaseModel):

field1 = models.CharField(max_length=1000)

field2 = models.CharField(max_length=1000)

objects = CustomManager()

app2/models.py:

class Model2(base_app.models.BaseModel):

field1 = models.CharField(max_length=1000)

field2 = models.CharField(max_length=1000)

objects = CustomManager()

base_app/models.py:

class BaseModel(models.Model):

user1 = models.ForeignKey(User, related_name="%(class)s_user1")

user2 = models.ForeignKey(User, related_name="%(class)s_user2")

another_field = models.CharField(max_length=1000)

class Meta:

abstract = True

These are the models I will use to walk you through this migration process. Essentially what we are doing here is removing the one-to-one relationships between Model1/BaseModel and Model2/BaseModel and instead placing each of the fields of BaseModel into Model1 and Model2 so they actually reside in the tables for Model1 and Model2. The tricky part is migrating all your data while we work through this.

Make a Copy of BaseModel

When you add the "abstract = True" property to the BaseModel (don't do this yet), south will delete the entire table BaseModel. If you don't make a copy of the data in BaseModel, after you add the "abstract = True" property you will have no way of copying the fields in the old BaseModel to the respective new models. Making a copy isn't bad:

Add BaseModelCopy within base_app/models.py that has an identical schema to BaseModel.
Run a schema migration (python manage.py schemamigration base_app --auto; python manage.py migrate base_app).
Run a data migration to copy the existing objects in BaseModel to BaseModelCopy (python manage.py datamigration base_app copy_contents; (then edit the contents of the data migration); python manage.py migrate base_app).

Add a Field on the Child Classes to Store the ID

The way django deals with concrete inheritance is a little interesting when you look closely at it. The Child classes Model1 and Model2 do not have 'id' columns in the database, and their primary_key is actually a column named "basemodel_ptr_id". Thus, the unique id is stored in the corresponding BaseModel object. When you eventually add the "abstract = True" property, south will naturally delete this "basemodel_ptr_id" column and you will lose all references to the original BaseModel object. This is clearly very bad, and we would have no way of recovering those relationships if we did that. Thus, we need to add a field to Model1 and Model2 that stores the id of the corresponding BaseModel object so we can reference it in the data migrations after we move to abstract base classes. Doing this is also straightforward:

Add a field "tmp_id = models.IntegerField()" on both Model1 and Model2.
Run a schema migration on app1 and app2
Run a data migration on app1 and app2 copying the id of the corresponding base model into the "tmp_id" field.

Prepare the BaseModel to be Abstract

There are two things you will need to change in your BaseModel class before you can add the "abstract = True" property. First, you will need to fix the related_name for every ForeignKey field in BaseModel. If you don't do this, there will be multiple tables with ForeignKeys on the User object with the same related_name and this conflict will cause an error. Adding the name of the class to the related name as shown in the "after" state of the database will solve this issue.

The second issue is that you will have to remove any managers on BaseModel. These managers will throw an error and instead need to be bound to the child classes.

Mark the BaseModel as Abstract and Create a New Primary Key

There is yet another gotcha with this step. When we add the property "abstract = True" to BaseModel, after django deletes this model, it will automatically add an "id" field to Model1 and Model2 and it will make this field a primary_key. This is fine if we have no data, but since we do have data, we have no way of specifying what to set these initial "id"s to, and since it is a primary_key and there can't be any overlaps, we simply can't do this. Conveniently, we have this "tmp_id" field full of unique values, and we can use this as our primary_key. We can specify this when we make our migration:

Add the property "abstract = True" on BaseModel.
Add a field "id = models.IntegerField(primary_key=False)" on BaseModel
Change the "tmp_id = models.IntegerField()" on Model1 and Model2 to be "tmp_id = models.IntegerField(primary_key=True).
Run schema migrations on app1, app2, and base_app.
Run a datamigration that copies the relevant data from BaseModelCopy to every corresponding object in Model1 and Model2. To do this, just make use of the "tmp_id" field we created and the new BaseModelCopy class. Make sure this datamigration also copies the "id" field.

Remove all our Temporary Models and Fields

Our database should be near-perfect now and you should be able to run the development server to view your site. Inserts will not work yet because we have not specified a way to auto-increment the primary key of objects when we insert them, but we'll get to that later. Right now, if we see all our data has migrated successfully, we are ready to remove all this temporary stuff we created.

Remove the "tmp_id" columns from Model1 and Model2.
Change the "id" field on BaseModel to "id = models.IntegerField(primary_key=True)".
Remove the model BaseModelCopy.
Run schema migrations on app1, app2, and base_app.

Build the AutoField Functionality on Your Primary Key

The final gotcha of this process is that Postgres handles the django field models.AutoField() a bit strangely. You can inspect this using pgadmin or whatever gui you have to look at how it adds a "serial" property to the field. Because of this peculiarity, south will not allow you to migrate from an IntegerField() to an AutoField() out of the box. There is a workaround for this migration, but if that seems to messy to you, you can simply handle the auto-increment manually. Handling it manually only takes a couple of lines, but make sure you implement some sort of database locking to ensure you don't run into race conditions.

If you are not using Postgres, you may be able to completely skip this step and instead just change the id field on BaseModel to "id = models.AutoField(primary_key=True)" and run a migration, but I can't confirm this.

Push Changes to your Production Server

I'm assuming you made all of these migrations on local servers and not a production server, so the final step is to make the jump to deploying the changes. We now have our new schema with all the data filled in, and it should be working nicely with our code base on our local server. However, our set of migrations poses some problems for when we migrate. We can see that each set of migrations we did in the steps above are reliant on the full set of migrations having been completed in the previous step. This means the migrations for one app are reliant on the migrations for another app so going through a full set of migrations for a single app would not work.

To avoid this issue, we could have pushed each individual set of migrations along with the code to the production server right as we generated them. Alternately, we can simply clear out the migration history for each of our apps, dump the local datastore, and then rebuild the production database using these migrations. This is the method I used because I happened to be doing this at 4am when no one users were generating new data on the stie.

So, these are the steps for how to do this:

On your local server, remove all of the migrations from from migrations/ folder for app1, app2, and base_app. The history is no longer important to us because we did not specify a way to do backwards migrations anyway.
On your local server, create initial migrations using south for app1, app2, and base_app, clearing any ghost migrations (python manage.py schemamigration app1 --initial --delete-ghost-migrations).
On your local server, run the migrations. South will tell you that nothing needs to be changed. This is good.
Save all your changes to git or whatever you use.
Dump the local database (pg_dump --no-owner --no-acl -U username postgres_db_dev > postgres_db_dev_dump.psql).
Clone your working repository onto your production machine.
Copy your sql dump to your production machine.
Delete the production db (dropdb -U username postgres_db).
Re-create the db (createdb -U username postgres_db).
Load the new data (psql -U username postgres_db < postgres_db_dev_dump.psql)

Now your production server should look identical to your local_server that you had a working version on. Now that they are synced, you can go back to using whatever regular deployment scripts you use and everything should be just fine.

Conclusion

The methods used in this tutorial are useful if you have a live server with data you don't want to lose, but also have a span of time where you can be confident that users will not be adding new data. If users had entered new data into the production site during this process, their data would be lost during the final deployment to the production server. My site has negligible traffic from 1am-7am when I did this, and the database is small (<50,000 objects of type BaseModel) so I could do this quickly. To overcome this on a medium-high traffic site that uses user generated data, you should really schedule some "read-only" downtime and alert your users of this to ensure they don't enter data that gets overwritten.

Installing libjpeg and PIL on OSX Snow Leopard with Python 2.7

Mark Liu — Sat, 20 Aug 2011 11:58:29 -0500

If you found this article through a google search, you've probably already read through tons of articles and forums where people gave advice on how to do this. Here's some examples of what I'm talking about:

http://www.brantsteen.com/blog/python-27-libjpeg-pil-on-osx/
http://proteus-tech.com/blog/cwt/install-pil-in-snow-leopard/
http://www.thetoryparty.com/wp/2010/08/31/pil-on-snow-leopard-_jpeg_resync_to_restart-error/
http://www.brambraakman.com/blog/comments/installing_pil_in_snow_leopard_jpeg_resync_to_restart_error/
http://appelfreelance.com/2010/06/libjpeg-pil-snow-leopard-python2-6-_jpeg_resync_to_restart/
http://djangodays.com/2008/09/03/django-imagefield-validation-error-caused-by-incorrect-pil-installation-on-mac/
http://colbypalmer.com/index.php?/colbyworld/blogEntry/install_libjpeg_and_pil_on_os_x_leopard/

It's clearly an annoying problem, and I'm guessing none of those worked for you. While this is clearly an annoying problem and I spent at least a couple hours of head banging myself, I believe most of peoples' issues can be solved by completed a thorough cleanup. If you tried one or more of those approaches above, you probably have libjpeg and PIL folders in all sorts of places on your system that are screwing up stuff.
To remove the libjpeg files, make sure you remove anything that looks like libjpeg* or jpeglib*. You'll also want to remove all your PIL directories. Some of the most common directories these might be living in are:

/usr/local/include

/usr/local/lib

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/

/sw/lib

/sw/include

/opt/local/lib

/opt/local/include

Be sure to do this between trying any different installation methods or you will end up wasting a lot of time.

For me, once I removed all these files, getting libjpeg and PIL to work was actually quite simple:

Download this combo installer and install it with the default settings.
Run "pip install pil"

I hope this helps someone!

Edit 1/7/2012: It seems that PIL is broken. Some other people revived the project by creating "Pillow", so you can just change step two above to:

pip install pillow

Setting up my Hackintosh on my Toshiba Satellite M55-S329

Mark Liu — Fri, 19 Aug 2011 08:52:25 -0500

I'll go through the process of turning my 6 year old Toshiba Satellite into a brand new Hackintosh. So far it's working great! Wifi and all! And best of all, it didn't take much time at all to set up.

First, I have to give credits to the two best sources for this that I found:

http://www.insanelymac.com/forum/index.php?showtopic=167283

http://wiki.osx86project.org/wiki/index.php/HCL_10.5.2/Portables#Satellite_m55-s3293

Not everything went totally smooth for me, though, so I'll walk through each step in a bit more detail than in those links above.

First, you should download iDeneb 1.4 via torrents. This is an iso image of an OSX installer that works phenomenally.
Burn this iso image onto a disk. If you're already using OSX like me, use disk utility as explained here.
Partition your harddrive. Since you probably already have windows installed on your Toshiba, you can download the free Partition Wizard which works beautifully. You need to make a fairly large OSX partition. Since you're in windows you won't be able to format this partition as Mac OS Extended Journaled, but you need to create a blank partition and give it a drive name such as "D:\".
Insert the iDeneb disk and restart the computer. Boot from the disk by hitting F12 at startup.
When you reach the first iDeneb install screen, it asks you to choose which drive you would like to install it on, but doesn't show any drives to choose from. This is because you need to format your blank partition. Simply choose Disk Utility from the utilities drop down, select your blank partition, and erase that partition while choosing Mac OS Extended Journaled. You can then exit the Disk Utility and choose this new partition you created as the place to install osx.
At the screen where it prompts you to install, you need to first click on the "options" or "configuration" button to choose which drivers (kexts) you want installed. Choose exactly the options specified here.
Click install and wait a while. Then just go through the regular startup questionaire.
After it boots up, everything except wifi/ethernet should be working. Download the post-install files from this insanelymac thread using a computer with internet. Also download Intel PRO/Wireless 220BG driver. Place them on a flash drive, and open the folder on your Toshiba.
Install the AppleYukon kext, then the Seatbelt Kext, and finally the Intel Pro/Wireless driver kext that you downloaded from project: camphor. You can use this guide for installing kexts.
Reboot and cross your fingers!

The thing to note that I did differently than in those links above is that I used a different method for getting wifi up and working. I didn't use the iwi installer, and instead used the camphor installer.

After using my new hackintosh for a little bit, the biggest shortcoming is that the wifi can connect only to unauthenticated or WEP encrypted networks, not WPA or WPA2. This is a bit inconvenient. Further, since this computer is so old, even though OSX is running smoothly, the hardware limitations make streaming video choppy, and of course the battery life kind of sucks.

That being said, I now have a fully working hackintosh that I can use for development when I'm away from my desktop. Awesome.

Scheduling Regular Backups on Webfaction

Mark Liu — Sun, 14 Aug 2011 10:30:31 -0500

For Leaguevine, we take backing up our information seriously and have deployed a custom backup system for our database. I figured I would share how we back up our databases here. Note that this sort of backup technique is certainly not a replacement for source control or backing up user uploads, but that's a topic for another day.

First, we have to decide on a schedule for what copies of the database we want to keep, and how long we want to keep them for. For Leaguevine, we decided on the following scheme:

Hourly backups for the past week
Daily backups for the past month
Monthly backups indefinitely

After defining a schedule, creating these backups on webfaction is as easy as creating some folders and specifying commands to periodically dump the database information into those folders. Thus, we have directories for each the hourly, daily, and monthly backups. To dump the data into files within these directories and write the success/failure of these dumps to a log, we create new cron jobs. For example, if we open up our list of cron jobs, we can add a single command to do all this. It might look something like:

30 * * * * /usr/local/pgsql/bin/pg_dump -Ft -U db_username db_name > /path-to-backups/leaguevine/hourly/leaguevine-hourly-`date +\%a\%H`.tar 2>> /path-to-backups/leaguevine/hourly/backups.log && echo "Database backup completed successfully on `date`" >> /path-to-backups/leaguevine/hourly/backups.log

There is a lot going on here. First, we notice that this job will run every hour on the half hour due to the way we defined the cron job.

Next, we specify the database name and username for the PostgreSQL database that Leaguevine uses. For this to run without needing to prompt the user for input, we need to set up the .pgpass file which just takes a second.

The next thing this command does is specify the folder we want to dump the database to, along with the day of the week and the hour of the day. This works for us because the database dumps from previous weeks will get written over by files of the same name after a week passes.

The command then writes any errors to a file called backups.log, but if there are no errors it writes a success message that has a date stamp on it to that file.

This single command will create all of the hourly backups for a week without needing any manual maintenance. However, this has the shortcoming of residing on the same server as the production database. Thus, if something happened and all of the data on that server were wiped out, all of the data would be lost. Thus, we need to additionally copy these backups to a different computer regularly. To do this, you can just install another cron job on a different machine as so:

40 * * * * . ~/.ssh-agent; scp -o PreferredAuthentications=publickey username@web160.webfaction.com:/path-to-backups/leaguevine/hourly/leaguevine-hourly-`date +\%a\%H`.tar /local-path-to-backups/leaguevine/hourly/ 2>> /local-path-to-backups/leaguevine/hourly/backups.log && echo "Database backup completed successfully on `date`" >> /local-path-to-backups/leaguevine/hourly/backups.log

This cron job is very similar to the previous one in some ways. First, it runs 10 minutes after every supposed backup should have happened. Next, it uses scp to grab the backup and store it on the local disk. And finally, it handles the logs the same way as before.

The trick here is to set up your .ssh-agent so that scp can run without needing a login. This isn't too hard, and webfaction has good docs for how to use ssh keys.

And that's it! With just a few lines in your cron files, you can have customized automated backups of your important data.

Backing up my Hackintosh

Mark Liu — Thu, 16 Jun 2011 23:30:29 -0500

I am moving soon and every time I have to physically transport a desktop computer I worry the jolting will mess up a hard drive. Because this Hackintosh took me the better part of a day to set up, I decided it would be wise to make a bootable clone of my OSX partition. I didn't want to just back up my files, but instead want a copy of the operating system so if my current boot drive dies, I am back up and running in at most a few minutes. Backing up files is easy, and to do that, just use Mozy, Carbonite or Crash Plan (I use Crash Plan and absolutely love it).

I looked at some forums that pointed me in the right direction for a solution to creating this bootable clone, but didn't find detailed instructions for the best way to do this. Thus, I'm going to write down the whole process I used here. It was surprisingly easy and free!

Add a new hard drive to your machine or create a paritition where the backup will sit. It must be equal to or larger than the size of your OSX boot partition.
Format this new partition using Mac OS Extended (Journaled). You can use the OSX Disk Utility for this.
Download Carbon Copy Cloner. It's totally free and is amazing.
Open the Carbon Copy Cloner .dmg and follow the easy instructions to make a clone of your current boot partition to your new backup partition.
Download iBoot and burn this to a CD. You probably already have a copy from your original Hackintosh install.
Turn off your computer, unplug the cable to the original boot drive.
Boot your computer and open up the bios to allow you to boot from the iBoot CD you created. It will show you your new cloned partition, and you can select it and boot from it!
You still need to install a boot loader if you don't want to always have to boot from a CD. Thus, download Multibeast and run it. Select only the option to install the Chameleon Boot loader. All the rest of the options are already installed on your clone.
Eject your iBoot drive and restart your computer. It should boot up!
You probably want to switch back to your original OSX partition for booting up, so just plug it back in, go into the bios to ensure it is higher on the HDD boot priority, and then restart your computer.

I'm really happy about how easy this was to do, and now I have the peace of mind of knowing I won't have to go through the process of setting up a Hackintosh again if my current drive fails.

Connecting to a Mac from an iPhone over 3G

Mark Liu — Mon, 23 May 2011 18:22:19 -0500

A couple hours ago I realized I may need to connect to my Hackintosh this weekend while I'm out of town. Turns out, there are some really cool tools to do this securely for only $2! It took me a bit more than an hour to set up, so I thought I'd write an entry for future reference.

In this tutorial, I'll be setting up both VNC and SSH on OSX 10.6 and connecting to it over 3G using an iPhone 3GS.

The first thing you need to do, is set up a VNC server. There are a number of free VNC servers for OSX that you can download, and I chose to go with Vine Server. You can use the default configurations and simply start the server.

Next, you'll need to open up a port on your router so that incoming requests won't be rejected. The standard port for SSH is 22. I followed this tutorial for opening up my port. Basically, all I had to do was:

Open up my router's control panel by typing http://192.168.1.1/ into my address bar. If you don't know your router's default password like me, you can look it up here.
Find the section in the settings where you can add port forwarding, and for the port put 22 and for the IP address, use the internal IP address of your machine that the VNC server displays. This will be something like 192.168.1.101. If you can't find the settings for port forwarding, this port forwarding tutorial site has good instructions for a number of routers.

Now that the router is forwarding external requests to your Mac, you need to allow SSH access to your Mac. To do this, you can go into System Preferences and enable "Remote Login".

Your Mac is now set up for accepting SSH connections on port 22, and someone who has tunnelled in can then connect to the VNC server on port 5900. Here is how to actually do this:

Download an iPhone VNC client app such as Remoter VNC. This one costs $1 for basic VNC connections and has an additional $1 addon for SSH connections.
For the SSH hostname, enter the IP address of your home computer. If you don't know this, just go to http://www.whatsmyip.org/ and it will tell you.
For the SSH username and password, use the username and password for your Mac.
For the VNC hostname, enter the internal IP address of your home computer. This is the one that looks something like 192.168.1.101.
For the VNC port, enter 5900.
If you set up a VNC password, be sure to enter that as well.

After all that, you should be able to have full access to your home computer through a secure connection no matter where you are! It's a big relief for me knowing I can access my stuff while I'm away this weekend.

Webfaction Rocks!

Mark Liu — Thu, 21 Apr 2011 10:14:08 -0500

About a year ago I began using Webfaction shared hosting for this blog and a handful of webapps I'm running (2 Django apps, 1 Rails app, and a wiki). I have been blown away by the service they offer for the amazingly low price. I'll give you an example of what I'm talking about.

How they deal with bad neighbors

The first thing they do that's great is they don't overload their servers so you are not likely to see a performance drop because other people's misuse of the server.

Second, they automatically send warnings to users if they go over what they pay for, and if the user does not fix the problem, they automatically terminate some of that user's processes to ensure the server stays healthy. How do I know they do this? I was recently a bad neighbor myself (on accident of course :P).

My webapp, Leaguevine, relies on fetching data from Twitter at regular intervals to keep a local copy of relevant Twitter information up to date. This had been working fine for many months. However, when Twitter went down for a bit, the app began building up an excessive amount of processes trying to fetch this Twitter information, and I neglected to build a way to systematically kill off these processes if Twitter was down for a while. I ended up going noticeably over my alotted memory usage.

Here's my experience with webfaction when this happened:

Webfaction contacts me with an automated and friendly worded email saying I went over the memory limit. This email is extremely helpful, and shows a list of processes and how much memory they are using. Further, it tells me how to reproduce the data that they showed me and gave me helpful links for how to reduce my memory usage.
Webfaction sends me a second email a few hours later with the same basic information. I was away from my computer, so I didn't see the first one right away. This email, like the last, asks me to email them back to tell them if I was able to remedy the problem.
I fix the problem by adding a single line in my crontab file.
I send them a response to their email saying how I fix it, but wasn't overly specific since I honestly didn't expect them to read it.
Webfaction emails me back, saying they appreciate that I fixed it, but they were concerned that my fix will hurt the performance of my webapp. This really showed that they cared about my site! Before this email, I had never experienced a hosting provider that cared about what I was actually hosting beyond simply ensuring that I didn't surpass the limits.
I emailed them back, explaining the fix in more detail and why it wouldn't hurt my site's performance.
Webfaction emails me back cheerfully, saying they think what I did was good.

I'm sure that most hosting providers provide these automated messages and the first four steps of that experience would be identical even with a lesser hosting provider. However, that 5th step shows how far they go above and beyond the typical hosting service. Because of that, I'm sure I'll be recommending Webfaction to anyone who will listen for a long time to come.

Other things I love about them

Their documentation is absolutely amazing. They cover detailed step by step instructions for every reasonably popular web app, and a lot of less popular ones as well.

For less popular web apps, they have tons of custom install scripts with step by step instructions, so you custom installation is a breeze.

Finally, they have a great community that will help you with basically any problem you're having. I've asked three questions, and two of the three times I've asked those questions I've gotten answers that solved my problem within an hour.

Just use them! You won't regret it

I feel so strongly that you should use webfaction for shared hosting that I'm not even going to include a referral link here. I don't want you think that this entry was just a plug so that I could make a few bucks. I am honestly impressed with their service, and it has been far better than any other web hosting services I've tried in the past. If you browse the web for other reviews, I think they'll all say the same thing :)

My Recent Hackintosh/Windows/Linux Install

Mark Liu — Fri, 14 Jan 2011 11:08:23 -0600

I recently built my first Hackintosh and I am extremely pleased with it so I thought I would write about my experience. I needed a new computer and really like OSX, but I was not willing to pay the price for a new Macbook or iMac. Since a desktop is perfectly fine for my purposes and I already had the monitors for it, I figure I'd save some money and build an awesome machine running OSX. Thus, I bought everything on Newegg on black friday.

Here are the specs:

Intel Core i5-750 Lynnfield 2.66GHz LGA 1156 95W Quad-Core Processor
OCZ Vertex 2 3.5" 90GB SATA II MLC Internal Solid State Drive (SSD)
Antec Sonata III 500 Black 0.8mm cold rolled steel ATX Mid Tower Computer Case 500W Power Supply
ASUS DRW-24B1ST/BLK/B/AS Black SATA 24X DVD Burner
EVGA GeForce 9500 GT 1GB 128-bit DDR2 PCI Express 2.0 x16 HDCP Ready SLI Support Video Card
SAMSUNG Spinpoint 2TB 5400 RPM SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive
G.SKILL Ripjaws Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory
GIGABYTE GA-P55-USB3 LGA 1156 Intel P55 USB 3.0 ATX Intel Motherboard
Snow Leopard Install CD (the cheap $29 upgrade one that Apple sells)

Of those, I learned the most important things you need to check for compatibility with are the motherboard, the processor, and the video card. In short, what I found after research was that Gigabyte boards seem the most well supported, basically everyone uses the i5-750 or i7-860 processor, and there are a ton of video cards that work if you're willing to put in the time to fiddle with it.

To get going, I primarily used these four articles which I highly recommend reading:

I'll now go through my experience. It was a bit more complicated because for some reason I wanted a triple boot OSX, Windows 7, and Ubuntu installation.

First, I installed OSX (partitioning my hard drive into 3 partitions along the way), and updated the OS to 10.6.5 using the tonymac link above. I had no problems with this! Next, I jumped over to the first lifehacker tutorial and installed Windows 7, and had no problems there either. Finally, I installed Ubuntu 10.10 which I downloaded for free via bit torrent and didn't have any problems with that either. To get these working, I needed to play with some internals and get the chameleon boot loader working, as described on lifehacker. However, once I got the chameleon boot loader to work, OSX no longer worked for some reason... I never could figure out why.

At this point the chameleon boot loader, the Windows 7 partition, and the Ubuntu partition all worked perfectly, but I had a dead OSX partition. I decided to wipe OSX clean and start with a fresh install. I booted using iBoot, and then put the OSX 10.6.3 install disk I bought from the store in. I updated the OS using the 10.6.5 combo update (1gb free download from apple). It is important to not restart the computer after completing this update!

After doing this, my OSX partition worked, but there was no audio and the video resolution was terrible because it wasn't utilizing my video card at all. As a side note, even though OSX didn't "recognize" my video card, the display still worked even though I connected it to my video card. To get the video card and audio working, I installed Multibeast from tonymac with the following things selected:

UserDSDT (placed the DSDT-GA-P55-USB3-2.0-F8.aml file on the desktop)
System Utilities (all)
Under advanced options:
- NO AUDIO
- NO DISK
- NVEnabler
- Realtek R1000SL
- FakeSMC
- 64-bit-Apple Boot Screen

After this completed, I restarted the computer. Next, I installed the realtek driver named RealtekRTL81xx-0_0_67.pkg located at http://lnx2mac.blogspot.com/p/realtekrtl81xx-osx-driver.html.

I then restarted the computer again. Next I installed VoodooHDA.kext (version 0.2.56) by downloading it from http://nawcom.com/osx86/files/10.6/Audio/VoodooHDA/0.2.56/kexts/ and dragging onto something called kext Utility which I downloaded at some point.

Finally, after restarting my computer one final time, everything worked perfectly! And now I have a relatively inexpensive iMac thanks to about a day of work and Newegg's insane deals during black friday. Its XBench score is 273 in case you were curious.

Looking back on it, I don't think I needed to partition my hard drive 3 ways. It has been working for over a month now and I never touch Windows or Linux anymore. My OSX partition absolutely never crashes which is fantastic. For reference though, if something were to happen with the OSX partition, these partitions could access each others' file systems by using this guide.

I hope this helps someone out there!

Reflections on Creating Stall9

Mark Liu — Wed, 21 Jul 2010 12:40:10 -0500

Okay, so Stall9 is not actually finished yet, but I thought I would still reflect on the journey so far. In case you haven't heard, I am building a website called Stall9 that will allow users to collaboratively update the ultimate frisbee community on what is currently going on. The premise of the site is that any person, regardless if he/she is a player, fan, or administrator, can update the information on stall9 to provide online followers with the most real time information possible. The goal is to make updating the site as easy and simple as possible, supporting already widely used technologies like twitter.

I got the idea for this site last summer in India while on a bus daydreaming about ultimate and looking forward to playing with a Madison club team called "Test Tickle Me Elmo" in a couple months. Soon after I had this idea, I went on a 21 day backpacking trip in California (the John Muir Trail if you were curious) and had just a ridiculous amount of quiet thinking time to flesh this idea out. I came back to Madison, WI to start my Master's program in September 2009 and hoped to finish this site before College Nationals came to Madison in May, 2010. However, I realized I knew nothing about programming real websites.

I had dabbled with web programming for several years, but had always used pre-built content management systems and just hacked certain extra features in. For Stall 9, I knew I would need a much more robust user permissions system, and a very dynamic way for users to enter, modify, and moderate all sorts of information. To do this, I decided to learn Python web development and a web development framework called Django. I don't want this post to turn out technical, so I won't talk about any of those specifics.

So that fall, I played around with creating a simple version of Stall 9, but was unable to get very far. I didn't put in nearly as much time as I wish I had, and instead ended up spending a lot of time getting acclimated with my new Master's program. Then I took 5 weeks off for winter break which included visiting family and a 3 week trip to Kenya with Engineers Without Borders.

By the time I got back, it was February and I still had made no progress on this site. However, by this time I had gotten comfortable with my Master's research to the point where I had enough free time to really put some good work into Stall 9. So I plugged away at it and put in a reasonable 200 hours over the next 2.5 months.

Then I hit a wall.

The code base for my site was enormous and if it wasn't already unmanageable, it was quickly become so. I took a step back and thought that there must be a better way to do this. I bought a couple programming books that were more advanced than my previous ones and learned about how to reduce redundancy and keep even the most complex projects manageable. After my first few hours of reading, I knew I still had a lot to learn.

I then completely shut down development and spent about a month intensely reading up on best practices and learning from the experts in the community. It was towards the end of May 2010 that I decided I learned enough to get by.

At this point, college Nationals in Madison was right around the corner and I was extremely disappointed I didn't have a site to show for all my hard work. I was second guessing myself for the first time in the process and wondering if I should continue on with the project. However, I was greeted with a really nice surprise that got me right back on my horse. USA Ultimate unveiled their new website and shockingly it had no more functionality than their previous one. In fact, it seemed that most Americans were really annoyed by the fact that their new site did not really do much to address a fan's needs. What fans need are real time updates and easily navigable information. Since I saw a problem that needed to be fixed, it was easy for me to get back into the swing of things.

First though, I needed to test my new found programming prowess. I built this site in about a week using the best practices I learned, and then deployed it so I could practice deploying a Django site.

Finally, about a week into June, I began working from a clean slate on a new version of Stall 9. That old site will never see the light of day, but that's probably best for everyone :). Over the last 5 or 6 weeks, I am back to the point where I was with the first version of stall 9, but this time my code base is much more concie, robust, and manageable. Instead of my development slowing down because of the size of the code base like before, it is actually accelerating at an astonishing rate. At this pace, a somewhat unpolished version of Stall 9 will be ready for launch sometime in September! As a side note, I am lucky enough to be going to Kenya for 3.5 weeks again in August followed by a bachelor's party in New Orleans, so that is why Stall 9 won't be ready sooner!

If you are interested in learning more about Stall9, please don't hesitate to contact me. I am building this site to fill a void in an ultimate fan's experience, and since I'm certainly not representative of all ultimate fans, I can always use more input from others.

Code For This Site

Mark Liu — Mon, 12 Jul 2010 09:07:20 -0500

I've decided to make the source code for this site publicly available. You are welcome to download it, use it, modify it or do anything else you want with it. It is available here. I have also uploaded a rather detailed README that should get the site up and running for you if you do decide to play around with it.

Getting this site up and running took a bit of time and during the process I was always wishing I could just see the full source code for some other django sites. This site uses the re-usable apps philosophy where the site itself is just a settings file and a url conf while all of the actual work is done by a number of reusable apps. While this philosophy makes perfect sense and seems like a great way to build a site, almost all of the django tutorials out there do not preach this view. Instead, they tend to show overly-simplified example sites that have a monolithic design. Thus, I hope my source code provides a useful reference to someone else out there who wants to adopt a reusable apps approach to their django site.

Enjoy!

MySQL and Python on 32 bit Snow Leopard

Mark Liu — Wed, 09 Jun 2010 15:52:29 -0500

I am running Mac OSX 10.6 (Snow Leopard) on my home machine and had previously been chugging along just fine with my Python 2.5 and MySQL 5 setup. However, this morning I decided to try and get MySQL to play nicely with my Python 2.6 installation and I ran into all sorts of problems. I searched online for solutions and found that a ton of other people had problems as well, but none of their solutions worked for me. I did get mine working after a couple of hours of fussing and all of the problems ended up being the result of incompatible architectures.

I'll outline the steps I would take to get this running again on my machine and why. Here is the target setup:

MySQL 5.1, 32 bit
OSX 10.6.2, 32 bit
Python 2.6, 32 bit

First, I'm going to remove my previous MySQL installation. But even before that, we obviously need to back up our database. Here's how we would create a dump of it if we have the mysqladmin installed.

$ mysqldump -u username -p -v databasename > databasename.sql

Next, we need to thoroughly uninstall MySQL:

$ sudo rm -rf /usr/local/mysql*
$ sudo rm -rf /var/db/receipts/com.mysql.mysql
$ sudo rm -rf /private/var/db/receipts/com.mysql.mysql*

Now that we've removed our previous install, we'll go to MySQL's download page and download the 32 bit .dmg package. Installing this is as easy as double clicking it after the download finishes.

Once we have MySQL installed and working fine, we need to install the MySQL-Python bindings. We'll go to the download page, and download the most recent version. Once you have this installed, cd into the directory and build it using the correct architecture flags.

$ ARCHFLAGS='-arch i386' python setup.py build
$ sudo ARCHFLAGS='-arch i386' python setup.py install

Finally, we need to ensure that we are using the 32-bit version of python. One way to tell if you are using the 32-bit or 64-bit version is if you enter your python interpreter and type:

import sys
sys.maxint

If the output is 9223372036854775807 then you are using the 64-bit version of Python. If it is 2147483647 then you are using the 32-bit version.

The way to force your system to use the 32-bit version is to use the command:

$ export VERSIONER_PYTHON_PREFER_32_BIT=yes

I added this to my .bashrc and then refreshed my terminal by typing:

$ source ~/.bashrc

Other alternatives to forcing the 32-bit Python are listed here.

Having done all that, you should be up and running! Hopefully this saves someone time, and if not, hopefully it will at least be a useful refresher for me next time I need to do something similar.

UPDATE: It turns out this was not the end of my problems since I also am using virtualenv to isolate my Django project installations. With the default way virtualenv is set up, it inherits a python interpreter from somewhere that does not take into account the versioner flags we defined. Unfortunately, the default Snow Leopard installation comes with python in 64-bit mode, even if you are running OSX in 32-bit mode and thus, this is the python interpreter loaded into your virtual environment. After a lot of work and fiddling with things, I found the way to get around this is to do the following:

Install Python from a .dmg which you can download on the Python website. The .dmg package from the Python site is the 32-bit version so this is perfect for us in this case. Mine installed by default into the directory /Library/Frameworks/Python.framework/Versions/2.6/bin/python
When you create your virtual environment, specify which python interpreter you would like to use. For me, this was:

$ cd /srv/python-environments/
$ virtualenv --python=/Library/Frameworks/Python.framework/Versions/2.6/bin/python new-environment

Also, if you need to check on which version of python is being called at any given moment, the following two commands will give you useful output.

$ which python
$ python --version

Single settings file for Django deployment on Webfaction

Mark Liu — Sat, 05 Jun 2010 11:30:17 -0500

After only a few days of this blog being up, I already became annoyed with having to maintain separate settings files for development and production. Thankfully, fixing this problem was incredibly quick and easy and should save me some headaches in the future. All I did was determine what host the settings file resides on and then set the parts of the settings file that change between development and production. Here is what the relevant part of my Django settings file looks like:

That's it!