Excel remains the de-facto graphing tool at National Priorities Project. A simple chart is often the best way to convey information about federal spending and budgeting, and Excel is the common language among our researchers and IT team.

Using Excel, however, is no excuse for ignoring style and the best practices of information display. So many organizations put out amazing, well-researched publications and then tack on default Excel graphs as an afterthought. But graphs are often what people look at first, and they deserve to be first-class citizens in the editing process.

I created some Excel chart templates for NPP, drawing on two sources for inspiration and practical advice: the classic Visual Display of Quantitative Information by Edward Tufte and The Wall Street Journal Guide to Information Graphics by Dona Wong.

Tufte is big on eliminating “unnecessary ink” that distracts from the information, and Wong advocates requiring the least amount of work on the reader’s part. With their advice in mind, I modified Excel’s default bar chart from this:

Bar chart: Excel default

To this:

Bar chart: new template

• Smaller gap between bars
• Don’t make readers guess the numbers; if possible, label the bars directly
• Direct labeling means you don’t need the noisy gridlines or even the x-axis
• Remove the y-axis tick marks for even more noise reduction
• Get rid of those zeros by showing data in millions or billions
• Make sure the entire length of the bars is shown (in this case, by setting the horizontal axis minimum to zero). HT Jon Peltier.

The pie chart got a similar treatment. The Excel default:

Pie chart: Excel default

The new template:

Pie chart: new template

• Label the pie slices directly—don’t make people use a legend to decode
• Avoid the default Office color palette and develop your own (ours is based on colors from our website)
• A white line between pie slices emphasizes the boundaries

Excel isn’t perfect, but it’s out there in the world, and you can’t ignore it. Luckily, a little extra effort goes a long way.

So when writing a brief paragraph to introduce a new dataset in the Federal Priorities Database, I thought I nailed it:

Federal tax collections represent the amount of federal taxes paid in a year.  The IRS publishes this information by state, breaking it into ten categories.  Because part of our job is to make government-published data more useful, we condensed these ten categories into three: taxes paid by businesses, taxes paid by individuals, and total taxes collected. We believe that these broad categories better reflect what most people want to know about tax revenue and are less confusing than the original data. In fact, the individual taxes calculation played an integral part in a recent publication, Federal Spending Keeps Iowa, New Hampshire Afloat.

I have a few trusted readers in the office, so I sent the text around for some quick feedback. About five minutes later, one of them appeared at my desk, saying that my language was non-accessible and “made her eyes roll back in her head.” Wow. Hard to hear, but she followed up with some specific suggestions that resulted in a much improved, non-eye-rolling version:

We recently took some data from the IRS and made it even better. We call it Federal Tax Collections, and it shows federal taxes collected in a year. Although the IRS helpfully provides these numbers for each state, they break the amounts into ten categories, not all of which are useful. For example, do any but the wonkiest people need to know how much railroad retirement tax was collected in 2010? To make things easier, we condensed these categories into taxes paid by businesses, taxes paid by individuals, and total taxes paid. We think these broader groupings are less confusing and more practical. Our recent publication, Federal Spending Keeps Iowa, New Hampshire Afloat, shows the individual taxes category in use.

Thank goodness for honest feedback! It seems obvious, but the lessons here are easy to overlook when you’re cranking through a to-do list.

• No matter what kind of deadline you think you have, it’s always worth the time to get other eyes on something your organization will publish, even if it’s just a small blurb, and even if your colleagues are busy. After all, it’s in everyone’s best interest to show the world your best stuff.
• Make sure your trusted readers include, as my friend calls herself, the dummy demographic, a term of affection for readers who aren’t experts in your subject matter.

Overall, however, being a technologist is a lot more exciting in 2012 than it was in the Y2K remediation days. That said, it’s time to stop worrying about every shiny new toy that comes along and focus on a few things that appeal to my interests and career aspirations.

NoSQL
I know my way around an RDBMS, but 2012 will be the year I stop reading about NoSQL and start doing something with it. Flavor TBD.

D3.js
Data visualization is more than a fad. Not those disjointed, meaningless infographics floating around, but numbers rendered as pictures that tell a story. I’m not a graphic designer, but I am a programmer, so it makes sense to learn a tool that can programmatically create pictures of the federal data I work with every day.

The administrator of a blog I follow recently asked the question, “should I force our blog’s authors to use Markdown?”

I then read a completely unrelated post on another blog that advised WordPress users to get rid of the visual text editor because “it makes you lazy” and you should “force yourself to learn some basic HTML.”

When I got into a heated discussion with another developer about the above two items, he used the same word to describe non-Markdown/HTML-writing, WYSIWYG-dependent online content authors: lazy.

This isn’t the first argument I’ve had with a technologist who likes to play the lazy card, and it won’t be the last. I’m not disputing that everyone in today’s workforce should always be learning; the days of doing the same job the same way for thirty years are over. But to imply that a blogging co-worker isn’t holding up her end of the learning bargain because she doesn’t want to learn HTML or Markdown is arrogant.

As technologists, we explore and experiment with new technology. Our content-creating colleagues presumably explore and experiment in their respective areas of expertise. Who are we to dictate that they should increase their cognitive overhead by worrying about valid XHTML markup?

Yes, WYSIWYG editors are pretty terrible. They puke out Microsoft Word detritus and let you change the font color to cyan. Forcing people to write in Markdown, however, shifts a technical problem from the technologist to the user, which is the opposite of ideal.

This rant isn’t about blog authors or Markdown. It’s about acknowledging that technology isn’t everyone’s primary concern, nor should it be.

Of course there are appropriate times to force technology changes. But even in those situations the resistors aren’t lazy–they just have a different job than you. In today’s hectic and stressed workplace, shouldn’t we give colleagues the benefit of the doubt and help them succeed with technology rather than slapping a label on them?

I achieved the goal, but not without a lot pain. If you’re a Windows user getting started with Python/Django, you might have an easier time installing a virtual Linux machine.

Here’s a re-cap of the Windows-specific instructions for installing Python, Django, MySQL, and a few necessary packages and tools.

• Part 1: installing Python, Pip, and iPython
• Part 2: virtual environments
• Part 3: making iPython recognize virtual environments
• Part 4: installing Django
• Part 5: installing MySQL

• Part 1: getting started
• Part 2: virtual environments
• Part 3: making iPython recognize virtual environments
• Part 4: installing Django
• Part 5: installing MySQL (you are here)

By now, you should have Django installed into a virtual environment.  These tutorials aren’t meant to cover building a django app, just to point out the quirks involved with getting a project up and running on Windows.  These tutorials also assume you want to construct real applications using a real development environment.

To that end, you’ll want a heftier database than sqlite.  We use MySQL at the office, so these instructions cover installing it and using it with Django.

Install MySQL

1. Download and install MySQL.
2. Once MySQL is installed, proceed through the configuration wizard. Check Include Bin Directory in Windows PATH box.
3. When prompted, set a password for the MySQL root account.
4. Once the installation wizard is done, open a command window and log in to MySQL with the root account: mysql -uroot -p (you’ll be prompted for the password).
5. After logging in, run the following commands to create a database, create a user for your Django project, and grant the user database access.

Install MySQL-python

You’ll need the MySQL-python package, a Python interface to MySQL.

1. Download the windows MySQL-python distribution here.  The author has some instructions about the appropriate version; assuming a 32-bit version of Python 2.7, you’d download this package (.exe).
2. After downloading, do not run the Windows installer. Doing so will install MySQL-python to your root python, which virtual environments created via –no-site-packages won’t be able to see.
3. Instead, install the downloaded package to your virtual environment by using easy_install, which can install from Windows binary installers:
   easy_install file://c:/users/you/downloads/mysql-python-1.2.3.win32-py2.7.exe (modify to reflect the location of the downloaded installer and its name).

Configure Django

Next, you’ll need to update the database-related settings of your Django project.

1. From the directory of your Django project, open settings.py using your favorite editor.
2. Update the default key in the DATABASES dictionary.  Set ENGINE to django.db.backends.mysql and set NAME, USER, and PASSWORD to the database name, username, and password you chose when installing MySQL.  See Part I of the Django tutorial for more information about database settings.
3. Open a command window, activate your virtual environment, and change to the directory of your Django project.
4. Type python manage.py syncdb. This command creates the underlying tables required for your Django project.
   
5. If the syncdb worked, you have Python, Django, and MySQL communicating in harmony.  Congratulations!  You can now proceed through the Django tutorial and create your first application.

• Part 1: getting started
• Part 2: virtual environments
• Part 3: making iPython recognize virtual environments
• Part 4: installing Django (you are here)
• Part 5: installing MySQL

These directions will simply make sure you can get up and running.

Installing Django

1. Open a command window.
2. Go to (or create) the virtual environment you’ll be using for your django project. For this example, I created a virtualenv called django-tutorial: virtualenv django-tutorial --no-site-packages

3. Install django: pip install django
 
4. Start an interactive interpreter by typing python (or iPython, if you’ve made it virtual environment-aware).
5. Test the install by importing the django module and checking its version: https://gist.github.com/1177372
6. Create a new directory to hold your Django projects and code. Change to it.
7. Think of a name for your first Django project and create it by running the following command: python -m django-admin startproject [projectname].
   Important: most Django docs show django-admin.py startproject [projectname] to start a new project, which can cause import errors and other trouble for Windows users. See this stackoverflow thread for details.
8. You should now see the project’s folder in your Django directory:
9. Change into the new project folder.
10. Test the new project by typing python manage.py.  Manage.py is Django’s command line utility; you should see a list of its available subcommands.
11. A further test is to start up Django’s development server: python manage.py runserver. You should see something like this:
    

If you’ve made it this far, you’ve successfully installed Django and created your first project.

Next up is Part 5: Installing MySQL.

• Part 1: installing Python, pip, and iPython
• Part 2: virtual environments
• Part 3: making iPython work in virtual environments (you are here)
• Part 4: installing Django
• Part 5: installing MySQL

The last installment covered setting up virtual environments.

Part 3: iPython and Virtual Environments

iPython is not virtual environment-aware

iPython is a valuable tool, but you’ll have to tweak it to work with virtual environments. By default, iPython isn’t aware of a virtual environment’s packages.

For example, if you install a package into a virtualenv and try to use it via Python’s built-in shell, everything works:

Try the same thing with iPython, however, and you get an import error:

A little intervention is required to make iPython virtual environment-aware. If you’re okay with using Python’s built-in interactive shell instead of iPython, skip ahead to Part 4: Installing Django.

Install iPython to the virtual environment

The easiest and most obvious solution to make iPython work with a virtual environment is to install it to the virtual environment.

However, you will have to do this for each environment you work in.  Furthermore, if you created the virtual environment without the –no-site-packages option (which tells virtualenv not to inherit anything from global site-packages), you may get an “already installed” message.

Modify iPython config file

Another way to ensure iPython behaves with virtual environments is to use its configuration file to check for an active virtual environment and modify the import path accordingly. The directions below assume iPython is installed globally but not in any of your virtual environments.

1. Open a command window.
2. Instruct iPython to generate a sample configuration file (called ipython_config.py) by typing ipython profile create
3. The sample file should now be in your iPython profile folder. On Windows, this is [your user folder]\.ipython\profile_default.
4. Open ipython_config.py in a text editor and add the following code to the bottom. I relied heavily on code from here and here, making a few tweaks.

   https://gist.github.com/1176035

5. Activate a virtual environment.
6. Start up iPython and look for the output confirming the current active virtual environment:

• Part 1: installing Python, pip, and iPython
• Part 2: virtual environments (you are here)
• Part 3: making iPython work in virtual environments
• Part 4: installing Django
• Part 5: installing MySQL

Part 1 of the Python, Django, and MySQL on Windows 7 tutorial covered the basics:  installing Python and some package management tools.

Set up virtual environments

We saw how easy it is to install Python packages such as iPython.  Just run the install command, and everything you need goes into Python’s site-packages folder.  But what if different projects require different packages or even different versions of the same package?  Of what if you have a project that requires an older version of Python?

If you’re planning to work on multiple applications, you need virtual environments to prevent a big mess of dependencies gone awry. Each virtual environment has its own installation directories, separate from other environments.

Virtual environments are brought to you courtesy of the virtualenv package.  To install the package and create a virtual environment:

1. Open a command window.
2. Install virtualenv by typing pip install virtualenv and waiting for the successfully installed virtualenv message.
3. Create a folder to hold your virtual environments (for example, c:\virtualenvs) and cd it (it’s easier if there are no spaces in the folder name).
4. Create a new virtual environment by typing virtualenv --no-site-packages [environment name].  The output will look something like this:
   
5. Look at your virtual environments folder. You should now see a sub-folder with the name of your new environment.
6. Create another environment, just for fun: virtualenv env2 --no-site-packages
7. Activate a virtual environment by running its activate script, located in the environment’s Scripts folder. In this example, the command is c:\virtualenvs\env1\scripts\activate.
8. The command prompt is now preceded by the environment name–that’s how you know the environment is active. The activate script also takes care of adding the virtual environment’s scripts folder to the beginning of your path.
9. Now that a virtual environment is active, try installing  a package (we’ll use python-twitter as an example): pip install python-twitter
10. Once the package is installed, you can see that it’s in the site-packages folder of the virtual environment, but the original Python install and the other virtual environments remain untouched.
    
11. To deactivate the environment, just run the deactivate script in [envrionment name]\scripts\deactivate

Bonus: Virtualenvwrapper is a handy set of extensions for virtual environments. This is a necessary tool if you deal with multiple environments on a daily basis, but it’s not essential right now if you’ll be working in a single environment. The Windows virtualenvwrapper port is here.

Coming next… getting iPython to work with virtual environments.

I found several helpful tutorials but ended up creating more detailed instructions to reflect my level of experience and our project structure. These notes became my dummies guide to setting up Python, Django, and MySQL on Windows 7.

• Part 1: installing Python, pip, and iPython (you are here)
• Part 2: virtual environments
• Part 3: making iPython work in virtual environments
• Part 4: installing Django
• Part 5: installing MySQL

Install Python

1. Install Python (I used the Python 2.7.2 Windows Installer).
2. Modify your path environment variable to include the location of the installed Python executable.  To change environment variables in Windows 7:
• Click the Windows start button in the lower-left corner of the screen.
• In the Search programs and files box, type environment variables
• When the search results appear, click Edit the system environment variables
• You should now see the System Properties window.  Click Environment Variables…
• When the Environment Variables window opens, choose Path from the System variables list and click Edit…
• Append the following location of the Python executable and the Python Scripts folder to the variable value, making sure everything is separated by a semicolon. For example, ;C:\Python27;C:\Python27\Scripts
• Click OK after modifying the variable value, and click OK again to exit the Environment Variables dialog.
3. You should now be able to bring up an interactive Python shell by opening a command window and typing python.
   

Get a package installation and management tool

Many kind people have written functionality-enhancing Python Packages, and sooner rather than later you’ll need to use some of them. Life will be much easier if you have a tool to install and manage these packages.

Two such tools are pip and easy_install (part of setuptools).  Despite some disadvantages on Windows, I chose pip after reading this StackOverflow thread.

Unfortunately, pip doesn’t have a Windows installer. The easiest workaround is to install setuptools (which does have a Windows installer) and then use it to install pip.  Directions below are a modified version of yet another StackOverflow thread.

1. Download the setuptools MS Windows installer that corresponds to your installed version of Python.
2. Run the setuptools install wizard.  It should automatically find your Python installation directory.
3. Make sure you have a way to extract .tar files (e.g., 7-Zip).
4. Download pip.
5. Uncompress the pip download
6. Open a Windows command prompt and cd to the directory that contains the uncompressed pip download (the folder structure looks something like pip/pip-1.0.x).
7. From that directory, type python setup.py install
8. After this script runs, you should see a confirmation message that pip was installed to the site-packages folder of your python install (this folder will house all of the Python packages you install).
9. If you get an error message, it’s probably because you haven’t added your Python directory to your path (see I above).

Install iPython

You don’t know it yet, but you need iPython.  It’s a richer version Python’s out-of-the-box interactive shell and has some extremely helpful features for people learning the language (object introspection, macros, code completion). iPython is easy to install now that you have a package management tool.

Even though I just had you install pip for this very scenario, it’s not the best choice for installing iPython on Windows. I promise pip will come in handy later, but to install iPython, use setuptools (setuptools will automatically install a module required for iPython’s code completion functionality, but pip will not).

1. Open a windows command prompt.
2. Type easy_install ipython
3. iPython should now download and install.  If you get an error message, make sure your path environment variable contains the Python \Scripts folder.
   
4. To use the iPython interactive shell, type ipython from a command window and peruse the built-in documentation.

The next part of Python, Django, & MySQL on Windows 7 will walk you through virtual environments.