jeffrey k eliasen

AWS

jeff — Wed, 13 Jan 2016 21:54:43 +0000

Amazon Web Services is the Budweiser of cloud computing.

Category:

Setting up virtual environments for Python in OSX

jeff — Sun, 21 Jun 2015 19:26:44 +0000

One of the things I wish I had known more about when I first started working with Python (and, really, any time before I finally learned about it a couple years ago) is the power of virtual environments for Python using the virtualenv tool.

The basic idea with virtual environments is that you have a project that might need a specific version of Python (2.7, 3, whatever), and specific versions of whichever packages you are using. For a developer working on multiple projects, this specifically allows you to avoid version hell where upgrading a package for one project breaks an unrelated project. This also ensures everyone on the team is using the same version of every package without having to add all the package source to the team's repository. All the virtual Python environments will exist in one place and you can switch between them with a single command.

This page will guide you through setting up virtual environments in OSX. The steps are similar for Linux but very different for Windows. Here are the highlights:

install the virtualenvwrapper package at the system level
set up your environment
1. disable pip for the system-level Python installation so that all packages are only ever installed in virtual environments (optional)
set up your top-level virtual environment directory
create your first virtual environment
learn the commands for using virtual environments

Prerequisites

You will need to have pip installed for this to work properly. In a terminal window, type pip. If you get a dump of how to use the command, you're all set. If you see a 'command not found' error, type the following:

$ sudo easy_install pip

You will need to enter your password. Accept the defaults for any options it gives you.

Install the virtualenvwrapper package

mkvirtualenv is the tool you will use to quickly create a new virtual environment or remove one you are no longer using. It's part of a package called virtualenvwrapper. To install this package, open a Terminal window and type:

$ sudo pip install virtualenvwrapper

You will need to enter your password. Accept the defaults for any options it gives you.

Set up your environment

You will need to set a couple environment variables for virtualenvwrapper to work properly. Open your .profile with TextEdit (or your favorite editor):

$ open ~/.profile

... and put the following three lines anywhere in the file:

export WORKON_HOME=~/dev/virtualenv
export PIP_REQUIRE_VIRTUALENV=true
source /usr/local/bin/virtualenvwrapper.sh

The first line tells the virtualenvwrapper set of commands where to find your virtual environments. I have a directory in my home directory called 'dev' and I put all external libraries and SDKs there... it's a reasonable place to put this as well. Some people put it at ~/.virtualenv, so you might see instructions elsewhere recommending this location. It doesn't really matter where it goes, but these two choices are probably better than most others if you have no strong reason to put it somewhere else.

The second line tells pip to refuse to install packages unless you are working in a virtual environment. This is a good thing. You might even consider removing all the packages you've previously installed so that you don't accidentally use one when you think you are in a virtual environment.

The third line tells bash (your terminal shell) to always turn on the virtualenvwrapper tools automatically.

You need to make these changes active in your terminal window because they weren't there when you started it:

$ source ~/.profile
$ echo $WORKON_HOME
/Users/seawolf/dev/virtualenv

You can also just close and restart the Terminal window and type the echo $WORKON_HOME command to make sure .profile is being loaded automatically.

Set up your top-level virtual environment directory

You need to create the directory where your virtual environments will live:

$ mkdir -p $WORKON_HOME

That's it!

Create your first virtual environment

I have a general virtual environment I call 'system' where I keep a hodge-podge collection of packages for testing, exploring, and just playing around.

$ mkvirtualenv system
New python executable in system/bin/python2.7
Also creating executable in system/bin/python
Installing setuptools, pip...done.

That was easy. You might see more lines of output as a few low-level tools are installed for the first time.

Commands for using virtual environments

The commands you will use the most are the commands to:

start a virtual environment
switch to a different virtual environment
install a package
uninstall a package
list all the installed packages
create a requirements.txt file
install all packages listed in a requirements.txt file
exit the virtual environment

Starting a virtual environment is easy: just type workon <environment name> (insert the correct name in place of the placeholder, obviously). Your command prompt will change to show you the name of the environment you are currently in:

$ workon system
(system)$

To switch between environments, you use the same command:

(system)$ workon my_happy_project
(my_happy_project)$

Tab works in all the commands to auto-complete where possible. In my case, system is the only virtualenv that starts with an 's', so I can just type workon s<tab> and it will auto-fill in the rest of the command for me.

Installing a package works just like you would expect using pip:

(system)$ pip install numpy
Downloading/unpacking numpy
  Downloading numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x
  Downloading numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x
  ... (lines removed) ...
86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (3.7MB): 3.7MB downloaded
Installing collected packages: numpy
Successfully installed numpy
Cleaning up...
(system)$

Uninstalling is equally easy:

(system)$ pip uninstall numpy
Uninstalling numpy:
  /Users/seawolf/dev/py_virtualenv/my_happy_project/bin/f2py
  /Users/seawolf/dev/py_virtualenv/my_happy_project/lib/python2.7/site-packages/numpy-1.9.2.dist-info/DESCRIPTION.rst
  /Users/seawolf/dev/py_virtualenv/my_happy_project/lib/python2.7/site-packages/numpy-1.9.2.dist-info/METADATA
  ... (lines removed) ...
Proceed (y/n)? y
  Successfully uninstalled numpy
(system)$

Listing all installed packages is done with pip freeze:

(system)$ pip freeze
numpy==1.9.2
wsgiref==0.1.2
(system)$

Notice that version numbers are shown, which makes it easy to see (and recreate) the combination of packages you are using.

By redirecting the output of pip freeze into a separate file, you capture the list for later use:

(system)$ pip freeze > requirements.txt
(system)$ more requirements.txt
numpy==1.9.2
wsgiref==0.1.2
(system)$

You can later use this file to recreate your environment on another machine (or in a different virtualenv):

(system)$ pip install -r requirements.txt
... (output as all packages are installed) ...
(system)$

Pip is smart enough to tell you if a particular package is already installed and will skip those, so you can just install from requirements any time someone else changes the file.

Finally, to exit the virtual environment altogether when you are finished:

(system)$ deactivate
$

That's it! Happy programming in your new virtual environments!

Sorry I slapped you...

jeff — Tue, 17 Jun 2014 23:24:43 +0000

Link:

Category:

Humor

I've got a package for you

jeff — Sat, 10 May 2014 21:09:34 +0000

Clever animated short:

Link:

Category:

Humor

Fresh guacamole

jeff — Tue, 06 May 2014 03:32:00 +0000

Awesome animated short:

Link:

Category:

Humor

Riding without a helmet may be good for you

jeff — Tue, 06 May 2014 03:30:27 +0000

Yes, you read that right... riding your bike without a helmet may be safer than riding with one.

Statistics are an interesting thing. They can very quickly relate large amounts of information. The problem is that without careful analysis they can create the wrong impression or give an altogether false understanding of the world.

Why It Makes Sense to Bike Without a Helmet is a simple example of this very concept:

People have tried to reason with me that because I’ve spent so much money and time developing my brain, and the cost of an injury would be so devastating, it’s clearly more important to wear a helmet. But if we start looking into the research, there’s a strong argument to be made that wearing a bike helmet may actually increase your risk of injury, and increase the risk of injury of all the cyclists around you.

The main points he makes are all valid, the sources are legitimate. What he's essentially saying is that if you get into certain kinds of accidents then having a helmet on is likely to decrease the severity of your injuries, but (and this is a big but) the fact that you are wearing a helmet possibly increases the likelihood that you will get into an accident in the first place.

That second point is important, but it is often overlooked. After all, our goal should be to reduce measurable injuries, right?

Well, no, it shouldn't be. The goal should be to teach awareness and responsibility. That's hard to do.

But we can at least try to reduce the overall number of accidents resulting in any kind of hospitalization. With this in mind, the fact that wearing a helmet increases the likelihood of an accident in the first place is an important piece of information. We don't know for sure that wearing helmets is a risk factor for accidents, but there are correlations that indicate something might be going on.

This sort of secondary effect is hard to measure, but it is important to understand. Simplistic rules or laws fail to recognize that the world is a made up of complex interactions between multiple systems. Just because a simple statistic sounds good doesn't mean it's actually good for society.

Category:

Politics

Building a log of songs played in iTunes over time

jeff — Thu, 17 Apr 2014 01:26:51 +0000

Recently I decided I wanted to perform a statistical analysis of all the music I've ever played in iTunes. This is non-trivial, as iTunes only stores the date and time each song was played last and does not keep any log showing previous plays.

With several copies of my iTunes library file, I can reconstruct the overall history. Luckily, I am using Time Machine and can therefore retrieve snapshots of the library. This is enough to reconstruct the history going back over a year.

The only problem is that Time Machine does not back up the .xml files, only the proprietary .itl files used by iTunes natively. This means that I must perform all the following steps:

retrive historical iTunes Library.itl files
convert each .itl file into the corresponding .xml file
parse each .xml file and build a log of songs played based on the last played parameter for each song
1. fill in any blanks with fuzzy data (for instance, if a song has been played twice since the previously-determined date, make a guess as to when the last-1 listening may have occurred)

The full description of each step is detailed in the pages that follow.

The strange human ritual of curling

jeff — Sat, 22 Feb 2014 12:46:04 +0000

Hilarious commentary by, of all people, David Attenborough:

Link:

Sochi 2014: BBC's David Attenborough reveals the frisky side of curling

Category:

Humor

At this moment in the world...

jeff — Sat, 22 Feb 2014 12:35:53 +0000

Get enough people together and suddenly the world is a busy place (click the image for the fully-animated display):

Link:

Frequency

Category:

Science

Complex interactions

jeff — Sat, 22 Feb 2014 12:25:58 +0000

The environment is complex. I love studying complexity, and observing ecosystems fascinates me because they are made up of many moving parts that each interact with the others in ways that are simple but in an overall way that is too complicated to predict reliably.

Several years ago, wolves were reintroduced to Yellowstone National Park. The expectation was that this would help manage the populations of deer and other large game. It did. But it also impacted the environment in a slew of unexpected ways in what's called a trophic cascade. This video explains it well:

Our earth's climate is undergoing small changes in median temperature (you know, that "climate change" or "global warming" think you're hearing about). One of the major sides in the debate is that, well, we can adapt to small changes in temperature... "so what if things warm up a little?". Here's the problem: there will be similar cascading effects, and in this case they will be on a scale we can't imagine or predict and affect every corner of the globe.

Take this year's massive cold events across North America: these are one of the major effects predicted by climate models from a decade ago. Yes, for you deniers out there, "global warming" can include significant cold spells in localized regions. But the models in this case were only designed to predict certain kinds of weather events; they were not designed (and could not be) to predict what this would do to to plant life, tourism patterns, electricity usage, or how many people would stay inside and watch an entire season of House of Cards who otherwise would have been outside hiking. And all these effects impact other systems in ways that are simple in isolation but extremely complex in aggregate. Some of those decisions feed back into the carbon emissions cycle and further compound the climate changes that caused them.

Our decisions as individuals and as a society impact our lives in ways that are too complicated to predict reliably. This has always been the case. Our challenge is to recognize when some aspect of our lives is causing changes that are significant, and then changing our ways to protect our future selves.

So what if the temperature goes up a few degrees?

So what, indeed.

Link:

How Wolves Changed an Entire Ecosystem

How Wolves Change Rivers

Category:

Science

Tron Dance

jeff — Sun, 27 Oct 2013 10:35:41 +0000

Sometimes something really different comes along. This is one of those times:

Link:

Amazing Tron Dance performed by Wrecking Orchestra

Category:

General

Fixing Homebrew on OSX Mavericks

jeff — Sat, 26 Oct 2013 02:26:32 +0000

If you're using OSX and Homebrew and are about to (or already have) upgrade to Mavericks, know that your packages won't work after the upgrade. Here's what you need to do to fix things:

$ xcode-select --install
$ brew update
$ brew doctor

If brew doctor returns any errors, you should fix them.

$ brew upgrade

At this point, everything mostly seems to be working. I did have to manually re-install Maven:

$ brew install mvn

... I'm not sure why.

Category:

Development

Coding conventions gone haywire

jeff — Fri, 25 Oct 2013 03:28:14 +0000

I generally have coding standards that I follow for every language I write in. For the most part, they make my code more readable and make spotting errors easier. Sometimes a business has standards that don't make sense. And sometimes, they are flat out horrible. StackOverflow has a thread all about the worst coding conventions seen in the workplace.

All C/C++ arrays shall start at index 1, instead of 0. Indeed, the use of 0 as first index of an array is obsolete, and has been superseded by Visual Basic 6's insightful array index management.

... and:

a friend of mine - we'll call him CodeMonkey - got his first job out of college [many years ago] doing in-house development in COBOL. His first program was rejected as 'not complying with our standards' because it used... [shudder!] nested IF statements

the coding standards banned the use of nested IF statements

now, CodeMonkey was not shy and was certain of his abilities, so he persisted in asking everyone up the chain and down the aisle why this rule existed. Most claimed they did not know, some made up stuff about 'readability', and finally one person remembered the original reason: the first version of the COBOL compiler they used had a bug and didn't handle nested IF statements correctly.

This compiler bug, of course, had been fixed for at least a decade, but no one had challenged the standards. [baaa!]

CodeMonkey was successful in getting the standards changed - eventually!

Coding conventions are good practice, but the best advice I can give in general is to use the conventions everyone else uses for the language you are using unless you have a damn good reason not to.

Category:

Development

Using drush and XAMPP together on OSX

jeff — Mon, 21 Oct 2013 06:21:43 +0000

I am using XAMPP to host local websites I'm developing, and one of those sites is built on Drupal. Recently I ran into an interesting problem: In trying to administer a site with drush I was getting the following error:

seawolf$ drush status
PDO::__construct(): [2002] No such file or directory (trying to connect   [warning]
via unix:///var/mysql/mysql.sock) environment.inc:523

I know my database is working, but drush can't see it.

Several people have posted a solution that changes your settings.php file for your Drupal site to point to 127.0.0.1 instead of localhost, but this didn't work for me (plus, there's a subtle downside to that approach: the connection is slower because of the overhead).

A very simple solution is to just create a link from the file path that drush is looking for to the actual socket created by XAMPP:

seawolf$ sudo ln -s /Applications/XAMPP/xamppfiles/var/mysql /var/mysql
seawolf$ drush status
 PHP executable        :  /usr/bin/php 
 PHP configuration     :               
 PHP OS                :  Darwin       
 Drush version         :  6.0          
 Drush configuration   :               
 Drush alias files     :

Note: if you also have a local installation of mysql other than the one in XAMPP, this probably won't work for you (/var/mysql will already exist and can't be both a link and a working directory).

Category:

Development

Data-Intensive Text Processing with Map Reduce

jeff — Sun, 13 Oct 2013 00:53:41 +0000

Lately I have been learning Map/Reduce programming on clusters such as Hadoop. While there are many books teaching how to get a cluster up and running, the number of resources teaching advanced map/reduce algorithms are much more limited.

One excellent resource for learning these algorithms is Data-Intensive Text Processing with MapReduce (free download) by Lin and Dyer (2010). This paper presents algorithms for word co-occurrence frequencies, building an inverted index, and computing PageRank, just to name a few. All algorithms are presented as pseudo-code.

In this series of posts I will be sharing the code I have written to implement these algorithms specifically within Hadoop. Each post describes one algorithm's implementation. It is assumed that you can follow basic Java without extensive explanation.

The posts also assume you have a Hadoop installation you can compile and run against. If you are running Mac OSX, you can install Hadoop in about 5 minutes.

While this is not intended to be a beginner's introduction to programming for Hadoop (there are many great online resources for that), I do explain why I do things the way I do when they deviate from the defaults. The algorithms themselves come directly from the document described above; please use it as a reference to understand the algorithmic choices.

Code can be downloaded from the repository on Github. Once you have a local copy, simply type mvn package in the top directory. The repository is an Eclipse package and can be imported the usual way.

Let's get into the code!

Government eavesdropping hubbub

jeff — Tue, 25 Jun 2013 00:40:15 +0000

Since the NSA leaks a few weeks ago, government eavesdropping has been in the news daily. I've been talking with people about this quite a bit, but haven't posted anything to date. Today I'm summarizing a few things that I feel are important into a single post.

First, on the leaks themselves. Snowden broke the law in releasing classified information. However, he hasn't done anything that will fundamentally endanger anyone. He should be treated as a whistleblower, not as US enemy number 1.

Next, about the government programs. The question really comes down to whether these programs should be tolerated in the first place.

It's clear that "national security" is being used as a blanket of fear to ease us into greater and greater surveillance, and that the public is generally mislead on the extent of this invasion:

Thanks to the Guardian’s scoop, we now know definitively just how misleading these numbers are. You see, while the feds are required to disclose the number of orders they apply for and receive (almost always the same number, by the way), they aren’t required to say how many people are targeted in each order. So a single order issued to Verizon Business Solutions in April covered metadata for every phone call made by every customer. That’s from one order out of what will probably be about 200 reported in next year’s numbers.

We're also ramping up offensive capabilities against governments and organizations:

All of this mapping of vulnerabilities and keeping them secret for offensive use makes the Internet less secure, and these pretargeted, ready-to-unleash cyberweapons are destabilizing forces on international relationships. Rooting around other countries' networks, analyzing vulnerabilities, creating back doors, and leaving logic bombs could easily be construed as acts of war. And all it takes is one overachieving national leader for this all to tumble into actual war.

In 3 Questions About NSA Surveillance, the authors ask the following important questions:

Why were the programs secret?
What have the programs accomplished?
How much do the programs cost?

I believe all of these questions are relevant, but most important is the first one. The secrecy of these programs accomplished nothing with respect to catching terrorists; it only served to avoid the kind of oversight that would likely curtail them. If the programs are not effective, they should be stopped; if they are, their secrecy still doesn't increase their effectiveness. They should not be classified.

In Why ‘I Have Nothing to Hide’ Is the Wrong Way to Think About Surveillance, the authors point out that:

We Won’t Always Know When We Have Something To Hide: you are very likely breaking laws you don't even know about, and that could be used against you in unexpected ways in the future.
We Should Have Something To Hide: the ability to break the law has been instrumental in achieving sweeping social change (think gay marriage); groups being unable to participate in social forums in an anonymous manner prevents the betterment of society in crucial ways.

But they are just collecting metadata, right? You should still care:

Context yields insights into who we are and the implicit, hidden relationships between us. A complete set of all the calling records for an entire country is therefore a record not just of how the phone is used, but, coupled with powerful software, of our importance to each other, our interests, values, and the various roles we play.

We need to understand the programs in order to decide whether we as a people think they are moral, ethical, and allowable. And we can't do that if their very existence is secretly hidden under the blanket claim of "national security". We need public discourse, not secret courts giving secret organizations permission to perform secret acts against secret targets.

Link:

3 Questions About NSA Surveillance

Why ‘I Have Nothing to Hide’ Is the Wrong Way to Think About Surveillance

Also Revealed by Verizon Leak: How the NSA and FBI Lie With Numbers

US Offensive Cyberwar Policy

Phew, NSA Is Just Collecting Metadata. (You Should Still Worry)

Category:

Politics

Security

I don't know why I find this so funny...

jeff — Mon, 10 Jun 2013 16:25:56 +0000

... I really don't know. But I do find it funny:

Link:

Babbling cow

Category:

Humor

Attention to Amazonian details

jeff — Mon, 03 Jun 2013 03:28:32 +0000

I look at details... it's just something I naturally do. So I caught this when I was looking at some music on Amazon the other day:

Apparently, I can get the CD plus the digital album for less than the cost of the digital album alone.

Category:

Humor

The flipped classroom

jeff — Sun, 24 Mar 2013 23:16:28 +0000

Our entire educational system is in need of a paradigm shift. I happen to prefer the "Flipped Classroom" approach. I believe knowledge is abundant and students should be gleaning it from sources other than the classroom, and that the role of school is twofold:

teach students how to learn
teach students how to search

If a student can find answers to their questions and adaptively grow as new information, approaches, systems, and data becomes available, they can accomplish just about anything they want. If they can't, they can't.

The flipped classroom embraces this by making the teacher a mentor and coach rather than a source of knowledge. Teachers help you build your skill set to do homework or complete projects.

The knowledge comes from online sources, which is better for a variety of reasons. Most simply, the best teacher for any particular topic is rarely the person in front of you. Imagine learning each lesson from someone who is an expert in that field and also a skilled presenter! MIT could put out a series on programming (oh, wait, they already have!), another university could put out a series on psychology, another on philosophy, another on biology, another on mathematics...

Many schools have already started doing this, so teachers have a plethora of resources to teach the knowledge. Now the teacher can focus on the individual students as he or she notices that they are getting stuck on a particular facet of the lecture or one kind of problem.

I sincerely hope that we start seeing this kind of change soon.

Link:

The Flipped Classroom Infographic

Category:

How to write a love song

jeff — Wed, 20 Feb 2013 05:09:37 +0000

The Axis of Awesome teaches us how to write a love song:

Link:

The Axis of Awesome: How to Write a Love Song

Category:

Humor