Ubuntu: A Love/Hate Relationship

Reincarnated Blog

2011-08-25T10:43:00.000-04:00

It's been a long time since I posted. About four months to be exact. I have a series of excuses, but they are mostly irrelevant so I won't really go into them here.

Instead I'll announce I'm starting a new blog. In an attempt to become less reliant on Google for services I've decided to move the blog onto a server where I have control over things: robbritton.com. The blog is not likely to have the same content as this one, there will be a lot less Linux/programming/statistics posts and more on just random stuff that I like. If you feel like hearing about those sorts of things feel free to start following that one (which I hope to start updating on a semi-regular basis).

ProjectDrinks, Take 2

2011-04-23T16:31:00.000-04:00

Just a reminder to anybody who is interested, this Monday is the second ProjectDrinks meetup. It will be at 6:30pm at Trois-Brasseurs, at the corner of St. Catherine and Crescent, downtown Montreal.

Last time people had some trouble finding the table, so it will be the one with a laptop on it.

I was contacted last time about Notman House, which is supposedly a meeting place for web developers and tech entrepreneurs. It looks kinda neat, although the idea of the house seems to be very startup-oriented, which isn't quite in the same direction that I'd like to see ProjectDrinks go. I'll check it out at some point once I'm done exams and let people know how it goes.

Bad Statistics: My Mistake

2011-04-21T16:07:00.001-04:00

I was watching this video today and one comment the speaker made was that men's incomes have not increased but women's incomes have. I thought this was interesting and decided to look into it.

However while I was thinking about it I realized that in my last wage analysis I was using 1986 vs. 2006, which wasn't a great idea.

Here's why. What's the difference between this graph:

And this one:

Although these graphs look nothing alike, when you only have two data points you can't actually tell which of the two graphs created this.

That's the issue with my last analysis. We have these weird things in the economy called "business cycles" which you may have heard of, where the economy is doing "well" or "poorly". Unfortunately these cycles could screw up the analysis that I did before since it could be possible that wages haven't gone up at all in a general sense, it could have just been the case that in different economic states in 1986 and 2006 rather than that wages have actually gone up. In 2006 for example, the Canadian economy was doing very well due to high commodity prices, where 1986 wasn't a very big boom period.

This is an example of a very basic statistical problem: model misspecification. In many cases when you're doing statistics you have some sort of model that you are trying to fit the data to. If the model you propose is correct then cool, but if it is incorrect you might not know. Many statistical procedures such as least-squares regression will still work when you give a bad model, but will give meaningless results. Unfortunately it might be the case that the results look reasonable, but are still completely incorrect - this is the most dangerous case, since without further analysis there is no real indication that your results are wrong. Fortunately there are tests like the RESET test that can test for this kind of thing, however you have to know about them in order to actually use them (obviously). They also aren't foolproof - on one hand they will tell you that you do have misspecification, on the other hand they will tell you that you might not have it. As with most things in statistics, you don't get a crisp yes/no answer.

The moral here is that even if you have a representative sample and you have good intentions (ie. not screwing with the results so things look the way you want them to) you can still get bad results by applying the wrong procedures.

Back to the original question: how have wages changed over the years with respect to men and women? Keeping in mind all the stuff that I just said, here are the results:

Median income (2010 dollars)
	1986	2006
Men	37472 (106)	38442 (182)
Women	19331 (72)	25628 (87)

The numbers in brackets are the standard errors, this is how statistics are reported in real papers. If you ever read a paper that reports statistics with comparisons, you should see if they show these numbers. If not, you can't really be too sure about the comparisons - a good example is a survey on game piracy that I critiqued a while back.

So women's wages have gone up a fair bit, while men's wages have not - consistent with the claim in the video. Whether this is a general trend or some blip due to the time periods chosen we can't say with the data here. I can probably find data if I dig around a bit if people are curious.

Fixing a Dead MySQL Server on Ubuntu

2011-04-18T14:36:00.003-04:00

I'm doing some freelance work for a company and unfortunately the other day their DB server crapped out. On top of that, there was some issues with the backup script, so the last backup done was quite some time ago.

Fortunately though, the hard drive was still intact. Here is a little guide to restoring your MySQL DB in the case that your DB server dies and you have no backups. This is a guide for Ubuntu server 10.04, but it shouldn't be too different with other versions of Ubuntu or other Linux distributions.

There is one requirement here: the hard drive still needs to work. If the hard drive is toast, then you're in trouble.

1) We had another computer that wasn't being used that had been purchased to be set up as a backup server. It wasn't set up yet, so I just stuck Ubuntu Server on it and plugged the old hard drive in.
2) Make sure MySQL is installed: sudo apt-get install mysql-server
3) Stop the MySQL server: sudo /etc/init.d/mysql stop
4) MySQL keeps the database files in /var/lib/mysql. Copy these from the old hard drive to the new hard drive. It might be a good idea to not copy over /var/lib/mysql/mysql, however I didn't have any problems after copying that over (yet). Try it and if the database doesn't work, copy that folder too. Also, don't copy over any .pid files.
5) Set permissions: sudo chown mysql:mysql -R /var/lib/mysql/*
6) Restart the MySQL server: sudo /etc/init.d/mysql start

Now the MySQL server should be just like it was before the system died. If you didn't copy over the /var/lib/mysql/mysql folder then you'll have to recreate any users that you had. If the whole thing just doesn't work then just copy that folder too.

Hopefully this saves some people some pain! I was particularly annoyed since I forgot step 5, so the MySQL server was able to list the other databases but kept saying that there were no tables in any of them. Not good! In the end it just turned out to be a permissions issue which is no big deal.

Anyway the more important thing to take out of all this is to make sure that your backup scripts work.

Empirical SoEn is Hard: Cost

2011-04-17T11:44:00.000-04:00

In economics, there is a small sub-branch called experimental economics, which focuses on applying the experimental method to better understand economic phenomena. It has had some interesting results in small cases - especially in the field of game theory with things like the ultimatum game - but most of the papers I've read are done at such a small or simple scale that they don't really apply to a real economy. In order to make the results more interesting, the number of people involved must be a bit higher, and the experiments need to be done with a longer timeline. Unfortunately, the cost of this is extremely prohibitive (university departments are typically strapped for cash).

I've done a lot less reading of software engineering papers, but in my head it also seems like cost would pose a very large problem on empirical software engineering research. You could do some simple experiments with small groups of developers for small projects, however in reality many projects take large groups of developers working full-time months or even years to complete. Having a researcher pay these developers to do experiments might be prohibitively expensive. On top of that to be able to make any decent claims, you'll need to do a rather large number of these experiments which just blows the cost of everything through the roof.

At this point you can't even use experiments anymore to do detailed study of software engineering. In the business world where you might have the resources to do basic experiments with large-scale projects, you typically have an aversion to risk and so businesses typically won't go too far from "proven" techniques since they may or may not increase the costs of doing business.
In economics typically we rely on observational data to do our analysis: we can collect data from the actual economy and analyze it, however we can't reach out and change whatever parameter we like to see how it will affect things. We also can't "start over" to see what happens if we change a few of the initial conditions: unless somebody develops time travel and goes back to 1950 to fiddle with unemployment rates or government policy to answer "what-if" questions. You can't impose controls the same way that you can when experimenting.

I believe that software engineers are in the same boat. Due to cost concerns, you must rely more on observing actual software engineers at work than in the laboratory. This comes with all the problems with observational data: you can't go in and change the number of developers, or swap developers out with developers of different skill levels, or any of these other things that you might be able to try in an experiment. Also in observational data it is often the case that not just one thing changes at a time: sometimes one factor will change, but another will change at the same time since they are correlated - it is difficult to distinguish the effects of one variable from the effects of another when you aren't able to hold them fixed.

ProjectDrinks

2011-03-23T16:43:00.000-04:00

Way back in September I tried to get together a group of programmers around Montreal to hang out, have drinks, and talk about various projects that we might be working on at the time. Unfortunately I only did the event once so it didn't really pick up any steam and ended up fizzling out.

I've decided to give it another go, but this time I'm giving it a name and a website: ProjectDrinks, largely inspired by StartupDrinks but without the startup aspect - these are projects and other fun little apps that may or may not have any commercial value, they're purely for enjoyment.

The meetups will be the last Monday of every month starting next Monday (March 28, 2011) at 6:30pm. The location will be Trois-Brasseurs at the corner of St. Catherine and Crescent in downtown Montreal, chosen largely because it's the first place I found that has both beer and wireless - at least according to Île sans fil.

I'll be heading out there with my laptop on Monday evening, having a few beers, and fiddle with a little project I've been messing around with. If anybody wants to come out and chat about any projects you might have on the go feel free to head on down!

Why We Have Financial Crises

2011-03-18T01:05:00.001-04:00

How economists view the world:
1) Individuals are all walking in a line
2) Individual i drops a $20 bill
3) Individual i + 1 says, "oh sweet, a $20 bill!"
3) Individual i + 2 do nothing, since such a state of dis-equilibrium normally doesn't exist

A more realistic viewpoint:
1) Repeat step 1 and 2 from before
2) Individual i + 1 says, "hmm, that guy dropped a $20 bill. Does he know something I don't know?" Individual i + 1 drops a $20 as well.
3) Individual i + 2 says, "whoa, those two guys just dropped a $20 bill. That must be a good investment strategy!" Individual i + 2 drops a $20 bill as well.
4) Repeat n times.
5) Individual i + n says, "whoa, what a bunch of suckers!" Individual i + n starts grabbing up $20 bills.
6) Individuals i + n + 1 to i + n + k start grabbing up $20 bills.
7) Individuals i + n + k + 1 start thinking, "whoa, those guys made a ton of cash picking up $20 bills. I'm going to take out a bank loan and expect that I'll grab enough $20 bills from other suckers to pay it back and make fat stacks of cash!

And so it progresses...

Controlling Rhythmbox with your Wiimote

2011-03-16T21:58:00.002-04:00

I've been fiddling with one of my Wiimotes tonight to try and get it to control Rhythmbox. Turns out it's actually really easy! You can do it via the cwiid library for Python, and with a Rhythmbox plugin.

For this to work, you need a Wiimote and some sort of Bluetooth device for your computer that works with Ubuntu. You'll also need the correct package:

sudo apt-get install python-cwiid

Here's the step-by-step guide to getting it to work:

Create a folder at ~/.gnome2/rhythmbox/plugins if it doesn't already exist.

[RB Plugin]
Loader=python
Module=WiimoteControl
IAge=1
Name=Wiimote Control
Description=A way to control Rhythmbox with your Wiimote
Authors=Rob Britton 
Copyright=Copyright (c) 2011 Rob Britton
Website=http://lovehateubuntu.blogspot.com/

Create a file called __init__.py, and put the following in it:

import rb
import cwiid

rbshell = None

def callback(mesg_list, time):
  # Called whenever a button is pushed on the Wiimote
  global rbshell
  for mesg in mesg_list:
    if mesg[0] == cwiid.MESG_BTN:
      if mesg[1] == cwiid.BTN_A:
        rbshell.props.shell_player.playpause()
      if mesg[1] == cwiid.BTN_LEFT:
        rbshell.props.shell_player.do_previous()
      if mesg[1] == cwiid.BTN_RIGHT:
        rbshell.props.shell_player.do_next()


class WiimoteControlPlugin (rb.Plugin):
  def __init__(self):
    rb.Plugin.__init__(self)

  def activate(self, shell):
    # called when Rhythmbox starts
    global rbshell, bus
    self.shell = shell
    rbshell = self.shell
    self.wiimote = None

    print "looking for wiimote..."
    self.wiimote = cwiid.Wiimote()

    print "found wiimote"
    self.wiimote.enable(cwiid.FLAG_MESG_IFC)
    self.wiimote.mesg_callback = callback
    self.wiimote.rpt_mode = cwiid.RPT_BTN

  def deactivate(self, shell):
    # called when Rhythmbox closes
    self.wiimote.close()

This code sets up the Wiimote and binds left/right to previous/next song, and the A button to Play/Pause.

Open Rhythmbox, and while it is opening push the 1 and 2 buttons on the Wiimote. It will take a sec for Rhythmbox to open since it is looking for the Wiimote.

All this should make it so that your Wiimote can connect and control Rhythmbox. To get debug messages, you can run Rhythmbox from the command line:

rhythmbox -D WiimoteControl

The program isn't ideal, since if you don't connect the Wiimote in time it will not properly set everything. It might be better to put some sort of timer that goes periodically to check to see if the Wiimote is there and if it is not, attempts to connect it. Also I think it would be neat to be able to control the volume using the up and down buttons, but I'm not quite sure how to do that yet. It doesn't seem to be in the Rhythmbox plugin guide, and the other plugins don't really do that sort of thing.

Impressed

2011-03-15T11:00:00.000-04:00

This canvas test runs at the same speed on my machine with IE9 and with Firefox 3.6. The difference? The IE9 one was running in VirtualBox.

I always thought it would be a strange day when I was congratulating Internet Explorer, but today I gotta say, not bad! Maybe I should try looking at various other canvas examples to see if this trend is not just confined to this one example.

If2

2011-03-10T00:43:00.000-05:00

I was browsing on Rosetta Code this evening in an attempt to find some unfinished tasks to fiddle with, when I found a rather interesting example. The task is to create a new "if2" statement that takes two conditional statements rather than one. In addition to the regular "if" block, it takes three "else" statements that are executed depending on which of the conditions are true. I've actually had this situation come up when I've been programming and at the time it never occurred to me that I could extend Ruby to allow for this scenario.

It turns out you can! It takes a little bit of combinator and anonymous class hackery but it does the job:

class HopelesslyEgocentric
  def method_missing what, *args; self; end;
end
 
def if2 cond1, cond2
  if cond1 and cond2
    yield
    HopelesslyEgocentric.new
  elsif cond1
    Class.new(HopelesslyEgocentric) do
      def else1; yield; HopelesslyEgocentric.new; end
    end.new
  elsif cond2
    Class.new(HopelesslyEgocentric) do
      def else2; yield; HopelesslyEgocentric.new; end
    end.new
  else
    Class.new(HopelesslyEgocentric) do
      def neither; yield; end
    end.new
  end
end

Then you can go ahead and use it like this:

if2(5 < x, x < 7) do
  puts "both true"
end.else1 do
  puts "first is true"
end.else2 do
  puts "second is true"
end.neither do
  puts "neither is true"
end

I thought that was pretty nifty. I doubt performance is amazing, but it is fairly easy to use!

3D Turtles and L-Systems

2011-03-09T20:34:00.001-05:00

A while back I posted about these things called L-Systems, which are a way of programmatically generating images based on some fairly simple rules.

As it turns out, you can do some pretty nifty things. The initial images that I put up in my last post used a turtle-graphics type of rendering based on the output of the L-System after a certain number of iterations. I'm doing the same thing here, except it is now a 3D turtle with a number of other things it can do like change colour, change the width of the line, draw polygons, etc.

Here's a few examples. A tree (this one is generated completely deterministically, which is kinda cool):

Here's a nice little flower bush. This one uses stochastic rules for colouring the flowers:

These programs are all in 3D, so if you download the actual code you can circle around the plant, zoom in, etc. Unfortunately I was a bit lazy with the camera system so it is a bit annoying sometimes, but it works well enough.

You can check it out yourself here at the Github repo. I'll be putting up a little guide in the Wiki shortly, so check back there if you want to know how the "language" for the system works.

Testing Real-Time Software

2011-03-06T20:45:00.000-05:00

I've come to an interesting problem at work and I've decided to ask you all for your opinion on the matter. Since it's likely that many of you are better software testers than I am, maybe some of you will have some advice.

My issue is that the software I'm writing has to take timing into consideration when doing its calculations: it's a program that analyzes stock behaviour. For example, if I call foo() twice with 2 seconds in between, the result might be different than if I call foo() twice with 4 seconds in between, even if all the inputs are exactly the same. What I want to do is make an automated testing system to verify that the code is doing what it is supposed to be doing.

The first step is to pull out any timing related code into a separate module, so that when running an automated test the script that is supposed to do something over the course of 30 minutes doesn't actually have to wait 30 minutes for the test to pass. Instead, there is a test module for timing that will tell the program that 30 minutes has passed, even if it's only been a few milliseconds.
This also means that the program can't use the traditional timing methods available (since the code is in .NET, that means no System.Timers.Timer or DateTime.Now). This makes things a little bit tricky.

My first thought to try and test this sort of stuff was to use a script like this:

call foo()
simulate 5 second pause
call foo()
assert that result is as expected

The main issue with this is that while the test process does simulate time passing, it does not send the program data during that time. An example of something that wouldn't be easily testable in this case is if I have a program that says, "if the data received over the last minute follows pattern X, do Y." In this example the program is active during the entire minute, it is just watching and waiting for something interesting to happen.

My second thought is what I am thinking of implementing now: I basically have a CSV file for the data. The first column is the time passed since the last amount of data is received, in milliseconds. The other are just arbitrary data, with the first row being a name given to the data. For example I'll have something like this:

Time  Bid BMO  Ask BMO
0     61.73    61.75
500   61.73    61.76
500   61.74    61.77

Then to trigger a test, I have something called a checkpoint. When a checkpoint is hit, it is telling the system to trigger a series of tests. Those tests are run, the results will be reported, and any errors/exceptions thrown will halt this particular test. After that the data will continue until the end of the file is hit.

To me it seems like the testing system is much more data-driven than a normal testing suite, so I'm not entirely certain I'm doing it the right way. However at the same time the nature of the data is not quite the same as with other software, so maybe this sort of approach is a good one. What do you guys think?

Empirical SoEn is Hard: Unobservability

2011-03-02T00:27:00.000-05:00

One thing that I find interesting is attempting to apply some of the methods of econometrics and labour economics to measure productivity within software development. At the moment it seems like many of us (myself included) base our opinions of "what works" from our own perspectives and from anecdotal examples (which I've said before doesn't really count as evidence for something) - although this here is an anecdote which may or may not be true.
It would be nice if we could come up with some good analysis techniques to give real support to our claims of what works and what doesn't, and better yet when something works and when it doesn't.

It turns out though that tackling the problem of measuring productivity within software development is quite hard. There are all sorts of problems that can arise here that make analyzing data and testing hypotheses rather difficult. This article will be the first in a series talking about some of the problems that I've though of, and I'd be more than happy to hear about what you guys might think about these issues.

The first problem is that of unobservability (also known as latent variables), which is where you are unable to measure a certain variable because you can't really see the variable directly the same way we can with observed variables. An example of this is ability: it is common knowledge (I think) that there is varying degrees of ability when it comes to developing software. But can we give somebody a stamp saying, "this person has productivity X?" Sure, we can use some indirect measures like lines of code produced per hour or bugs closed per day or some junk like that, but these are simply proxy measures that are the result of ability, not ability itself. Compare this with observed variables like years of experience, or language/methodology used, or team size: we can directly give a value for these variables with some specified unit, so therefore they are observed.

These types of variables are problematic because it is difficult to hold them fixed. Since we can't observe them, we often end up with some omitted variable bias as these variables are often correlated with our other observable variables. This happens when we see an increase in productivity and think it is due to one of our observed variables, when it is actually caused by an unobserved variable that is also influencing an observed variable.

I've thought of a few unobserved variables in software development, feel free to add any more that you think of:
1) Ability (the obvious one). I've already talked about this one to death, so I won't go into much more detail here.
2) Team Chemistry. You can throw a bunch of people into a room together, who individually are extremely good at what they do, but doesn't mean they'll get a lot done - if they all sit there bickering over testing frameworks or variable names or other things like that, they're not going to get a lot done. Likewise you can put a group of people together who may not be super geniuses, but if they work well together you still end up getting some good results. This is an important factor in a team's productivity, and you can't really stamp a number on it.
3) Productivity itself. All this time I've been talking about measuring the effects of various factors on productivity, but we don't actually have a way to directly measure productivity. You can see the indirect effects of the productivity: milestones get reached sooner, bugs get fixed faster, less bugs get introduced in the first place, etc. etc. But we don't have a measure to say something like, "the combination of development methodology X with N programmers of E experience and ... gives us P units of productivity."

I thought about putting the complexity of the project in, however I'm not sure if that affects actual productivity. It can affect some metrics used to measure productivity, but if you view productivity as how much someone is getting done per unit time, then I don't think complexity makes a difference in the same way that say for example, having two monitors does. Maybe I'm also on crack and need to go to bed soon, so this point is up for debate.

All these unobserved factors in software development make things rather difficult to do real quantitative analysis. One possible solution is using experiments to analyze various factors, however those come with their own set of issues that I will discuss in future posts.

Values of Different Degrees

2011-02-20T19:11:00.005-05:00

As I said in my last post, I would put up some details about how much various types of degrees were worth, according to Canadian census data - as I said in the last post, this type of data should be fairly similar across countries that have similar economies as Canada, like the US, UK, or Australia. I'll make my code available so that if you want to test against your own country, you shouldn't have too much trouble.

Let's take a look at what we have. Here is a ranking of the types of degrees based on a 95% confidence interval of median 2006 incomes (all the values are inflation-adjusted to 2010 Canadian dollars):

1) Engineering: $54.4k - $58.4k
2) Commerce: $51.1k - $55.2k
3) Sciences: $49.8k - $54.3k
4) Education: $47.4k - $48.6k
Median income for university graduates: $45.3k - $47.5k
5) Social Sciences: $43.2k - $46.2k
6) Health/Food Sciences: $38.5k - $42.3k
7) Humanities: $35.0k - $37.4k
8) Fine Arts: $24.5k - $28.4k
Median income for non-university gradates: $25.5k - $25.8k

So it looks like engineers are on top, although their confidence interval has some overlap with commerce graduates which means that if we want to be 95% sure we are right, we can't say that engineers make more than commerce grads. If we reduce our confidence in our results, the data says that once you get down to about 80% confidence you can say that engineers make more than commerce grads. In other words, given our data here, there's a roughly 80% chance that engineers make more.

In the last post I also looked at the data from 1986. Let's see how the rankings change over time:

1) Engineering: $67.4k - $70.8k
2) Sciences: $53.1k - $56.4
3) Commerce: $51.7k - $54.7k
4) Education: $49.0k - $51.0k
Median income for university graduates: $47.9k - $48.9k
5) Social Sciences: $42.7k - $44.8k
6) Humanities: $38.6k - $40.7k
7) Health/Food Sciences: $35.3k - $38.7k
8) Fine Arts: $23.9k - $28.6k
Median income for non-university graduates: $23.0k - $23.2k

The most striking difference between these results and the ones from 2006 is the drop in wages for engineers. They used to be, by far, the most well-paid university degrees, whereas now they are only slightly higher than commerce degrees (which haven't changed all that much). This is somewhat disappointing for up-and-coming engineers (well, necessarily)!

The rest of the degrees haven't changed too much. A couple of them have statistically significant changes (such as education), but for the rest we can't really distinguish any change here from statistical noise - at least not at a 95% confidence level, if we accept a higher probability of being wrong then we can say things have changed.

This analysis here focuses on the median. Why do I do that? Why not use the average/mean? This is because incomes are something that are very skewed, meaning that that the average will be influenced a lot by the outliers that make tons of cash, making it look like a certain degree may be worth a lot when in fact most of the people with that degree are making a lot less than the average. The difference is most pronounced with commerce degrees: in 2006 the average income for a commerce degree holder is around $78k, which is the highest average for all the degree types. However the median is roughly $53k, showing that there is a massive amount of skewness in the distribution of commerce grads: those MBAs that are making big 6-figure salaries are pulling up the average. To get a feel for how most people with a certain degree are doing it is better to look at the median, which is much less likely to be affected by huge outliers than the mean is.

Again I'll repeat that these are correlations, not causations. In fact since the variance of the incomes for all of the degree types have increased dramatically, that could indicate that your degree is less important in determining your salary today than it was 25 years ago. In order to isolate the actual effect of the degree on wage you'd have to have a more sophisticated model that takes into account all sorts of other relevant factors like age, experience, ability, etc.

As always, the code is available here and the results of running the code are here.

Wages Over The Years

2011-02-17T00:29:00.001-05:00

I recently read this article which makes a claim that real average wages (aka inflation-adjusted wages) have fallen over the years and supposedly "continue to fall." I looked at this and thought the idea was preposterous, so I rolled up my sleeves and did some real data analysis. Since I've been talking about statistics, I figured some of you might be interested in this knowledge.

tl;dr: Wages overall have actually increased, but not for university grads. University grads still make more than non-university grads. Income inequality has increased a lot, especially among university grads.

First off, my sources. I grabbed the data from the 1986 and 2006 Canadian Censuses (Censi?). Unfortunately I can't share this data with you since I am not legally allowed to distribute it, but if you have access to a university you should be able to dig it up somehow. This data is probably some of the best you can get, since it is much less likely to have selection biases compared to other surveys - people are legally required to fill this data out.
Second source was the Consumer Price Index (CPI) which can be used as a measure of inflation. That's how I adjust the raw figures in the census data for inflation. Statistics Canada is kind enough to list these figures here. All dollar values in this post will be inflation-adjusted to 2010 Canadian dollars.

Now I know that many readers here are not Canadian, however these results should be similar for countries with a similar economy to Canada like the US, the UK, or Australia. For those of you so inclined you can probably find the same data for your respective country to do the same analysis.

Let's get started. How would you go about figuring this stuff out? Well, you need the data. Once you get that, it's pretty straight-forward. I did my analysis using the following steps:
1) Filter the data. I want to look at people who are at least 15 years old, and have a regular old job. This excludes self-employment. This isn't a huge deal, just keep in mind that the averages here are for employed people.
2) Adjust wages for inflation. This is done by dividing by the CPI for the year of the data (1986 or 2006) and multiplying by the CPI for 2010.
3) Construct a confidence interval for the average wage. This lets us see if the wages between the two periods are actually statistically different. The formula for a 95% confidence interval in R-pseudocode is:

mean(wages) ± 1.96 * sd(wages) / sqrt(length(wages))

The 1.96 is the critical value of the normal distribution (sample averages follow a normal distribution) for a 95% confidence interval.

What are the results? Here's the R output:

[1] "Average wages for all employed individuals:"
[1] "Confidence for 1986: 30221.611097 to 30436.550551"
[1] "Confidence for 2006: 38479.736828 to 38790.160974"
[1] "Standard Deviation for 1986: 27382.937843"
[1] "Standard Deviation for 2006: 52381.845010"
[1] "Median Confidence for 1986: 25083.328237 to 25325.457532"
[1] "Median Confidence for 2006: 29704.694341 to 30054.387142"

What are we looking at here? Well, the average wage in 1986 was roughly $30k/year, where the average wage in 2006 was roughly $38.5k/year. Looks like wages in general are not falling.
The median is a better measure here though, since medians are a bit more robust to outliers (aka those few people who make hundreds of thousands of dollars a year). As we can see, the shift in the median is not quite as big as the shift in the mean, meaning that while the wages have gone up, they haven't gone up quite as much as the mean might indicate.
What does all this tell us? Well it is difficult to say for certain, but it would appear that in 2006 people are in fact making more money (as shown by the higher median), but there are also more people making giant salaries than before which will skew the average. Given the big jump in the standard deviation, we can see that there is an increase in income inequality.

Now, here's the real interesting part. I decided to run this again, but with one more twist: I filtered for people who have a bachelor's degree or higher. Let's see the results:

[1] "Average wages for bachelor's degree or higher:"
[1] "Confidence for 1986: 49543.651801 to 50383.047856"
[1] "Confidence for 2006: 59994.274103 to 61070.608811"
[1] "Standard Deviation for 1986: 37181.295254"
[1] "Standard Deviation for 2006: 82837.561885"
[1] "Median Confidence for 1986: 47938.828738 to 48884.408393"
[1] "Median Confidence for 2006: 46310.094270 to 47522.585319"

The average and the median wages here are much higher in both periods than for the entire group. This gives us a pretty good indication that university graduates make more money than non-university graduates.
However, it would appear that while the average university graduate makes far more in 2006 than they did in 1986, this result is misleading. When we use the median we can see that university salaries have actually gone down slightly over this time period. One conclusion that we can guess from this data here is that there are some university graduates in 2006 that are making huge salaries, while the bulk of the university grads aren't doing quite as well.

So there we have it, some statistical data. While the interpretations of the data and the methods for analysis here are up for debate, the numbers calculated are not. They're taken directly from census data, which is pretty darn good data (unfortunately it might not be quite as good for 2011, since the Conservatives have scrapped the long-form census and I'm not sure if this data will be on the short-form one).

Keep in mind that these are purely correlations, not causations. This data is not saying, "if you get a university degree then you will make more money." This data is saying, "people who currently have university degrees make more money on average."
It is also looking at the aggregate. I'm sure most of you can come up with examples of non-university graduates who are making good salaries, and of university graduates who are not making great salaries. These people are the exceptions, not the rule.

One thing that I could do in a later analysis is split the groups up into the different types of degrees. Both these censuses provide the level of education the people get (Bachelor's, Master's, etc.) and the discipline (sciences, engineering, arts, etc.). However this post is getting long enough, so that can wait for another day.

For the code, it is written using R and is available here as a gist on Github. Feel free to fiddle with it if you feel like it. You'll notice I generate some histograms of the data, however I found that they don't really reveal much info so I didn't include them here.

Oops!

2011-02-09T13:03:00.000-05:00

In case you read the post earlier today, just ignore it. I got lazy and didn't do enough research beforehand, which was pointed out. My apologies!

Rug

2011-01-08T01:52:00.001-05:00

Over the last few months I've been working with Concordia's CART CGD group, which is a group of computer art students who want to make video games. We did a few things called "game jams", which is where a group of people meet together and try and churn out a game in a few hours. We didn't actually come up with anything that great this last semester, but then again there was only two game jams (if you're interested in participating, they have them every Saturday starting January 15th, send me an email and I can give you more details).

We did the game jams with Lua and a game framework called Löve2D, which helps simplify game development.

I found after learning a bit of Lua that it is a lot like Javascript (although I have to say I still prefer Javascript). It isn't an object-oriented language in the same way that C++ or Ruby are, but it is still possible to create objects with methods and inheritance and so on. I found in the end that I need a little bit more structure when I'm programming than is offered by Lua - I actually also found the same thing with Javascript when starting to build some larger Javascript apps like Colonial.

However, I found that I really liked Löve2D. A lot of things were very simple and straight-forward. Before that the nicest gaming library I worked with was pygame which is a wrapper around the C version of SDL and a few extra bells and whistles like collision detection and sprite handling. There's a port to Ruby called RubyGame, but I find it suffers from the same problem as pygame: it's just a wrapper around SDL with a few bells and whistles.

I decided to go and roll out my own Ruby gaming library called Rug. It's mostly written in C++ for speed, but the API is much more Ruby-ish than Rubygame or Pygame with a few hints from Löve2D. At the moment it doesn't have a lot of features; it supports some rudimentary collision detection and animations. It doesn't stick completely to Löve2D's API, there were a few things in there that I found kinda clunky when it came to animations that I fixed up.

You can see an example of Pong here with a screenshot:

Or a very simple platformer here, with a screenshot:

The library is still very new, and there are a few things that I'm not happy with that will change in the near future (in other words if you choose to fiddle with this, don't be surprised if things change - but then again if you've had any experience with Rails you will be used to this sort of thing ;) ). Documentation still has a ways to go, I started documenting the features religiously but in the end I found that I changed the way I wanted the library to work often enough that documenting can wait until things are a little bit solid. For example the physics module for collision detection has undergone quite a few API changes, and after having to update the documentation several times I decided that I will wait until I am happy with how the module works before telling everyone else how to use it.

So anyway, feel free to play around with it and tell me what you think. Installing it is pretty easy in Linux, however in Windows it can be a bit annoying with all the Ruby headers and the various SDL libraries. I managed to get it to compile in Windows, so shortly I'll put up a zip with all the various DLLs that you need to run Rug.
I have no idea how easy/hard it will be to set up on a Mac, if it works anything like Linux you can probably just use MacPorts to install SDL and the Ruby development headers.

Your 30 Before 30

2010-12-23T11:03:00.000-05:00

One thing that I see a lot of these days is all sorts of things like this blog or this book telling you to do your own thing, do your own projects, branch out on your own, take initiative, etc. Whenever I read these things, I can't help but find myself inspired by them: I enjoy doing my own projects, and I would love to be able to just work on them all day and maybe someday be able to live off them.

So here is my first try. I'm launching a project (it's too small for me to consider it a startup) called Your 30 Before 30 that might be a little fun to do. The idea behind it is to create a "30 before 30" list (similar to a bucket list), where you write down 30 things you want to do before you're 30. They don't have to be anything special, but they can be. Then at some point when you accomplish what you want to accomplish, you mark it off and tell people what you did.

I'm putting it here to invite any of my readers to check it out and "beta" test it. Let me know what you think. What's cool, what's not? Do you have any suggestions/constructive criticisms? Any feedback is welcome.

Second Thoughts on IronRuby

2010-12-07T15:42:00.001-05:00

Unfortunately after using IronRuby at work a bit, I've found that there are still a few too many bugs with the thing for me to want to depend on it for production software. The bugs are already reported, however they are tagged as low priority and will not likely be fixed any time soon. Rather than spending lots of time fixing the bugs myself (I get paid to make profit, not to work on open-source pet projects), I'll just continue doing things the way that I have been!

Hopefully someone else will find IronRuby useful!

In case you're wondering, the bug that really killed it for me was the inability to use gems with RVM under Linux. The following command fails with issues installing RubyGems:

rvm install ironruby

No Rubygems drastically reduces the usefulness of Ruby, enough that I'm not really wanting to use it anymore.

Getting Your Files

2010-12-03T15:41:00.000-05:00

I suppose I'm talking to the wrong group of people here, but I decided to write a quick post about how to access anybody's files without needing their username/password for their computer.

It's quite simple:
1) Put in an Ubuntu LiveCD/LiveUSB
2) Boot off it
3) Access their files by going to Places -> X GB/TB Volume

This doesn't work if they have any sort of encryption on their drives or a BIOS password, but otherwise you can do whatever you feel like! So if you have a laptop or a machine that is easily reachable by random people, then you should probably secure your files from something like this if there are things on there that you don't really want people getting to.

Embedding a Ruby REPL in a .NET App

2010-12-02T14:41:00.001-05:00

I've finally gotten fed up with using VB.NET as a scripting language at work, so I decided to try dropping an IronRuby interpreter into the system. It's fairly simple to do, I'll describe here how to build a REPL within your app.

Step 1: Make a window for the REPL. Give it a box for your input and a box for your output. How you do all this is up to you, the interesting part is how to use Ruby within it.

Step 2: Initialize the scripting engine. You need a config file, I call it ironruby.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
 <configSections>
   <section name="microsoft.scripting" type="Microsoft.Scripting.Hosting.Configuration.Section, Microsoft.Scripting,
     Version=1.0.0.5000, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="false" />
 </configSections>

 <microsoft.scripting>
   <languages>
     <language names="IronRuby;Ruby;rb" extensions=".rb" displayName="IronRuby 1.0" type="IronRuby.Runtime.RubyContext, IronRuby,
       Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
   </languages>
 </microsoft.scripting>
</configuration>

You need to make sure you have the DLLs available that IronRuby comes with. Just put them in the same folder as your app.

Now the code to load in your Ruby stuff. Initialize a few objects:

var runtime = new ScriptRuntime(ScriptRuntimeSetup.ReadConfiguration("ironruby.config"));
var engine = runtime.GetEngine("Ruby");

// create a scope - you need to do this in order to preserve variables declared across executions
var globalScope = engine.CreateScope();

// I'm assuming you have a textbox or something named command, just adjust
// this to what you want if you have something else
var result = engine.CreateScriptSourceFromString("(" + command.Text + ").inspect").Execute(globalScope);

And, jackpot! The result variable is a string that is the result of whatever is in the command.Text variable.

This code could be a little bit more robust, so make sure to handle exceptions nicely. An exception will be thrown if say the syntax is invalid, or there are other problems.

The nice thing about this is that you can tinker with the classes within your own application. In the REPL if you put:

# replace MyApp.exe with the .exe or .dll file that your app uses
require "MyApp.exe"

You can then access your namespaces and classes within your own app, and fiddle with them while the app is running!

Stack Overflow

2010-11-22T18:50:00.002-05:00

Now is the perfect time to stop using Stack Overflow unless my reputation score changes:

Software Metrics and Instrumental Variables

2010-11-17T18:31:00.000-05:00

I recently read this article about how software metrics are mostly useless and tend to cause more problems than they solve. This reminded me of a topic in stats which apparently has some application in software development.

The use of software metrics is an example of a statistical technique called instrumental variables. Often when you want to understand some phenomenon or relationship, you run into problems because many factors are unobservable. This means that they are not concrete things that you can stamp a number on to get a clear measurement of the factor. One example that constantly crops up in economics and software development is ability. A person's ability can have extremely strong effects on other factors such as productivity, wage, etc. However, you can't really come up with a solid measurement for a person's ability: suppose some programmer you know has an ability of 10. What's that mean?
Compare that to a metric like lines of code per hour. A measurement of 10 has a very clear and concrete meaning: given the changes that they made in the hour, the code they have produced contains 10 newlines.

This is where instrumental variables come in. An instrument is an observable variable that you use in place of the unobservable variable that satisfies two characteristics: it has to be correlated with the variable that it is standing in for, and it can't be correlated with random errors or other omitted factors. The power of the instrument is based on the strength of the correlation between the instrument and the unobserved variable. This is why people use things like lines of code per hour, years of experience, etc. etc. for attempting to measure the productivity of a programmer.

Unfortunately there are a number of shortcomings with the instrumental variables method. The biggest issue is finding a good instrument. We know the criteria required for a good instrumental variable (there's all sorts of math proofs that you can look up if you like), however that doesn't mean that there are any instruments that satisfy it. On top of that when dealing with people who know the metric you're using, they can perhaps attempt to cater to the metric - thus introducing a correlation between the instrument and other omitted variables like their ability to cater to metrics.

In short, the problem is not entirely with the method, but more with finding good instruments. Unfortunately if you can't find a good instrument, you'll have to resort to a different method. According to a stats professor of mine, one method for dealing with unobservable factors is to use what's called a mixture model. It supposedly works, however the procedure appears to be much more complicated and can be less precise than having a good instrument. I'm still working on figuring out how to do this sort of thing, perhaps I'll talk a bit more about it another day.

Parse vs. TryParse

2010-11-16T11:45:00.000-05:00

I was having a problem not too long ago where the performance of a specific bit of code was abysmally slow. I scanned through it to understand theoretically why it might be going slow, and then ran it through a profiler to see where it was spending most of its time. The results were very strange: most of the time was spent parsing strings into integers and decimals (for those who have never done .NET, they have a decimal type in addition to floats and doubles which are better for processing financial data - with decimals, 0.1 + 0.7 actually does equal 0.8). This was very odd, so I figured I'd look at it in more detail.

It turns out that in .NET, the regular Parse() functions throw an exception when the string is not well-formed. If the string is "N/A", then an exception is thrown. This is a common occurrence in the data I was receiving, so I was just catching the exception and setting things to null right away. The problem was that this code was being executed hundreds of times a second, the overhead from exception throwing was adding up!

In high-performance situations, you're better off using a function like TryParse(), which accepts the int/decimal variable as a reference and returns true if the parse was successful. This does not throw an exception and thus is much faster - a massive performance increase was noticed!

While my example here is with .NET, it applies to pretty much any language where a string to int/float method throws an exception when the string is malformed. If your language has something like TryParse, it is definitely recommended over Parse for high-performance situations!

Or an even better moral: don't throw exceptions in code that needs to be fast!

Why I Prefer Linux Servers

2010-11-11T09:49:00.000-05:00

Say what you will about the quality of Linux servers vs. Windows servers, etc. etc. but here is one huge reason why Linux servers are better (all prices in USD):

Windows Server 2008 Licence with 5 users (not accessible through Remote Desktop): $1 029
To enable 5 users with Remote Desktop: $749
To add 5 more users: $199
To allow those 5 more to use Remote Desktop: $749

Ouch.