This Week in Hax

Easy solution to the melting ice caps

2013-02-13T15:56:00.000-05:00

People are always belly-aching about how the polar ice caps are melting. "Oh, the sea level is going to rise and flood coastal cities!" Blah blah blah, as if this is some unsolvable problem.

But, as we all know, human ingenuity knows no bounds, and there is a simple, cost-effective solution to the "problem" of arctic ice melting: just re-freeze it. A quick back-of-the-envelope calculation reveals how simple this solution would be.

According to the Polar Science Center at the University of Washington, the arctic pole is losing an extra 280 km³ of ice per year beyond the seasonal fluctuations.
Ice has a density of 0.9167 g/cm³, so 280 km³ (2.8e17 cm³) is 2.56676e17 grams of ice lost per year.
1 Watt can freeze 8 grams of water per hour, according to the ever-trustworthy beowulff on the Straight Dope message boards.
The state-of-the-art nuclear reactor design is the European Pressurized Reactor, which, for the low-low price of $11 billion, pumps out a hefty 1,600 MW
A single EPR could freeze (assuming 100% efficiency) 1.28e10 grams of water per hour, or 1.12e14 grams per year.
It would therefore take 2.56676e17 / 1.12e14 = 2,291 EPRs to re-freeze the yearly arctic ice loss. At $11 billion a pop, that's $25 trillion to build enough power plants to stop the melting of the polar ice caps

So all this worrying about melting ice caps just boils down to a lack of creativity. For a mere $25 trillion, we could build 2,291 nuclear reactors in the arctic circle and use them to run giant freezers. It's so obvious, I can't believe nobody's thought of it before.

You're welcome, world.

Using a Wiimote in Mupen64plus

2012-12-17T15:20:00.000-05:00

After much head-wall-banging, I got a Wiimote to pair successfully with my computer and work as a controller in Mupen64plus! There was a lot of trial and error, so I've created a guide to getting a working setup so future generations don't have to go through what I did. The details are in this git repo, which includes detailed instructions and config files.

Trying an Android ROM

2012-12-12T11:25:00.000-05:00

Against my better judgment, I am going to take another shot at installing a ROM on my phone. The battery life could use some improvement, and people have claimed that some of the 3rd-party ROMs can help with that. I know I have promised myself not to play with ROMs on my phone again, since the result is usually a big waste of time with nothing to show for it, but this time is different.

Brace yourselves for a follow-up in which I immediately regret this decision.

Another Ubuntu update, another b0rked system

2012-12-11T21:49:00.000-05:00

*sigh*

I updated to 12.10, and Alt+middle-click to resize windows was broken. After about an hour of circular Googling, I found the correct incantation to sprinkle. For some strange reason, the setting org.gnome.desktop.wm.preferences.resize-with-right-button was set to true. Set it back to false and the system returns to its former functioning state.

Better than "Understanding OOP"

2012-11-18T00:50:00.001-05:00

I came across a page talking about the joys of OOP via this blog post. The original page was a thing of horror. I modified the Elisp code from Irreal, adding a few additional features:

In addition to equality matches, it can match using regexps and arbitrary functions. Let's see Java do that in 20 SLOC.

Voting third party and government job "creation"

2012-09-29T15:00:00.000-04:00

This is copy of an email I sent to my dad just now.

My concern about the libertarian candidate, besides the fact that he cannot win,

That's no reason not to vote for him. A well-functioning democracy requires a choice of more than the people who are "obviously" going to win. Otherwise, it isn't democracy so much as a decree by the political parties who will be the next president.

My is that his economic plans are hopelessly naive. He pretends that, given Congressional cooperation, one could dramatically cut spending in 2013.

Again, for democracy to work, you need people who are going to do more than what will easily pass.

My The prospect of a much smaller cut is labeled the fiscal cliff and would send us into recession.

I'm not so sure it would be quite that bad. It strains credulity for a trillion-dollar increase in federal spending to be little more than an academic nuisance, but a symmetric decrease to bring about the apocalypse.

I might love to see federal spending go down to 15% of GDP, but if that happened in one day we would have a depression that would make the 1930's look like the good old days.

It just seems weird that spending can shoot up by an enormous margin in one year, but for it to drop by the exact same amount would throw the country into ruin. As a counter-example, I would point to the demobilization following WWII. Government spending fell through the floor, yet the economy boomed, even as it had to absorb the mass of people coming home from the war.

Since Obama will carry MA by a landslide, I want the message of my protest vote to be unambiguous. Voting for Romney says my concern is economic. Voting for a fringe candidate could mean anything.

Well, the fringe candidate has a much clearer message: that neither of the two parties offer particularly attractive directions for the country and that both parties are to blame for the mess we're in.

Besides, whether you vote Democrat, Republican, Libertarian, Green, Wizard, or not at all doesn't make a measurable difference in the outcome of the election, so you might as well vote for your highest preference. See the paradox of voting.

Since you personally are never going to be the tiebreaker, vote for the person who most closely matches your preferences. But what if everyone thought that way? They don't. You are not a collective. And your choice of candidate does not effect a change in other people's voting behavior.

As long as the Republicans hold the House we may stave of the Spanish food riots here. I have great sympathy for them, but they are still thinking there is the alternative of just spending more of that free money.

Incidentally, this is one of the reasons I am in favor of a smaller role for government: there's no one to whom to complain when things don't go your way. People riot against the government in the hopes that the bureaucrats will deign to bestow to each according to his need. Against whom do I riot if the ending of Mass Effect 3 is a major letdown? I can raise money for charity in protest of the botched opportunity, or send ending-related colored cupcakes to the developer's offices (both of which people did), but that's about it. The most impactful thing I can do is refuse to buy their products in the future, since I possess all of the power in that relationship. I bear the burdens and benefits of my actions.

When the target of protest is the government, on the other hand, I can lobby or attempt to vote myself a job and cash. Politicians can try to sell me a free lunch, and since my vote is infinitesimal, which way I vote doesn't have any personal ramifications. It is quite understandable that the Spanish and Greeks want jobs, but when the government is seen to be the solution to unemployment, the focus changes from job creation to job bestowment.

Change GTK keybindings from Emacs back to default

2012-09-20T12:05:00.000-04:00

Like any virtuous person, I use Emacs for everything, and so tried out using Emacs key bindings for all GTK applications generally. After a year or so of working like this, I have decided to go back. The main problem is that other applications can't handle the awesome power of Emacs. Firefox, for example, uses C-k to jump to the search box, but if you are in the location bar, C-k will run the equivalent of kill-line which means that in order to get to the search bar, you have to remove focus from the location bar. Not convenient.

Since I do all serious typing within Emacs anyway (using the It's All Text! addon to edit textareas in Emacs), I figured I might as well take full advantage of the standard key bindings of the lesser programs.

A lot of the guides online talk about editing .gtkrc-2.0 or setting /desktop/gnome/interface/gtk_key_theme through Gconf, but this has ceased to work under Gnome/GTK 3. To fix it, you need to set the value through gsettings (which uses dconf>) rather than gconf:

$ gsettings reset org.gnome.desktop.interface gtk-key-theme

Why would they break backwards compatibility? Who knows. Also, due to the brilliance of the Gnome 3 developers, the data for dconf is stored in some opaque binary blob, rather than the straightforward XML files used by gconf. This is a huge step backwards from the Unix philosophy for what appears to be no change in functionality.

It's things like this, along with brain-dead vomit that is Gnome Shell (or Unity), that have convinced me that all of the Linux desktop developers have simultaneously taken crazy pills and have gone insane. Why else break perfectly good code and interfaces in favor of less functional and less useful crap?

Those Dumb Pipes

2012-09-06T18:04:00.000-04:00

This is an older idea which had been floating around in my head, half-finished for a few years.

I was inspired by this Ars Technica article to think about the relationship between network operators — both cellular and landline — and consumers. The key quote from the article is by "one mobile operator" and is:

Some people like 3 may be in a position where it could make sense to accept that. But if you spend upwards of £40m per year building your brand, you don't want to be just a dumb pipe do you?

This is the fundamental tension between consumers and network operators: one wants internet access to be a commodity, the other doesn't. Why should I care about their brand? Electricity or water utilities don't have a brand. In most (all?) cases, they are government-granted monopolies, so there isn't any choice, but still, the only time I think about a water or electric company is once a month when the bill comes due. The only thing in the world I want from those companies is that when I plug in a lamp or turn the faucet, electricity or water comes out, (hopefully!) respectively. I want the same to be true of ISPs. It annoys me that the word "Verizon" is printed on back of my phone; we would think it bizarre if refrigerators had "National Grid" printed on the doors.

Luckily, we have been moving in this direction since the article was written. The incredible explosion of smartphones has put huge competitive pressure on text messaging, to the point where basically all plans have unlimited voice time and text messages, removing that significant avenue of differentiation. There's less and less room for the networks to do much of anything aside from providing a pipe to the public internet. Despite the fears of Network Neutrality advocates (of which I was formerly one) and a lack of legislative action, no traffic discrimination has taken place. Gradually, the carriers are being ground down into commodities, forced to compete only on quality, coverage, and price, and having their profit margins squeezed ever tighter.

We live in glorious times.

Correctly Dispatching on Generic Methods in `*apply()` in R

2012-08-02T17:50:00.001-04:00

I've been hacking in R recently for work, and it is a bizarrely amazing
language. I just ran into a situation in which I needed to apply an S3 generic
function to a list (actually, a row-wise iterator on a data.frame)
and hit an annoying rough patch

For those not as well-versed in R (I've been learning it for about two weeks at
this point), "method dispatch" is done in a quirky yet effective way. You define
a generic function, say foo like this:

Which means "when foo() is called, dispatch to the function
matching the class of bar". Setting the class and calling the
generic method is simply:

Calling foo(bar) causes R to look for a function named
foo.myClass and calls that. If foo.myClass is not
defined, it will try to call foo.otherClass, then
foo.default and if none of those functions exist, it will throw an
error.

You might have noticed that UseMethod("foo") doesn't pass along the
arguments to foo. R passes the arguments along automagically.

If you want to apply a function to each element to a list, R makes it easy:

identity() and identical() do what you would expect.
There are a bunch of variants of apply; RTFM for the details.

Now let's say we want to call foo() within lapply()
rather than identity. There are some complications:

This throws an error because UseMethod() tries to dispatch on the
class of its first argument, which is a string
("character", in R parlance). One way to make this work is to
reverse the order of the arguments so the iteration variable comes first and
force UseMethod() to dispatch on a different argument:

Yuck! Wouldn't it be great if there was an easier way? Preferably one which
doesn't involve hacking R's method dispatch system? Well luckily, there is. We
take advantage of the fact that R can use named arguments:

It's a bit wonky when coming from normal languages, which wouldn't appreciate a
named argument being assigned to a variable earlier in its argument
list. Again, R is amazingly bizarre and bizarrely amazing. The explicit naming
of quux causes it to be "used up" from the argument list, so that
any remaining arguments are assigned to the "unclaimed" variables. Taking
advantage of this feature of R lets you apply a generic function to a list (or
other iterable R objects) without having to mess around with its
method-dispatching system.

Hopefully this spares a fellow hacker the pain of digging through R's
documentation on method dispatch.

Listing available monospace fonts in Emacs

2012-07-05T15:49:00.001-04:00

After finding out that the Monospace font doesn't support italics, I set about trying to find a suitable replacement. This EmacsWiki page lists a way to print some example text in all of the fonts on the system, which is very helpful for comparing what's available. The problem with it is that it displays all fonts, not just the monospace fonts, which isn't useful. (For those of you not in the know, all programming is done with monospace or uniform-width fonts, meaning that l and m are the same size).

I modified the code so that it displays only monospace fonts. There's probably a property on the font family which one could check, but I used a much more primitive method: measure the pixel width of 5 consecutive lowercase L characters and 5 lowercase M characters. If they are the same, then the font is monospaced.

It isn't perfect (it has a couple of false positives, such as courier 10 pitch), but it's close enough to be a big improvement.

Use FileMaker Web Publishing in Apache

2012-06-19T17:09:00.000-04:00

I've been tasked with nice-ifying the configuration and setup of the systems at work. They store a huge amount (244 GiB) of protein analysis data in a FileMaker database and have PHP scripts accessing the data through the FileMaker XML HTTP API. The problem is that the PHP scripts are running on Apache (as is an SVN repo), but FileMaker wants to publish its API only through IIS. Having spent more than a minute wrestling with configuring IIS, I want it out of my life ASAP. But that can't be done unless scripts have a way of talking XML to FileMaker on port 80. All hope seemed lost.

I thought about reverse proxying requests to FileMaker, since it exposes part of its API on a separate port (16020), but not all of the required functionality can be reverse proxied so easily. I could reverse proxy all of the FileMaker paths on IIS, but that doesn't help get rid of IIS and only makes things more complicated.

The answer came from this blog post from 2009 about this very same issue. I couldn't quite get things to work by following his directions and dug into the FileMaker web server config files.

The IIS module provided by FileMaker was nothing more than the generic IIS-to-Tomcat connector. There isn't anything IIS-specific about Tomcat, and the filter does nothing but point IIS towards a Jakarta config. That being the case, I read the Apache httpd docs and configured Apache the same way as IIS. Rather than replicating the Jakarta files, as the 2009 blog post did, I simply pointed Apache at them, leaving me with this configuration:

This was more hours of agony than it had any right to be, so I hope I can spare you, dear reader, some of the pain I felt.

When European "Austerity" Isn't

2012-06-18T23:32:00.001-04:00

I've been skeptical of the rhetoric that there has been "brutal, crushing austerity" in Europe. When these claims are made, I have seldom heard any quantification of how brutal these "brutal" changes have been. So I looked it up.

Short version: there's been minimal to negative austerity since pre-meltdown 2008.

Longer version:
There was a bump up in spending in the PIIGS (Portugal, Ireland, Italy, Greece, and Spain) countries now in the news and some of those countries went back to around their 2008 level of spending.

I went ahead and looked at the Eurostat data on Eurozone government spending, and it certainly does not paint a picture of a draconian, totalitarian decrease in state spending. I compiled the Eurostat data into a spreadsheet ~~(with US data included for flavor)~~ and plotted out some charts.

Update: I've removed the US data because they are for the federal government only whereas the EU data is for total spending across all levels of government. If anyone has a pointer to a source for aggregated US local, state, and federal spending, I'd be interested in that as well. The claim that US government spending cratered post-2008 seems too politically convenient to be true.

First, the total government spending of the European Union, the Eurozone (those countries which use the Euro), ~~and the US (Dollars converted to Euros)~~ since 2002:

Next, the GPI (Greece, Portugal, Ireland) countries:

And finally, the biggest spenders in Europe. I included them separately from the GPI (or PIG) countries because the chart ends up mostly empty with all included.

As you can see, not nearly the inhumane government takeover of governmental reduction that it's sometimes claimed to be.

As a final piece of analysis, I looked at 2011 spending as a percentage of 2008 (as in pre-meltdown) spending. Only 10 out of the 41 countries (Switzerland didn't have 2011 numbers) had decreased spending since 2008: Iceland, Latvia, Greece, Romania, Hungary, Estonia, Lithuania, UK, Ireland, and Bulgaria. With the exceptions of Iceland and Latvia, all of the countries spent at least 90% as much in 2011 as they did in 2008. Take a look:

Update: It was suggested that I compare 2011 spending to that of 2007 as well, since some countries peaked around then. It actually ends up being more of an argument against the existence of "austerity," with only three countries decreasing their spending over the period: Iceland, the UK, and Hungary.

~~By the way, the US -- the pinnacle of ruthless Reganomics -- spent 120% of its 2008 level in 2011, third only to Luxembourg and Norway in governmental largess.~~

All of my raw data are available here.

Solving mod_dav_svn loading faliure on Apache in Windows

2012-06-08T11:33:00.001-04:00

Yesterday, I spent way too much time trying to get Subversion working under Apache in Windows 7. I really have a new appreciation for GNU/Linux package managers; what would have been sudo aptitude install libapache2-svn under Ubuntu turned into hours of staring at

Syntax error on line 270 of C:/Program Files (x86)/Apache Software Foundation/Apache2.2/conf/httpd.conf: Cannot load C:/Program Files (x86)/Apache Software Foundation/Apache2.2/modules/mod_dav_svn.so into server: The specified procedure could not be found.

I tried copying all of the SVN libraries into bin within the server root, downloading different versions from the web, and "unblocking" the libraries (Windows blocks code downloaded from the internet. Nice idea; I only wish it would tell you about it). Finally, after hours of Googling, I found some post on some forum somewhere that mentioned using the libapr-1.dll from the Subversion distribution rather than Apache's would help. It did.

This would seem to undermine somewhat the idea of the "Apache Portable Runtime". But whatever; I have a working system. Now onto trying to get LDAP set up for SVN and Ubuntu. Wish me luck!

TL;DR: If Apache in Windows fails to start with mod_dav_svn.so loaded in your conf/httpd.conf and you get this error in your Event Viewer:

Syntax error on line 270 of C:/Program Files (x86)/Apache Software Foundation/Apache2.2/conf/httpd.conf: Cannot load C:/Program Files (x86)/Apache Software Foundation/Apache2.2/modules/mod_dav_svn.so into server: The specified procedure could not be found.

, you can solve it by copying libapr-1.dll from Subversion/bin to Apache2.2/bin, replacing the version which ships with Apache.

Metaphor for Europe's Troubles

2012-06-05T01:08:00.000-04:00

Based on my decades of experience in the sovereign debt markets, I have come up with a perfect metaphor for what is going on in the Eurozone. It is as if the Grace government promised 200,000 people one boat each, but now that the boats are due, it turns out that she doesn't have 200,000 boats anywhere, there are only 100,000. Grace tries telling half of the people not to try to use their boat; to pretend that it is sitting in a port somewhere, waiting for them. If some people use their boat half of the week, other people can use it for the other half of the week, so it is "like" having 200,000 boats. Uncle Jeremy has 1.5 million boats, but he isn't so thrilled about just handing over a bunch of his boats to Grace, who knew she promised more than she had. Jeremy says "I'll give you 50,000 of my boats, but only if you tell 50,000 people that they aren't getting a boat at all." Grace is furious, shouting "you're forcing boat austerity on me! I need to implement boat-growth policies!" Of course, "boat-growth policies" is a euphemism for "Jeremy gives me more boats."

Grace then proposes a consolidated "Euroboat" concept, in which all of Grace's, Jeremy's, Ira's, Talia's, and Pete's boats are lumped together and then given out to each of their citizens, regardless of how many boats each person (government) has. It just so happens that Jeremy has 1.5 million boats and has promised 750k of his citizens one boat each, but all of the other governments have promised more boats than they have, and have even promised more than the 750k spare boats that Jeremy has. Jeremy, understandably, doesn't want to give out all of the spare boats he has to people who promised more than they had, but when he tells the other people this, they get angry and call him mean names. Jeremy tries to explain that even if he was to give out all of the extra boats he has, and cut back on a bunch of the boats he already promised to his citizens, there would still not be enough boats for all the people who promised more boats than they knew they had.

Some people want their acquaintance Mario to print them a bunch of boats.

EA support is wonderful

2012-06-01T16:00:00.000-04:00

I contacted EA support today to get my reward for participating in last weekend's Mass Effect 3 Bounty Weekend: Operation Shieldwall. I achieved the individual goal, but didn't get the priiiize, so I went to EA for help. Maybe 10 minutes later, it was done. Thank you to "Larry S" for helping me. Here is the log of my chat with him; I want to give credit where credit is due for his professionalism and speed.

You are now ready to chat with Larry S.

Larry S: Thanks for contacting EA Help! My name is Larry S how may I help you?

you: I participated in the previous weekend challenge but haven't gotten my commendation pack

you: I'm pretty sure I promoted more than 3 classes within the time window

Larry S: No problem. I will be glad to provide you the pack.

you: I certainly promoted more than 3 within the weekend, but I'm not sure it was exactly in the window

Larry S: May I know the name of the weekend challenge?

you: I only started promoting after the operation description popped up

Larry S: May I know the name of the weekend challenge?

you: Shieldwall, I believe

you: let me double check

you: yup, shieldwall

Larry S: Okay. I can provide it. May I have the Origin email id in which you have registered the game?

you: the email I used is dan@haxney.org

you: my origin ID is "haxney"

Larry S: Thank you.

you: Just about everything else related to me is "haxney" :)

Larry S: May I know your first name and last name?

you: Daniel Hackney

you: the "haxney" is a play off my last name. I'm a hacker (original meaning of the word: computer programmer)

Larry S: Would you mind holding for some time while I provide the pack in your account Daniel?

you: sure thing. Thank you very much

Larry S: My pleasure.

Larry S: Please check now in the game.

you: sure, give me a second for it to launch

Larry S: No problem.

you: It's there! That was very quick

Larry S: Great.

you: Thank you, you have been very helpful

Larry S: My pleasure.

Larry S: Thank you for contacting EA. Is there anything else I can help you with today?

you: That's it for today. I shall sing praises of your responsiveness and professionalism to the blogosphere

Larry S: Thank you Daniel.

Larry S: If you need help with anything else, please feel free to contact us anytime!

Larry S: Have a nice day.

you: Have a great day!

Larry S: Take care.

Larry S: Bye.

The chat session has ended. Please contact us again if you need further help.

Simple recursive file traversal in Elisp

2012-04-22T19:09:00.000-04:00

Really quick: I wrote a simple bit of elisp to eval a body over each of the files in a directory, recursively. Looking around for a little bit, I found a couple of options, including Findr, which has its own queue implementation (pretty short), and whose main function is a whopping 55 lines!

There has to be a quicker way, I thought. I didn't need anything terribly fancy, I just wanted something that worked cross-platform (which ruled out find) and was simple. I ended up with this little guy:

11 lines for the main function, recursive-files itself, and another 3 (including docstring) for simplifying macro. It's main weakness at this point is that it incorrectly ignores any files with leading dots. So not only does it (wisely) avoid . and .., it rather foolishly avoids .foo as well. I couldn't easily figure out how to exclude . and .. in a single regex easily without also excluding leading-dot files. For what I'm doing, it's not important.

Also, note that this barfs on symlinks. It thinks they're directories and tries to call directory-files-and-attributes on them. Again, a more careful implementation would deal with this.

I was surprised that Emacs didn't have a "recursively visit files" function or macro built in, but this gets me close enough. Enjoy!

The Ongoing Saga of Making Ubuntu 11.10 Usable

2012-02-27T20:47:00.001-05:00

Alright, I installed Ubuntu 11.10 (aka "lost all common sense" edition) on a new computer, so I have the memory of trying to make it bearable still in my head. The absolute necessary first thing to do is:

And set your login session to "Gnome fallback". This gives you a passable facsimile of Gnome 2 (aka "worked and was not useless" edition). I am a little... offended is to strong, but miffed that "fallback" means "what to use if you want to get work done and are not, in fact, a hipster," but whatever.
You have to Alt+right click (M-<mouse-3> for the enlightened Emacs users) in order to right click on the panels now. Now I know what you're thinking, "what did they set to right click which necessitated bumping right click to Alt+right click?" The answer is "nothing". That's right, if you right click on the panel in Gnome 3, nothing happens. Which naturally might lead you to wonder "so why take something which worked and was simple and make it more complicated while substituting nothing in return?" Because "liquid intuitive natural human smooth desktop metaphor", that's why. Probably something to do with netbook/tablet/$CURRENT_FAD.
Anywhoo, the next step is installing Compiz, which gives you more (read: any) customization and, most importantly, wobbly windows. It used to be a simple click in System → Preferences → Appearance → Effects and away you wobbly went! Now, you have to dig into compiz-config-settings-manager a bit. First, install compiz and the settings manager:

I won't go into all the options here, but if you open ccsm (short for "compiz config settings manager") and look around, you can fix the lack of wobbly windows as well as focus-follows-mouse. I really can't use a computer without those two.
To get rid of the global menu bar (in what drug-fueled rage did this seem like a good idea), just run:

Who doesn't love the idea of scroll bars which try their darnedest to evade your mouse pointer? I know that it becomes even more of a ~~huge pain~~ metaphorical adventure with focus-follows-mouse. If you move the mouse off of the scroll... thing, it changes your window focus. Oh, and did I mention that "off of" can mean "above or below, if the scroll thing thinks you shouldn't have moved your pointer that direction?" Well, it does. If you anger the scroll-thing gods, they will become cross and punish you for your wickedness by making the scroll thing forever evade your grasp. It's kind of like Sisyphus, but you are forever chasing a scroll thing you can never quite catch.
Anyway, to make it die the horrible flaming death it so rightly deserves, run the following (you may need to log out and back in for it to take effect):

Now, to make the window decorators okay again, like they were when everything was fine, run this (you'll probably have to log out and back in again):

And no, I couldn't find an easier way to do this. The advantage the default decorators have over the crappy (aka new) ones is that you can actually tell the difference between the active window and background ones. In both "Ambiance" and "Radiance" the (only) theme options, the difference between the current and background windows is that the window title — not the window bar, but just the title text — goes from gray to grayish. Thanks Ubuntu.
This also has the advantage of restoring the window borders, so that if, for example, you are trying to resize a window, you don't have to hit the one-pixel-wide target area in order to resize. Apparently, according to the geniuses in charge of designing the new interface, being able to resize windows easily is not NEW INTERFACE METAPHOR-y enough.
At this point, you have restored your computer to a reasonably-functional state. You may be wondering, as I did, why the developers of a distribution would take something which was just fine, did what you needed, and was instantly familiar to everyone who has used a computer before, and replace it with the user-interface version of Interior Semiotics. I have two hypotheses:

A couple guys glanced at an iPad, saw that it couldn't do as many useful things as a real computer, and thought "hey, we could make our OS not be as useful as a real computer too!" So they set out to build the worst piece of useless filth they could imagine. One thing that sprung to mind immediately was
I know! Let's make it so that to launch any application, the user has to move the mouse over to the left of the screen, but not the top-left, that's for the close-maximize-minimize buttons.
But that was thinking too small. It took another act of ~~auto-lobotomy~~ brilliance to take it to the next level.
Oh, how about, when the user has opened an application, any indication of other applications on the system or how to launch them disappears! Then, in order to launch other applications, the user needs to go back over to the "Dash" — which is totally not a stupid name, concept, or idea — and try to select something!
The designers were proud of their progress, but frustrated that novice users might still figure out how to launch applications not in the small list of things included by default on the DASH HOME DOCK METAPHOR PANEL. Then came the final piece of brilliance:
Aha! How about instead of having an "Applications" menu which contains a list of applications organized by their function — Graphics, Internet, Sound & Video, etc. — we only have four applications, and if the user wants to launch something else, they can start typing its name! No one will ever be able to get work done once we include that! We'll finally be competitive with tablets in the "least amount of useful functionality" race!
Finally, they were satisfied. Nobody who had ever used a computer before would be able to figure out this turd, and the designers could live their life-long dream of turning something useful into a CES tech demo. Surely, the highest aspiration of any piece of software is to look kind of novel for a couple minutes when filmed over the shoulder of a PR representative at a loud and crowded tech convention. In other words, you want this:
Yeah. That's what people want.
Drugs. LOTS of drugs.

Now, armed with these tools, you can venture forth into the world and use your computer like a computer, rather than an imitation magazine. If you want a magazine you can read while you lean back on your postmodern sofa, go buy a copy of Clash magazine (it's pretty underground, you've probably never heard of it). If you want to lean back and watch TV, then watch TV. If you want to lean back and listen to music, get an iPod, or better yet, an Android phone with a vast multitude of listening options.
But if you have any interest in doing actual work, you get a computer and pray to Turing that the Metaphor Brigade hasn't left the coffee shop and ruined it yet. I suppose if your work entails filming a tech demo of you performing the three tasks you can do with a hypothetical tablet that nobody would pay (or even receive) money to acquire, then keeping up with the latest fads might be important. For those of us who want to open up the computer, start composing a document, and have characters appear on the screen as buttons on the keyboard are pressed, we have been immensely harmed now that "creative" people discovered they could ruin whole operating systems, not just individual websites.
The one kernel of solace is that, because there are users of these formerly-useful operating systems who posses the skills to fix these monstrosities, solutions will continue to exist. And hopefully, once the project leads sober up, all of the trash of the last year can safely be discarded.
Anything is possible, I suppose.

Set ordering in Python: why I didn't sleep last night

2012-02-15T23:46:00.001-05:00

In yesterday's class of the Python-infused Linear Algebra course I'm taking, a fellow student noted the strange ordering of items in a set when printed. It seemed that the printed order of the set items always swapped the first two items, but left the rest unchanged. I don't know what sets he tried, but I was interested in the general behavior, and so began my quest through the Python source code...

BIG IMPORTANT WARNING!!!

These are some interesting properties of the order of iterations over Python's set objects. They are NOT to be relied upon for the correctness of your program! There are a bunch of hardcoded, opaque magic numbers which determine the order, so it could easily change across micro version, platforms, or compilations with different options. Treat sets as having undefined ordering!

All of this is looking at the tag "v3.1.1" in the cpython repository on the Python repositories site. "3.1.1" is the version of python3 on the computers here, so I figured that would be the one to check, though I assume the set code hasn't changed that much. The root of that version is here.

Sets are defined in the source files "Include/setobject.h" and "Objects/setobject.c" for the headers and implementation, respectively. The relevant part of sets is that their elements are stored in a hash table using open addressing for collision resolution. Python (version 3.1.1, anyway) uses the following algorithm for resolving collisions, in Python (with cheats):

entry is actually a pointer (the whole function is in C); I'm fudging the specifics here a lot. mask is table.array_size() - 1 where table.array_size() is the size of the underlying C array. The essence is the updating of index; it is intended to distribute keys all over the array, making collisions less likely. There are a bunch of trade-offs, and they are described in "dictnotes.txt" and "dictobject.c". One interesting fact is that basic integers hash to themselves (and are ANDed with mask for indexing into the array).

The use of open addressing means it can get all objects simply by walking the table. Entries in the set are defined as:

The set object itself comprises an array of setentries and some additional fields, like the number of in-use entries and the size of the array.

Calling list(o) converts a "sequence object" (there is an abstracted interface at the C level) by calling PySequence_List, declared in "Include/abstract.h" and implemented in "Objects/abstract.c". PySequence_List does some basic checks and calls _PyList_Extend, in "Objects/listobject.c", which creates a new list and calls listextend(self, iter(o)) for the real work. It then simply walks through the iterator, appending the items returned to itself, doubling its length if it runs out of space. So the ordering is determined by the iterator for sets.

The work is done in the setiter_iternext() function, which walks through the backing array until it finds a non-NULL entry and returns that. So the order of objects from the result of list(s) is determined by the order of elements in the table. That is determined by PySet_Add which calls set_add_key which uses PyObject_Hash to get the hash of the object to add.

PyObject_Hash looks into the object's vtable (Python doesn't call it that, it's stored in the object's ob_type field) to find its tp_hash function and calls that. tp_hash returns a long. Each object has its own hash function; this is what is used for a couple of common types:

double: uses the integer part of the float (with some extra stuff)
pointer: rotates the pointer right by 4 bits (since the lower 3-4 bits are likely to be zero)
int: the absolute value of the int modulo the platform's maximum long value. This actually can get interesting, since Python longs are arbitrary precision. The hash is computed through some additions, rotations, and fun stuff like that.
tuple: the hashes of its objects, XORed together and multiplied by a sequence of constants to mix up the value, and something having to do with primes.
string: XOR of bytes (not "characters"! Know your Unicode!).

So the order items from list(s) is "mostly" stable for sets containing the same objects. The exception is when objects with the same hash are inserted in different orders; because of open addressing, the order of insertion determines their place in the array. The following case should hold true:

So it is possible for sets with the same contents to have different orderings, but it requires a collision of hash(a) % mask. Remember, (for ints) hash(i) == i when i is not an arbitrary-precision numeric. You can create a collision easily by exploiting knowledge of the initial size of the internal array. Rather cleverly, the set object includes an immediate array of PySet_MINSIZE elements (defaulting to 8) within the structure itself. This means that if your set doesn't grow beyond 5 elements (the hard-coded load factor is 2/3), there is no extra memory allocated for the table. Given those two facts, all you need is (a, b) such that a & 7 == b & 7. How about (0, 8)?

You can then do:

I'm not sure why the order of the elements is reversed; this only happens for certain combinations:

I started investigating deeper into how Python fills its hash tables, but I decided to stop after a dozen hours, reading through Python's parser code, and trying to figure out pixel offsets to pass to PyCairo during visualization. My progress thus far is here. I'm trying to get a visualization of exactly how the entries are being added, but porting Brandon Rhodes' dictionary code to work for sets is non-trivial. Also, getting the visualization to display in a Gtk window rather than writing to a file took some fiddling around with PyGtk and PyCairo.

tl;dr: the order of objects in a set when converted to a list (such as during display on the repl) is dependent on the hash of the object and the collision resolution strategy. The pattern found by the student in which the first two elements of the set were reversed was a coincidence, as the indexing function attempts to distribute keys uniformly over the table.

I spent WAY too much time on this. Interesting, though.

What a functional desktop looks like

2011-12-13T21:00:00.001-05:00

As a followup to my first Un-Terrible-ize
Ubuntu 11.10 post, I thought I'd show everyone what my computer looks like
after reverting the desktop to a working state.

Here is what it looks like now:

Nice, neat, with all of the things I want and none of the things I don't want. I
have a couple of customizations for Firefox (like the tabs on the side), and
along the top is the "System Monitor" applet, which displays (from left to
right) the current CPU usage, memory usage, network activity, swap space, load
average, and disk usage. That's undoubtedly overly geeky for many people, but I
find it quite useful.

Now, for comparison, here is what the default Ubuntu 11.10 interface looks like:

This is a direct screen shot. True story.

Later on, I may post a step-by-step description or even a script for getting
from the bottom image to the top.

Configuring Printers in Ubuntu 11.10

2011-12-13T17:55:00.002-05:00

Welcome back to my ongoing adventures in making
Ubuntu work once again. Today's episode features printing!

So upon trying to print a document, I found that my printer hadn't been set up.
No problem! I guess I had forgotten to do so. So I jetted on over to the "System
Settings" application, which helpfully had a "Printers" button. "Why hello Mr.
Printers button," I said. "Let's go on a magical printer-configuring adventure,
just the two of us!" Clicking on Mr. Printers, I was greeted with the Nice™,
Shiny™, Sexy™ Interface below:

Since my objective was to add a new printer to Mr. Printers' Magic List™, I
thought I'd click the "+" button.

"Not so fast!" Mr. Printers said gleefully. "That button is grayed out, so
clicking on it does nothing!" I should confess that he didn't actually say this,
since clicking the grayed-out button did exactly nothing, but I like to think
that this is what he meant, in his silence.

Despondent, I searched for a clue. My eye caught sight of the "Unlock" button in
the upper-right corner of Mr. Printers' window:

"Aha!" I exclaimed with joy, "I bet this will unlock the secrets of the
printer!" I clicked on the button, entered my password to confirm that yes, I
did want to modify system settings, and lo and behold, the "+" button was no
longer grayed out! Being the masterful designers they are, the team responsible
for the "+" button saw fit to make the difference between "stubborn, indignant
Mr. Printers" and "helpful, charitable Mr. Printers" as clear as day. Just take
a look!

What could be clearer?!

After that, I clicked on the now-enabled "+" button which whisked me away to
this lovely dialog:

Now, I should clarify, in older versions of Ubuntu, the "Add Printer" dialog
would automatically detect any printers on the network, allow you to connect
with them, and figure out which driver to use, all without needing to wade
through lists of devices. Now, with Mark Shuttleworth's Amazing
Intuit-Interface™, that pesky problem of things working has finally been solved!

So after a bit of Googling, I found out that you could use the old, Actually
Working™ printer-adding interface by opening a terminal (yes, yes, I know) and
running the program system-config-printer. That opens this old
friend:

Whose "Add" button actually works!

So thank you, Mark Shuttleworth, for once again taking something that worked and
turning it into something that doesn't work. This clearly is progress.

TL;DR If you want to add a printer, skip the "Printers" option
in System Settings and run system-config-printer instead.

Un-Terrible-ize Ubuntu 11.10

2011-10-24T01:33:00.000-04:00

Oh my god. After years of being a great, easy-to-use Linux distribution, Ubuntu finally has failed me and may need to be dumped. I just upgraded to 11.10 (Oneiric Ocelot) and they have made it more difficult than before to return to a not-terrible window manager. I think the idea behind Ubuntu's new "Unity" interface are cute, but is just god-awful for people who want to use their computer for things and don't like desperately trying to hunt them down. What was wrong with an "applications" menu that had an "Office" tab under it with office-related applications? Now, you open the side panel overlay bar thing and it takes up the whole screen but presents you with almost exactly zero useful information. It has something like 3 applications along the top and then says "fuck you, go figure it out" if you want anything more. Alt-F2 doesn't work to open a launcher to launch an application. Alt + middle-click doesn't resize windows any more. Just getting out of my damn way and letting me use the computer like I have for years doesn't even close to work.

GNOME 2 was just fine. Why did you have to take something which worked, in a non-terrible fashion, and remove all of the useful, discoverable, productive parts of it? I've spent a couple hours at this point trying to get back to the level of non-uselessness I had with Ubuntu 11.04, which was not useless. I even tried installing GNOME 3 in the hope that it would have a "stop doing stupid bullshit" mode, but it only kind of does. I've gotten close, but the "system" menu is AWOL, and I can't figure out how to get it back. It doesn't seem to want to let you customize the theme at all or even have something as simple as focus-follows-mouse. Yes, I've opened gconf-editor and set /apps/metacity/general/focus_mode to mouse, which seems to be the closest thing to focus-follows-mouse available, but it doesn't do it.

I'll post what information I've found to get it to the minimally-crappy state it's in now, but I'm still searching for how to de-terrible-ize it further. Alternatively, I might just dump Ubuntu altogether until it gets some sense beaten into it. I hear Linux Mint is supposed to have remained safe from the "enshittening" of Linux user interfaces of late. Also, I could try Xubuntu, though I've never really been a fan of XFCE. It's a heck of a lot better than Unity or GNOME 3.

No, nevermind. Not using XFCE.

Okay, after a few more hours of wasted time, I'm going to bite the bullet and back up all of my documents, wipe 11.10 off of my computer, and go back to 11.04, aka "the last sane version of Ubuntu". It really sucks because I used to recommend Ubuntu to my relatively non-technical friends, but with the crap they're pulling now, there's no way I'd let their first Linux experience be with this trash. Sad day.

Changing up the style

2011-10-18T17:17:00.000-04:00

You may have noticed a change in the style of the site. That's because I changed the style of the site.

Okay, seriously, I switched to the new Blogger interface/template, so things look a bit different. I tried fiddling around with some things, and hopefully nothing broke. To all of you massive numbers of readers of this blog, let me know if anything went wrong in the upgrade.

Genericizing Rails view templates

2011-08-21T22:18:00.000-04:00

In keeping with my previous post, I have put together a way of greatly reducing the redundant code in Rails views. On the HTML side, this really only applies to the simple case of rendering the default REST views provided by Rails' scaffolding mechanism.

Rather than have an index.html.haml (or erb if you're a bad person ;) for every model you have, why not consolidate all of those into a single template? They all do the same thing anyway: render a table of objects along with a few actions. Using the wonderful render_inheritable gem, you can define a single index.html.x file in a central location (such as app/views/application) and have all of the customization occur at runtime. Thanks to render_inheritable, the 4 index files for my 4 models can be reduced to this:

You'll need to define a entry_class method in each of your controllers which returns the model's class. So in the UsersController you would have:

The same applies to the _form.html.x file. Using the wonderful formtastic gem, creating a form which works for all of the models is as simple as this:

These are all pretty trivial and contrived; in any real application you wouldn't reuse the exact same view for all of your models. Where it becomes more useful is when crafting API responses, which tend to have more regular formats. Using the great rabl gem, you can create view templates for JSON responses like you would for HTML responses. Instead of mucking about with to_json in your models or controllers, define the format of responses in a view, where it belongs.

Used with render_inheritable, you can do some truly awesome things with your JSON (or XML) output. To create an index template which returns a per-model customizable collection of objects, simply create a file at e.g. app/views/application/index.json.rabl with the following contents:

You'll need to define a function called entry_views_folder_path in your ApplicationController which returns e.g. users for responses concerning User objects endpoint or groups for responses with Group objects.

Doesn't look like much, but mix that together with base.json.rabl files for each one of your models, such as:

you now have a set of templates which render a collection of models, specialized for each class. That's pretty cool if you ask me.

Stay tuned for a way to render these customized API templates when errors are encountered, while still preserving the original error codes. It took me hours of digging through the Rails source to figure out.

Scala on Android with Eclipse

2011-08-21T21:25:00.002-04:00

Yes, I'm back from the dead. I've been busy starting a company of late and have been pretty much full speed ahead on that, leaving less (read: none) time for blogging.

Anyway, on with the main event: a working setup for developing Android applications in Scala. I spent about 3-4 days total trying to get this setup working, using various different schemes such as the Treeshaker plugin for Eclipse, the Maven Android plugin and others. After hours and hours of fiddling, I finally found a solution that works: a custom SBT setup with the SBT Android plugin. The key is to avoid building the Android app in Eclipse itself, but to use SBT for the heavy lifting.

Eclipse seems to have some very non-deterministic behavior when building applications, especially Android applications. When you add Scala and Treeshaker into the mix, the whole thing falls apart. Furthermore, the build process is very opaque, so it's hard to figure out what is even going wrong, let alone to try to fix it. A build which was fine before might suddenly stop working, raising some exception or simply pegging the processor and making Eclipse totally unresponsive. There doesn't seem to be any pattern to the madness, so it's impossible to fix.

Anyway, as I mentioned, SBT with the Android plugin works beautifully for building apps, but as much as I love Emacs, it is easier to work with Scala in an environment that gives you clear and immediate feedback about type failures and requirements. On the downside, in order to do all of this wonderfulness along with the Android helpers, Eclipse expects a very specific directory layout, and will complain if it does not exist. SBT, by default, uses a completely wacky layout, with the Android manifest files (usually at the root of the project) in src/main/. This means that the actual source code files are in src/main/src/main/java/com/example/.... For a single project without references to other projects, this is unnecessarily complicated and destroys the tiny brain of Eclipse. So the solution is to change SBT's layout to something sane.

Note: I'm using SBT 0.7.x since the SBT Android plugin hasn't yet been ported to 0.10.x

First of all, you need to declare the plugins you will be using (for now just the SBT Android plugin) in project/plugins/plugins.scala. The actual filename doesn't matter, as long as it is in the project/plugins folder. The plugins.scala looks like this:

Standard fare for SBT plugins.

Next is the SBT project file. This is where the magic happens, though there isn't really that much magic to it. It simply changes some paths so that SBT looks in the correct place. Here's the file:

The keys are the five lines in the middle setting the path info. The androidManifestPath assignment causes SBT to look for AndroidManifest.xml in the project root, as do mainResPath and mainAssetsPath. Setting the scala and java source paths to src means that SBT will look in that folder instead of the redundantly redundant src/main/src directory. Finally, overriding aaptGenerateTask simply makes aapt write the generated resource files (R.java and friends) to the gen folder, as the Android Eclipse plugin expects.

With the hard stuff out of the way, you "simply" need to configure Eclipse to use the appropriate builders and classpaths, included in the .project and .classpath files, respectively. Here they are:

.project

and .classpath

Obviously, since the config files include the Scala libraries and builder, you'll need to install the Scala Eclipse IDE. With all of this in place, you should be able to issue a simple sbt start-emulator to build, Proguard, package, deploy and start your application.

Hopefully this post will save you from the pain I endured of trying to get Scala working on the one hand, and the thought of having to use Java on the other.

MineCabinet - a database-backed chunk manager for Minecraft

2011-02-11T15:45:00.000-05:00

As I alluded to in my last post, I have been working on modding Minecraft to use the excellent Tokyo Cabinet database to store the chunk data. I have finally gotten it to the point where it is ready to be released to the masses.

Here is a link to the zip file. It contains installation instructions in the README.txt. All of the basic installation information is contained there, so if you're really interested in how it works, hit that download link! :)

Basically, I replaced Minecraft's current ChunkLoader implementation with a new one based on Tokyo Cabinet. Rather than saving each chunk in a separate file, MineCabinet stores everything in a single Tokyo Cabinet database, which reduces the number of files for a 60MB world from 17,000 to 1. It also boosts performance by roughly a factor of 4, according to my basic profiling results. here are a few runs from JVisualVM for vanilla Minecraft, McRegion, and MineCabinet. There are two sets of results, one for creating a new world and running around for a few minutes and then saving and exiting the world, and another for reloading that same world.

First, the profiling for initial world creation:

Vanilla Minecraft - initializing new world. 1030 calls to oj.a(dn, ib) (aka saveChunk) in 4522 ms or 4.4 ms/call

McRegion - initializing new world. 1076 calls to oj.a(dn, ib) in 4060 ms or 3.8 ms/call

MineCabinet - Initializing new world. 996 calls to saveChunk in 728 ms or 0.73 ms/call

And for loading that world back from disk:

Vanilla Minecraft - loading existing world. 570 calls to oj.a(dn, ib) in 2605 ms or 4.5 ms/call

566 calls to oj.a(dn, int, int) (aka loadChunk) in 868 ms or 1.5 ms/call

McRegion - loading existing world. 590 calls to oj.a(dn, ib) in 2424 ms or 4.1 ms/call

581 calls to oj.a(dn, int, int) in 528 ms or 0.91 ms/call

MineCabinet - loading existing world. 562 calls to saveChunk() in 259 ms or 0.46 ms/call

556 calls to loadChunk() in 643 ms or 1.2 ms/call

So as you can see, a very large speedup. Additionally, Tokyo Cabinet has a number of tuning parameters which could affect performance, including switching from using a B+ tree-based database to a hashtable-based database. The hashtable apparently has higher space overhead, but can be more efficient in some circumstances. I didn't really investigate the impact of various tuning parameters on performance, so there is some potential for even more speedups.

One thing to note, though, is that MineCabinet spends a large amount in saveExtraData(). This is because I put a call to db.sync() in there (it was a noop before). It gets called when a world is unloaded, so it seemed like a good time to sync. Although this is a large amount of time, it is only done when switching worlds, so five seconds here isn't as noticeable as a lot of little pauses throughout the game. A more intelligent mod could force a sync (in a dedicated thread) periodically, say after a (configurable?) number of in-game ticks.

Another nice thing about using Tokyo Cabinet is that it obviates the need for managing the "session.lock" file in each world directory. In vanilla Minecraft, this file is checked (twice) for each chunk save to make sure that the directory isn't being written by another Minecraft process. This adds more (though not much) overhead to each chunk save, and can be avoided entirely with Tokyo Cabinet. When opening a database, TC takes a file lock on the file it's opening, so any other process (reader or writer) is blocked from opening that same file. Since a database is opened for the entire session of a world, no other process can modify or read the database while it is in use.

Current limitations

All is not perfect, however. Currently, I only provide a native library for 32-bit Linux, since that is the only build environment I have access to. Building Tokyo Cabinet (and the Java bindings) for Mac OSX shouldn't be hard, and it looks like someone already has a build made.

Windows is harder, since Tokyo Cabinet does not officially support it. It may be possible to build with MinGW, though I've never used it. Tokyo Cabinet doesn't have many dependencies (zlib, bzip2, and pthreads), so it may be possible to get it to build and run on Windows. I'll have to see.

Another possibility is to use the newer and improved-er Kyoto Cabinet, the successor to Tokyo Cabinet which includes (among other things) official binary packages for Windows. The downside is that unlike Tokyo Cabinet, Kyoto Cabinet is licensed under the GPLv3, making it ineligible for use in Minecraft without a commercial license. Mojang may decide that it's worth it to go for the commercial license, but I didn't want to assume that, plus, I wanted to remain closer to the side of legal distribution (Minecraft mods are always a gray area).

Closing remarks

My ultimate hope is that MineCabinet (or something similar using an efficient DB) is included in the main Minecraft client, since I think using an embedded DB would provide a number of advantages over even a well-done custom format (such as McRegion). One of the biggest benefits is simply letting someone else deal with the tricky problem of managing persistence and concurrent access. My opinion is that if you can make your problem someone else's problem, go for it, especially when they specialize in solving that problem. Also, databases like Tokyo/Kyoto Cabinet have effective methods for dealing with both intra- and inter-process concurrent access, which is a notoriously difficult problem to get right.

There's plenty more I could say, but this post is long enough as it is. Please feel free to give feedback and suggestions, especially in the form of code ;)!

P.S. A big thank you to Scaevolus, author of McRegion and Minecraft modding expert. Without his help, I never would have been able to navigate the Minecraft sources and make the modifications to put the whole thing together.