ruddzw.com

Lenny’s got some competition coming…

Wed, 02 Oct 2013 20:44:15 -0400

Lenny’s got some competition coming…

Time to fix the caches

Tue, 01 Oct 2013 17:52:43 -0400

Time to fix the caches

Attempting to make StarLicker work on low-memory devices...

Thu, 26 Sep 2013 19:04:05 -0400

Attempting to make StarLicker work on low-memory devices #instruments

starlickergame: The first StarLicker trailer I’ve been...

Tue, 28 May 2013 16:09:00 -0400

starlickergame:

The first StarLicker trailer

I’ve been working on this game for several months now, as co-founder of my company Heartonomy. This is the first trailer.

Data Ex Machina

Wed, 13 Jul 2011 00:00:00 -0400

We needed to start over. Our old app had reached its limits, and we were cutting down on functionality because of it. My team’s old app was extremely tightly coupled with a Microsoft SQL Server Reporting Services (SSRS) server, which had been hacked up to use the web app as a data source and generate reasonably nice-looking reports, and the hackage had determined most of the architecture of the web app code. In addition, despite lots of searching, we could not find a way to get results back from the database without loading them all into memory, using our current framework. We wanted to do large data exports, so that was unacceptable.

Besides, the web app was written in Java. And not good Java—the “Enterprise Java Bean”-filled XML-instantiated messy Java. Yuck.

Building a new web app would enable us to slowly move off of the old app, using the magic of a well-configured load balancer that we already had in place anyway. Our plan was to first build the data export feature, since that would actually have some customer impact. Of course, the wonderful side-effect would be being able to ditch SSRS, which had forever plagued our developers with impossible-to-update code and confusing logic. An app free from the chains of SSRS could be written well and let us build better, more interactive reports. But first, we needed to think about those data exports.

We held a naming contest, and its name from then on would be Data Ex Machina, or DXM for short.

The Search for a Good Python Framework

We were sure that we wanted to build our new app in Python, because each of us on the team had experienced firsthand the difference in productivity between Python and Java, and we knew that the amount of work that the app does is small enough that Python wouldn’t become a performance problem. Since we wanted to get the technology choice right this time, we spent a little while doing some investigation as to what framework we were going to use. The frontrunners were Django, Pylons, and Tornado.

I was pulling for Tornado, because it seemed nice to work with, plus its default model of working was in streaming data, rather than building up an entire response and sending it to the client. Since we’d have to do that exact thing with the data exports, it seemed like a natural choice. But after a little bit of time with that, it became clear that we’d actually have to build our own asynchronous database driver. Our queries are extremely long-running: several seconds for simple reports, and under load, multiple minutes for complex reports. Building an asynchronous driver is surely doable, but a normal framework would be fine for us. That meant that Tornado would have to have something monstrously better than Django or Pylons for us to commit to it.

Between Pylons and Django, we were having a hard time deciding which was better than the other. Each was mostly good with a few small negatives each. But when we started load testing our proof-of-concept apps to see which performed beter under load, we immediately noticed that each server had terrible performance. Somehow, our database queries were executing serially rather than in parallel. After some work, we narrowed the blame down to the database connections, rather than Python or the frameworks. We were using pyodbc to connect to our database, a Vertica cluster, since Vertica doesn’t provide a dedicated driver for any language, only ODBC and JDBC drivers. We even made a small test to show that the problem occurred with PyODBC connecting to two different databases, but not with psycopg2, a native PostgreSQL library. We tried posting on the PyODBC newsgroup, but after a lack of response, we had just about given up on Python. We’d have to go to Java after all.

One Last Hope

But wait! What about Jython? Sure, it only has compatibility up to Python 2.5, but that’s better than nothing. Both Django and Pylons were supposed to work under Jython, so we gave that a try. Luckily for us, switching our proof-of-concepts to run on Jython went fairly smoothly. They performed admirably under load, and dutifully streamed CSVs of data out to our load testing script.

So, we were left with two decisions. First: Django or Pylons? Pylons was more used within the company, so that would have been the default choice. However, no one else was using Jython, so that throws out the shared knowledge argument. I liked Django a bit more, even though we wouldn’t be using a lot of its core features like database model management. In the end, Django won out simply because it seemed like there was a larger community of people using Django on Jython than Pylons on Jython. And now, though it wasn’t a factor then, Pylons has been end-of-lifed in favor of some new thing called Pyramid, so other people who previously chose Pylons are having to rethink their decision.

The last decision was whether it was worth it to go with Jython instead of Java. This was entirely subjective, but we knew that Jython was sufficiently mature for us to depend on it. We would have preferred Python, but without a way to use PyODBC, we were out of luck.

⋙

Of course, over 6 months later, the PyODBC maintainer finally did follow up on the problem, and someone else emailed me privately with a solution (add Threading=1 in the driver definition in odbcinst.ini) that would have allowed us to use Python and PyODBC. By the time that happened, we had already built a few nice things that depended on JVM libraries. If we want to switch back, we’ll probably have to port the functionality ourselves. Besides, life is pretty nice on the JVM.

⋙

Update: September 2012

Thanks to the “fix” I listed on the PyODBC group, we were finally able to make time to test out the theory and confirm that the lack of Threading=1 was the culprit. The continual lack of energy in the Jython community led us to want to move back to CPython. We were missing out on several libraries that were only available for CPython, and even more internally built libraries that had to be forked and modified to work with Jython, which doesn’t support Python 2.6 or 2.7 syntax features. So, as of September 2012, we removed all the JVM libraries we depended on (not many), and switched over to deploying with CPython 2.7, using PyODBC. We’re deploying virtualenvs instead of WARs and using mod_wsgi instead of modjy. And we couldn’t be happier with the situation.

What Java is Missing

Tue, 28 Sep 2010 10:55:00 -0400

Java’s syntax has been sitting dormant for years. I started writing Java over six years ago, and the major addition since then, as far as I can tell, are generics and the “for each” loop. I’m happy to have these, most of the time, but the more I program in other languages, the more things I miss when I’m back in Java. Some things, like pattern matching for function arguments, I’m never going to get. They just don’t fit into the Java syntax, much less the way that Java’s programming model works. But there are a few things that would be easy enough to integrate into Java that it’s time to put them in.

List and Map Literals

This is one of the simplest features of other languages, because lists and maps (or dictionaries, or hashes) are built in as basic types. Unfortunately for Java, these fundamental types are tucked away in the java.util package. It’s not like these are ever unavailable, but it is a serious problem that to use the types, you have to import them. And then there’s the problem of multiple implementations of each. A Map is just an interface. It might be a HashMap, a TreeMap, or some other custom implementation. A List is also an interface, and in fact it’s a subinterface of Collection, and Iterable. Do we ignore the other Collections such as Sets and Queues?

Let’s assume a literal that essentially means an ordered collection and start from there.

["Alice", "Bob", "Charlie"]

This follows the example of many other popular languages to create a list. However, with Java, we don’t have the option of just having the compiler choose a type for us. We’ve got to specify it directly.

new ArrayList<String>["Alice", "Bob", "Charlie"]

Instead of creating an empty ArrayList using parentheses to invoke the constructor, we use this new square-bracket list notation. The great thing about this notation is that it doesn’t have to only specify a list. We can use the exact same syntax to specify a Set:

Set<String> names = new HashSet<String>["Alice", "Bob", "Charlie"];

There’s not much magic required behind the scenes. You could say that any class<E> that has the add(E) method could take advantage of this. Suddenly every ad-hoc list you would want to create is a one-liner rather than an (n+1)-liner.

But what about maps? Maps are based on pairings rather than single items. Given what symbols are used in different places in Java, I’d say that adding some parentheses would be the best choice in this case.

Map<String, Integer> grades = new HashMap<String, Integer>[("Alice", 92), ("Bob", 79), ("Charlie", 98)];

As long as you’re a class<K, V> that has the put(K, V) method, you’re good to use this syntax too. Java loses no expressiveness, as far as I can see, and we as the programmers finally can implement our simple data structures, without having to resort to multi-line ugly hacks. This code makes a few other things a lot more simple, as well. If you’re declaring some Collection or Map as final, then now your initialization code has been made significantly easier.

Tuples and Multiple Return Values

Tuples are a great feature of Python. They are at once overhyped and underused, but extremely powerful. When you’re learning Python, you’re likely to encounter this little gem, which allows you to swap two variables using the power of tuples:

var1, var2 = var2, var1

Awesomeness factor? Off the charts. Actual usefulness? Mediocre at best. But the underlying functionality is important and quite handy. The way the variable swapping works, as I understand it, is that we first evaluate the right hand side. It takes the values from var2 and var1 and puts them into a tuple of length 2 (a double), indicated by the comma. Based on that tuple, it uses auto-unpacking to take the values from the tuple formed on the right hand side and assign them to the variables on the left, which is also sort of a tuple. Another way to write this with more explicit tuples would be:

(var1, var2) = (var2, var1)

Tuples and this auto-unpacking make it easy to do all sorts of things, from extracting values of a list/tuple into separate variables, to essentially returning multiple values.

before_comma, after_comma = "Hello, World!".split(",")

first_value, second_value = ["Foo", "Bar"] # Only works if the list has 2 items

peanut_butter, jelly = retrieve_pb_and_j()

Tuples and auto-unpacking are sorely missing from Java. Say we’re building the Dee-lish Peanut Butter and Jelly Sandwich Maker 9000™ machine, and we’ve decided to use Java. If we had this tuple functionality, we could just tell Java to go and get the peanut butter and jelly from the fridge like in the last Python example. Of course, we don’t have that functionality.

Instead, our choices with Java are relatively poor. We can tell Java to get us the peanut butter, and afterwards, get us the jelly. We can also construct a special class to hold a peanut butter jar and a jelly jar.

class TwoGreatTastesBetterTogether {

private PeanutButter peanutButter;

private Jelly jelly;

// ...

}

Of course, if we’ve gone that far, we might as well construct a much more elaborate sandwich-makings housing. It can hold any number of sandwich ingredients. Whatever you want, however you want it. It’s ready for professional kitchens!

class SandwichIngredientsMonstrosity<Ingredient> {

private Collection<Ingredient> ingredients;

// ...

}

Yikes. I really just wanted a PB & J maker.

public PeanutButter, Jelly getPbAndJ() {

// ...

return peanutButter, jelly;

}

// ...

PeanutButter pb, Jelly j = getPbAndJ();

Ahh. Much better. It doesn’t quite have the dynamic qualities of the Python implementation, but it gets the job done.

Other people get this done with a generic Pair<A, B> class. Sure, we could do that. But then we also need a Triple<A, B, C> and a Quadruple<A, B, C, D>. And so on. This is not a good solution. What happens when we need just one more than people have included in the standard libraries? As a rule, when choosing how many of something we support, we should strive to either support zero, one, or n. Java has gotten down being able to return nothing or one thing, but returning more than one thing is the next step, not returning one thing that represents a predetermined number of other things.

First-class Functions and Lambdas

These are actually due to show up in the next version of Java. Or rather, they were until Oracle bought Sun and broke the schedule. Now they’re due to ship in the version after next. But regardless, it’s in the works. And thank goodness, because this is a big feature that I wouldn’t try to pass off as a simple addition. Based on what we’ve seen so far, this actually shows that Java is capable of more guessing than it’s letting on. The syntax available so far looks something like this:

// Type one

#()("No arguments").();

#(Object argument)("My argument: " + argument.toString()).(1);

// Type two

#(){ return "No arguments"; }.();

#(Object argument){ return "My argument: " + argument.toString(); }.(1);

The first type is exactly what we’re looking for in a lambda. We specify the type of the arguments, but not the return type. We just write the one-liner as an expression. The second type has much more potential. Nothing here is limiting us to one-liners. These are full-fledged functions. Java, which previously was home only to methods, now would have first class functions. The type of these things has been hidden so far, but it was shown that a thing like the following would work:

Object myFun = #(){ return "Hello, World!"; };

If we’re not lucky, the final syntax will be something like this:

Function<R><E, F> myFun = #(E arg1, F arg2){ return new R(); };

If we’re lucky, we won’t have to specify the argument types or return types using some sort of generics:

Function myFun = #(E arg1, F arg2){ return new R(); };

Either way, we’ll now be able to pass functions around, create functions inside other functions, and generally do all sorts of other cool stuff. This would be the biggest leap in functionality for Java in a decade, far more than the minor proposals for tuples, literal collections, and literal maps above.

⋙

It’s clear from the initiatives intended for Java 7, like Lambda, that Java’s creators want it to prosper and gain back developer mindshare, so I just hope that they will be able to get these much-needed features in there soon. I’m generally fine with Java’s horribly excessive syntax, as long as it’s got the features that put it on par with other JVM languages.

In the meantime, it’s Jython for me.

Reduce, Reuse, Recyle

Wed, 07 Apr 2010 10:48:00 -0400

It’s a trio of suggestions that you’re sure to have heard. Growing up as a kid, I heard this line all the time, from commercials, from school, from my parents.

Reduce, reuse, recycle.

That little triangle with the arrows for corners is burned into my mind for life. Yet, the meaning never really sunk in. To me it all boiled down to the last word: recycle. I learned to separate my normal trash from stuff like bottles, cans, jars, paper, and other things deemed recyclable. To me, that was the epitome of the whole movement.

The first two Rs completely went over my head. I never gave a passing thought to how I might reduce something, or reuse it. When I filled up a paper cup with water, I gulped it down and threw it in the trash can. I can distinctly remember a few times where I would fill a cup, drink it quickly from thirst, throw it out, then go for another cup to fill up, having thrown away the first cup before I realized I was still thirsty. Even though I knew that I should have just kept the first cup, I didn’t worry about it. Can’t do anything about it now, after all.

I never gave a thought to how I might reuse paper cups more than once, like using only one per day instead of four or five. Likewise, I never thought about reducing my use of paper cups to use, for instance, a water bottle. I could use a water bottle for months, at least. But none of that happened. As far as I was concerned, as long as I was putting my aluminum cans in a separate container, I was doing my part for the environment.

And it’s not just me. Plenty of other people missed the message. But recently they’ve started to get it. I see people carrying water bottles around all day. The recent concern for the environment has sparked a movement that people have really gotten behind. Companies and countries now strive to be sustainable. But what does “sustainability” really mean? On the personal level, there’s not much more I can do today to help the environment than there was 20 years ago. I think the main difference is that people are starting to get the other two Rs. Rather than just sorting their waste before they toss it in bins, now people are trying to make less waste in the first place. They’re reducing and reusing. They were important all along, but people understand them now.

As it turns out, the same principles make sense in the world of code. Except this time, it’s not the recycling we’ve all perfected.

In the programming world, we’re masters of reuse. Laziness is a virtue. We practice and preach “Don’t Repeat Yourself”, we actively seek out existing solutions, and we always try to build things for others to use. When 37signals extracts Ruby on Rails from one of their web apps and makes it into a full-fledged web framework, that’s reuse. When a single developer writes a function for some shared functionality, that’s reuse. From our companies’ actions to our modular code, programmers live and breathe reuse. It’s something that’s been drilled into us from our first years as programmers.

However, just like recycling in the world of waste, that’s not all you need to do. It gets you a long way, but not far enough.

What about recycling? That is, what can we do to make completely new and better things out of what we have or used to have? On the best end of this, it’s like when the PyPy project uses the power of Python to build a Just-in-Time compiler to be able to run a Python program faster than the standard C implementation. On the end that is achievable by programmers everywhere, it’s like when you use the knowledge of the many bad design decisions you made the first time around to completely rebuild your product for version 2.0 (or 3.0, or 4.0, etc.). When you’re spending days fixing simple bugs or adding simple features because of bad design decisions, or when you’re spending hours spending time trying to even find out if you’re going to be able to implement a simple feature, then it’s time to recycle your code. Armed with the knowledge you have now, you can save yourself days, weeks, months of time by building it right this time.

That’s the importance of recycling. However, it has been said that it’s almost always a bad idea to do a complete re-write of your codebase:

It’s a bit smarmy of me to criticize [Netscape] for waiting so long between releases. They didn’t do it on purpose, now, did they?

Well, yes. They did. They did it by making the single worst strategic mistake that any software company can make:

They decided to rewrite the code from scratch.

From Joel Spolsky’s Things You Should Never Do, Part I

You’ll lose all that knowledge that’s stored there in the code, supposedly. Sure, I’m not disagreeing with that. Save recycling for the most desperate of times. Ask yourself if it really needs a rewrite. There are cases where it does. When the Mozilla Foundation built Firefox as a lightweight browser to supplant the heavyweight Mozilla browser, that was a good case of recycling. When Netscape recoded Netscape and ended up with a buggier Netscape, that was a bad case of recycling. In most cases, it’s just time to reduce.

Reducing is an exercise best practiced when you’re writing the code, not fixing it. Think about whether you’re making something a little too reusable. Say you’re planning on supporting internationalization. It’ll make your text modular, thus making your codebase simpler and easier to maintain, right? Wrong. Trick question! Turns out your code can’t be exported out of the country due to proprietary encryption technology. And that’s never going to change. So, unless your country suddenly becomes multilingual, you ain’t gonna need it. Reduce.

Sometimes you don’t have the luxury of getting rid of things early on. Luckily, it’s possible when you’re trying to clean up your code though, too. Remember that complicated hierarchy you put into the code because you thought you’d be getting clients with more complex needs? Well, it’s a year into the project and you’ve been dealing with headaches from that model for the entire time, and it turned out Fred figured out a way to transform any inputs into the simple format you’ve been using all along. All that wasted effort for nothing. Don’t worry; it was a good thought. But now it’s time to simplify. Get rid of that bug-ridden complexity and patch up what you can. Then ta-da! You’ve successfully reduced. Feel free to relax.

If we embrace the three Rs with our code as we do with the environment, we can reap the same benefits. Reduce, reuse, recycle. It’s never too late to take action.

The Stray Space

Mon, 29 Mar 2010 10:44:00 -0400

My task was refactoring. That is to say, I wasn’t being thrown into the code to make it pretty. I wasn’t just doing superficial things like fixing the names of variables and adding comments. I was making a real change! You see, a bunch of different Python modules all ran separately, and they all had a bunch of slightly different threading code that was essentially the same. So, we got the idea to factor out that functionality, and create one super-elegant abstract class that the separate modules’ classes can use. For kicks, we figured we’d make it so we can run them all at once from a common class.

Sure! Sounds good. Nothing bad I can say about this. I get to put on my smart developer hat, take this crappy code, and make it good.

Then came a suggestion:

While we’re in the code, maybe we should bring it up to PEP 8 compliance!

I thought it was a pretty good idea at the time. Yes! Let’s do that! Then our code will be even better, right? Readable to the max!

The guidelines set out in PEP 8 are actually pretty reasonable. I’ve found that most Python code I’ve written has naturally fallen into the guidelines, with a few notable exceptions, like a habit of writing variable names in mixedCase as opposed to lower_case_with_underscores. Most of PEP 8 is pretty standard stuff, like avoiding extra spaces around parentheses and brackets. Those sort of things.

It’s short too. I mean, it’s more than just a few paragraphs, but it’s no Chicago Manual of Style. You could read it while you ate your sandwich at lunch. Brackets on rye. Mmm.

So what’s my problem with PEP 8?

Nothing. No, really; no problems here. It’s a good style guide. It’s certainly a good standard to aspire to. When I was going through our code and working on that refactoring, all new code I wrote was done in the correct style. All the existing code in those modules was brought up to snuff. I even endeavored to make most of the lines less than 80 characters long, even though my editor defaults to something like 125. Because that’s what you’re supposed to do!

When I finished, these Python modules were beautiful and immaculate: fully PEP 8 compliant. They’ll be on display in the Museum of Modern Art this spring. I crafted the final commit message and sent it up to the repository, working perfectly and looking sharp.

But those modules were the only Python code that we’d decided to fix up in terms of PEP 8. Everything else was to be left alone. I was disappointed, but I understood. I tried to get on with my life. But having spent a few days in the land of beautiful code, I noticed that coworkers’ checkins started to seem a little… off.

Someone’s tuple had no spaces after the commas. Someone else’s dictionary-ending curly brace was on the next line. One might say that it looked better that way, but in my heart I knew they were wrong. After all, my PEP 8 checker said so! Each violation of the style guide was like a shot through my chest. These poor, misguided coworkers. They just don’t know any better. I have to teach them somehow. I know! I’ll just go fix their code for them!

And I did. A few times I went to a file to make a tiny one-line bugfix, and ended up with a diff that scrolled off the screen. The meaningful change was buried in a sea of whitespace changes. But that was okay, I told myself, because now that file was tasteful and aesthetically pleasing. It was the right thing to do, right?

I suppose I realized that something was wrong after reviewing one particular changeset. One developer had checked in a one-liner: they added an explanatory comment between two lines of code. But it just so happened that the hash mark that began the comment was indented one extra space. A stray space had wiggled its way into a commit by accident.

No, I thought. No, no, no. I couldn’t deal with that extra space. It was so totally off! It broke the visual flow of the code!

But I knew I couldn’t just check in a change to get rid of that space. It wasn’t even on a line of real code!

Hmm. Time for a little introspection. Let’s maybe do an exercise with whys.

I’m very unhappy with these circumstances.

Why?

I can’t make the change to remove the extra space.

Why?

People would think it’s weird to be bothered by such a nitpick.

Why?

Because most people don’t care very much about those small bits of style.

Why?

Because such small style nitpicks don’t affect how the code works or even its readability.

Ah-ha! An epiphany! It turns out that the reason is simple: once you get past a certain point, code style just doesn’t matter. The purpose of code style guidelines is to make sure that everyone can read and understand a common codebase. To be so strict with adherence to guidelines, as I had been, is ridiculous. It’s even more so for a language like Python, which reads like pseudocode no matter how sloppily you write it.

We developers love to get caught up in the little things like code style. But the right way to go about it is not to tape the guidelines above your monitor with rays of light made of yellow construction paper surrounding it. Your style guide is not the final word. Just read the thing and get on with writing code. Your fellow developers will thank you.