Stavros' Stuff Latest Posts

A schemaless layer over SQLite

Sat, 18 Feb 2012 21:19:00 +0000

I've been hard at work these days, and to relax I decided to develop a small application/side project. As with many applications, I needed a reliable, persistent store, so I went with the most reliable, easy to set up, performant, and generally awesome store I knew: SQLite.

However, since this was a very experimental prototype, I got tired of the frequent schema changes and wished I had something that didn't dictate a schema, but that I could still query quickly. MongoDB is, theoretically, a very good fit for this, but I'm not very fond of it due to my previous bad experiences. I'm not sure how far along it's come, but it's not my first choice these days, and I figured I could very easily hack a schemaless layer over SQLite.

After pontificating for a bit, I realized I could just store my data (any sort of arbitrary data) in a blob column, and not care about what's in it. Querying would be trivial by loading each and every field from the DB and checking if all the properties are there, but this would be very slow. If, however, I dump the fields I care about (and their values) on another table, then I could index them and query them very easily and quickly, and an INNER JOIN between the index and data tables would give me the actual data blob very quickly. Data that didn't have one of the properties searched for wouldn't match anyway, so they could be omitted from the indexes.

After starting to work on this, my good friend Iain showed me a post that mentioned that this is the exact method FriendFeed used over MySQL to turn their MySQL instance schemaless, for easier creation of indexes, updating, etc. Postgres, of course, has the hstore, so it doesn't need any of this hackery.

One day later, I had something that works quite well, and I call it goatfish. It's small, fast, very easy to use (just have your data class inherit from goatfish.Model, and you're set), and, since it uses SQLite, it's embeddable.

Its documentation can be found at github pages, and it works reasonably well for rapid prototyping. If anyone would like to contribute, I would be grateful if you added tests (tests that fail when they shouldn't are a bonus), and generally any sort of feedback/request is appreciated.

I hope you find goatfish useful!

Winning at Puzzle Adventures

Wed, 08 Feb 2012 21:09:41 +0000

A few weeks ago, some friends of mine started playing a Facebook game, Puzzle Adventures. This is a (pretty fun) jigsaw puzzle game, so I played a few levels at some point. The game limits you significantly in many areas (e.g. you can only play three or four games before you have to wait for minutes to play more, or pay). The game is also geared against you in that the time limits are pretty tight, but what irked me the most was that the puzzle pieces don't snap together unless you get them really really close to each other, which is just a tactic for you to lose time, and isn't fair.

Another fact that really annoys me is that I suck at jigsaw puzzle games, so my friends were beating me pretty handily. However, I am pretty handy with them computers, so I figured I'd have a look at the Flash binary and see if I can change anything to my benefit.

My exploration of the game's internals follows.

Examining the server calls

After a short look at the server responses, it became clear that that wasn't a way to go. Although the responses were trivial to change, the game synced everything to the server before and after every level, and if something wasn't proper, the game produced an error. For example, I couldn't give myself more hearts or boosts or anything, as the game told the server about this after the level, and the server knew I wasn't supposed to have any hearts, so it didn't count the level.

Not only that, but the game sent the actual moves in the level to the server, so that was another place where attack wasn't feasible. I decided that it would be much simpler to change a local game mechanic instead.

Decompiling the flash file

Luckily, a friend of mine does some Flash work, and he owns a copy of Sothink Flash decompiler, a most excellent piece of software. I got a copy of the source code and started looking around for things I could do. I decided that changing the annoying snapping distance would be the easiest and most reasonable thing to do, so I searched for "snap":

Sure enough, the constant SNAPPING_DISTANCE_MUTIPLIER above looks to be exactly what I need. Now, all I need to do is change it, which should be simple enough!

Apparently, it's not that simple. Compiling the Flash binary again from the decompiled sources rarely succeeds (or so my friend told me), and, at any rate, I don't own a copy of Flash, nor is it even available for Linux. The only thing that can be done is finding the constant in the Flash file and editing it with a hex editor.

Finding the constant

Editing the constant should also be pretty straightforward, all we need to do is find potential locations of the number "1.15" above and change it. However, it turns out that some flash binaries (including, unfortunately, the one I'm analyzing) are compressed, so trying to find the constant in the compressed binary will not yield any useful results. I need to decompress the file.

Decompressing the binary

As it turns out, Flash binaries are compressed very simply. All one needs to do is read the magic bytes ("CWS", if the file is compressed) and the next 5 bytes, and then decompress the rest with zlib. Here's a very simple Python script that decompresses compressed Flash files:

import zlib infile = open("test.swf", "rb") infile.seek(0, 0) if infile.read(3) == "CWS": data = "FWS" + infile.read(5) + zlib.decompress(infile.read()) open("decompressed.swf", "wb").write(data)

Our Flash binary is now decompressed, and we can look for the number.

Finding the constant (again)

Great, we are now ready to find the constant, but what's the representation of a decimal in a Flash binary? Well, as it turns out, it's stored as a little-endian IEEE 754 64-bit float (or, double).

Converting 1.15 to double, we get "333333333333c33f" in hex. Using the excellent Bless hex editor, and we can change it to, say, 2, which is "0000000000000040" and save. All done! To make sure, I can send the binary to my friend to decompile again, but I was pretty sure that was the right place so I skipped that step.

Is that all? Not quite, because now we need to override the remote swf with the local, edited one.

Replacing the remote swf with the local one

To do that, I used the excellent debugging proxy, Charles. After setting it up and loading the game, Charles shows the swf request, at which point we can override it by right-clicking on it and clicking "map to local". Selecting the file, all is done!

Launching the replaced game, everything works normally, which is always a good sign. Playing a level, I see that pieces snap to each other from a fairly larger distance, so the change works! Finally, I can play without the trackpad messing my game up, and the game is now much fairer and more enjoyable.

And that's how you win at Puzzle Adventures!

How to handle Python package installation on GAE

Fri, 03 Feb 2012 22:17:22 +0000

As I've mentioned many times on this blog, I'm a big fan of Google AppEngine. I like the free tier, the effortless scaling, the zero administration, and how I can run Django almost unmodified and deploy in seconds. It has its downsides, but overall I like it very much.

The one part of deployment I didn't like was dependency handling. Due to the way GAE works, you need to install all packages in the root of your application, or in some other directory by placing the source there. I haven't been able to find a way to automate this, because pip doesn't handle this exact workflow (I haven't tried buildout because it confused me a bit, so I don't know if it does this), and it was quite a hassle to get all the necessary libraries every time.

To this end, I opened an issue on the pip bug tracker, and a few short hours and some small patches on my part, along with some help from Carl Meyer, pip now supports the "--target" switch.

What the "--target" switch does is download all the packages you specify, either from the command line, a textfile, or other location (as usual with pip) and extract them in the specified directory. So, to download install all your dependencies in a certain directory, all you need to do is:

pip install -t libs shortuuid django-annoying django-debug toolbar

and the three packages will be downloaded and placed in the "libs" directory.

This feature should make it to the next pip release (or you can try it now by downloading the latest head), and I'm very psyched about it. Hopefully there will be no more deployment headaches with GAE, and things will go much smoother. If you have any other way of solving this particular problem, please let me know in the comments!

On authentication usability

Sun, 18 Dec 2011 02:06:49 +0000

A few days ago, I tried to log into my alternate Facebook account, one I hadn't logged into for a while. After the first or second failed attempt at logging in (I couldn't remember the password), my mail client rang and I saw the following email from Facebook:

This struck me as a fantastic touch and a great usability boost. It also made me think about the usual login flow in sites, and I realized that email is, in my opinion, being underutilized in the login process. There's really no reason we shouldn't assume that a user has forgotten their password after three (or so) failed attempts, and proactively send them a password reset email or, even better, an instant login link.

Instead, most sites are forcing users through a time-consuming password reset process, usually making them re-enter their email address in the "forgot my password" box, even though the user has already offered it multiple times. The usability of the typical login workflow is really rather bad, and almost nobody takes the obvious steps to improve it.

Experimenting even further with this, I tried a different way of authenticating with YourPane, a small project I wrote when I wanted to easily share links with specific friends. YourPane doesn't use a username and password combination at all. Instead, you enter your email in the box, and it sends you an instant login URL (which you can save and use whenever you like).

That experiment has not been a tremendous success, as people expect the usual username/password boxes, or some sort of "connect with X" button. This doesn't mean that we can't ensure that the users' lives are made easier by anticipating and acting on some common scenarios, though. For example, most services that require email verification of the email address don't log you in automatically after you've clicked on the verification link, making you do the (useless) extra step.

The time when we were afraid of storing state for trivial things (such as the number of failed login attempts per user) because of performance considerations is long gone. We have redis now, and it's trivial to store that information for any number of users, as well as make it automatically expire after a few minutes.

I would like to see this behavior become the standard in authentication workflows, as well as some sort of "instant login" that the user can request to log in that one time. There are many services many of us use infrequently enough that going through the trouble of resetting the password isn't really worth it, especially since most services usually make you log in right after you reset your password!

Unfortunately, these patterns are so entrenched and ubiquitous that it's hard to see that they could be improved, even when a few things are obvious, when you know to look for them. We should challenge our standard workflows and look for ways to make them easier to use, even if we're doing it the way everyone else is.

A short update

Wed, 12 Oct 2011 14:20:08 +0000

As I have mentioned previously, this blog is running on Google's AppEngine, and I have just migrated it to the HR datastore to test out Python 2.7. Hopefully this will go well and the concurrent requests will keep the instances to a minimum. I am curious to see how well this will perform.

I am especially interested in the change of instance hours between the old and new backends. Based on a quick examination of the billing history of the past 16 days, the blog has used an average of 30 instance hours a day, with a max of 36.05 and a min of 26.55. I'll update this post in a few days with the new numbers. If they aren't significantly lower (well, 24 across the board, at least), I will be severely disappointed.

That's all for now!

P.S. After leaving it sit for a few days, the site is pretty much at 24 hours every day now. The most I've seen the past week is 24.03, so I think this change has been a great improvement for GAE.