andrew makes things

Archive a PDF of your Posterous blog

2013-03-12T23:23:00-07:00

My wife and I had a private travel blog on Posterous. Unfortunately, Posterous got aquihired by Twitter and is shutting down, so I spent a few minutes figuring out how to save a PDF of the blog. Chrome and Firefox did pretty poorly at saving a decent looking PDF for my long blog so I installed wkhtmltopdf and made one myself. Here’s how.

Install wkhtmltopdf, then run:

wkhtmltopdf --enable-plugins --margin-bottom 0 --margin-top 0 --margin-left 0 --margin-right 0 "http://yourblog.com" archive.pdf

If your blog has a password, log in with a browser and copy the cookie. Firefox makes this easy in the Developer Toolbar by entering cookie list.

wkhtmltopdf --cookie cookiename cookievalue ...

Finally, if you’re on a Mac and used the pre-built wkhtmltopdf disk image, you can still use the command line.

/path/to/wkhtmltopdf.app/Contents/MacOS/wkhtmltopdf ...

Command Line Accounting with Ledger and Reckon, an update

2013-02-16T14:08:00-08:00

I’ve been using ledger, combined with a custom Ruby gem called reckon, to balance my small business’s accounts for the last few years. The command line, Bayesian statistics, and Double Entry Accounting! What could be better? Here’s how I do it.

First, I export the year’s transaction history from Chase (in my case) and save it as a CSV file called chase-2012.csv. It looks something like this:

Type,Post Date,Description,Amount
DEBIT,12/31/2012,"ODESK***BAL-27DEC12 650-12345 CA           12/28",-123.45
DEBIT,12/24/2012,"ODESK***BAL-20DEC12 650-12345 CA           12/21",-123.45
DEBIT,12/24/2012,"GH *GITHUB.COM     FP 12345 CA        12/23",-12.00

Then, I make a new ledger file called 2012.dat and start it with:

2012/01/01 * Checking
    Assets:Bank:Checking            $10,000.00
    Equity:Opening Balances

Where the $10,000.00 is the hypothetical starting balance of my bank account on the first day of 2012. Since I’ve been using ledger, this is just the balance of the account from the summary that I generated at the end of 2011.

Now, I run reckon, initially with the -p option to see its analysis of the CSV file:

> reckon -f chase-2012.csv -v -p --contains-header

What is the account name of this bank account in Ledger? |Assets:Bank:Checking| 
I didn't find a high-likelyhood money column, but I'm taking my best guess with column 4.
+------------+------------+----------------------------------------------------------+
| Date       | Amount     | Description                                              |
+------------+------------+----------------------------------------------------------+
| ...        | ...        | ...                                                      |
| 2012/12/24 | -$12.00    | DEBIT; GH *GITHUB.COM FP 12345 CA 12/23                  |
| 2012/12/24 | -$123.45   | DEBIT; ODESK***BAL-20DEC12 650-12345 CA 12/21            |
| 2012/12/31 | -$123.45   | DEBIT; ODESK***BAL-27DEC12 650-12345 CA 12/28            |
+------------+------------+----------------------------------------------------------+

It looks like reckon has guessed the correct columns from the CSV, so now I run it in “learning” mode. It loads in my data from 2011 and uses it to guess at labels for my 2012 data, using a simple Naive Bayes classifier.

> reckon -f chase-2012.csv -v -o 2012.dat -l 2011.dat --contains-header

...
+------------+---------+------------------------------------------------+
| 2012/12/24 | -$12.00 | DEBIT; GH *GITHUB.COM FP 12345 CA 12/23        |
+------------+---------+------------------------------------------------+
To which account did this money go? ([account]/[q]uit/[s]kip) |Expenses:Web Hosting:Github| 
+------------+----------+-------------------------------------------------+
| 2012/12/24 | -$123.45 | DEBIT; ODESK***BAL-20DEC12 650-12345 CA 12/21   |
+------------+----------+-------------------------------------------------+
To which account did this money go? ([account]/[q]uit/[s]kip) |Expenses:Programming| 
+------------+----------+-------------------------------------------------+
| 2012/12/31 | -$123.45 | DEBIT; ODESK***BAL-27DEC12 650-12345 CA 12/28   |
+------------+----------+-------------------------------------------------+
To which account did this money go? ([account]/[q]uit/[s]kip) |Expenses:Programming|

In each of these cases, the Bayesian classifier correctly guessed the appropriate label for these expenses based on last year’s data.

Now, with a fully-updated 2012.dat file, I run it through ledger and get the following hypothetical results:

> ledger -f 2012.dat -s bal

    $20,000   Assets:Bank:Checking
   $-10,000   Equity:Opening Balances
    $258.90   Expenses
    $246.90     Programming
     $12.00     Web Hosting
     $12.00       Github
   $-10,258.90  Income
   $-10,258.90    Some source of income that makes this math work

I like to do all of this work inside of my Dropbox folder in case I delete or overwrite a file by mistake.

Want to do all of this yourself? Start by visiting ledger-cli.org, or by installing ledger with homebrew:

> brew install ledger

Then install reckon:

> (sudo) gem install reckon

Have fun!

Reckon is available on GitHub, and, for updates, you should follow me on Twitter.

Running Ruby inside of Ruby (in the best way ever)

2013-01-01T13:39:00-08:00

There are no good Ruby sandboxing options right now. You can sort of use $SAFE levels and taint checking, you can sort of use Shikashi, you can use the secure gem to run in a separate process, and you can, with much care, use chrooted and jailed virtual machines or Linux containers. None of these options met my exacting standards, meaning they’re not ridiculous. Therefore, I’m introducing…

RubyOnRuby!

An unholy amalgam of therubyracer’s V8 engine and emscripted-ruby to allow a truly sandboxed Ruby-on-Ruby environment.

Check it out on GitHub!

Make an expanding text UI with jQuery Expando

2012-09-21T20:34:00-07:00

RecentlyRecently, after receiving a couple of requestsfriendly requests, I extracted the codejQuery code powering the UIexpanding text UI of andrewcantino.com. You can get itdownload or fork the source on GitHub: https://github.com/cantino/expando

Compressing Code

2012-06-15T21:15:00-07:00

What can we learn about a code base or a language based on its compressibility? My pet theory is that less compressible code will be, on average, better code, because less compressible code implies more factoring, more reuse, and fewer repetitions.

Larger numbers (longer lines) indicate more compressibility.

Below are the compressibility results for some popular libraries and languages. To generate this data, I downloaded each package or library, extracted the source files, removed comments, and gzipped the files with maximum compression. The numbers represent the ratio of uncompressed source file size to compressed source file size. Smaller ratios imply less compressible code.

Unsurprisingly, Java was the most compressible language- all that boilerplate! Python was the least compressible language, with Java, on average, being about twice as compressible. I was somewhat surprised that JavaScript was the second best language in terms of incompressibility.

It’s probably disingenuous to draw strong conclusions from these results, but I still find them intriguing. What, if anything, do you think they mean?

View data for:

Aside: Compression itself is a fascinating subject. Being able to compress something is fundamental to being able to understanding it. If you can rewrite a 2 page document into 2 paragraphs, while still expressing its core ideas, you’ve deeply understood the material. Hence the existence of the Hutter Prize, a standing challenge in Artificial Intelligence to further compress a corpus of English text. Hence, also, the existence of specialized image compression algorithms that compress human faces better than anything else because they understand, in software, what a human face generally looks like.

If I can compress an image of your face, I can probably also recognize it. Imagine that I have thousands of photos of faces. Using some linear algebra and creative encodings, I can figure out the commonalities and differences among these faces. Basically, I can derive a set of common noses, a set of common eyes, a set of common brows… and, given a new face photo, I can compute a mixture of these common attributes for the new face. Perhaps it, roughly speaking, has 60% of Common Nose 6 and 40% of Common Nose 12. Well, then I can represent the picture of this new nose as roughly two numbers, the “amounts” of Nose 6 and of Nose 12, and suddenly I’ve compressed a collection of hundreds or thousands of pixels- a photo of a nose- into just two numbers. We could also go in reverse, taking a photo, calculating the percentages of different common features, and then looking up in a database to see who we know whose face expresses those same feature percentages. We can compress, and, thus, we can recognize. (Interested in this? See Eigenfaces as well as PCA and ICA for face analysis.)

My experiences with personal outsourcing

2012-06-10T16:42:00-07:00

Over the last few years I’ve been experimenting with outsourcing. I’ve done this both for personal and business projects. In the personal domain, some people call this “lifesourcing”: the practice of modularizing and outsourcing parts of your life that you don’t enjoy so that you can maximize the parts that you do. It’s outsourcing (with many of the same pros and cons), but for your personal life.

A growing number of sites have popped up recently to facilitate lifesourcing, and while these sites aren’t strictly needed- you can still find skilled people to help you on Craigslist, for example- they make this sort of outsourcing even easier.

I’d like to talk about some tips and tricks, but first, let me list a few of the things that I have outsourced over the last couple of years.

Personal Things

My wife and I outsourced hand written, cursive wedding invitations on TaskRabbit. (My mother-in-law preferred them to be hand written, my hand writing stinks, and my wife didn’t have the time.)
Also for our wedding, someone on Fiverr polished our save-the-date photo.
After the wedding, we hired a wonderful, well-traveled woman on TaskRabbit to help plan our honeymoon in South America.
We paid a TaskRabbit to scan all of our wedding cards for posterity.
Currently, we have a virtual assistant from oDesk who helps us with the ocasional life task. She has proofread documents and called gyms around San Francisco, looking for ones with good pools.
We’ve given friends custom, hand painted watercolor birthday cards from Fiverr and custom wedding presents from Etsy. (Which, one couple swears, is their favorite wedding present!) We designed the art (roughly) and then it was made with skill by the artists.
When I have outstanding questions, I turn to Aardvark, Yahoo! Answers, and other outsourced question answering services. I would gladly pay for a better one.
A carpenter on Craigslist designed and built a custom, adjustable standing desk for me. (Arguably outsourcing, arguably not.)

(Micro-)business Things

I’ve hired artists and logo designers on Fiverr. (I actually bought some excellent artwork and had an ongoing business relationship with an artist who I found for $5 on Fiverr.)
Workers on oDesk and TaskRabbit have helped me brainstorm domain names.
I’ve paid users on Mechanical Turk and later on oDesk to label data for me for some Machine Learning research.
I’ve brainstormed with a worker from Coffee & Power.
oDeskers have also written software, done graphics, wrote blogs, sent emails, maintained website communities, and researched ideas for me.

Okay, so clearly I’ve experimented with this a fair bit. Here are some of the things that I’ve learned:

Outsourcers take maintenance. It only makes sense to outsource something that would take you more than some threshold amount of time to do yourself. Repetitive tasks are great candidates.
Be creative- what could you do, if only you had the time?
Outsource things you’re bad at, or simply hate doing.
Art is a great thing to outsource. Finding someone’s work that you like on Etsy is fun and addictive and custom gifts make a lasting impression, often costing the same as something far more mundane.
If you’re trying to get art off of Fiverr, I recommend contacting 5-10 different providers, having them all do the work for $5 each, and then continuing to work with your favorite. This same strategy, of redundant hiring and then consolidating, works well across many lifesourcing and outsourcing domains.
You should think about hiring people on oDesk in the same way as you would any other interview process. Ask to see work, look at portfolios, and, ideally, provide interview challenges that directly map to the work they will be doing for you. In my case, when I hired someone to maintain one of my websites, my interview questions revolved around writing example emails and deciding which links were worth posting. When I hired people to classify a dataset, I gave them access to the real classification application and had them do a sample set. If they did well, I hired them.
If you’re going to go through the trouble of interviewing and hiring on oDesk, I strongly recommend codifying your interview and training instructions as reusable documents. When your current worker(s) inevitably leave or flake out, you can hire and train the next set more quickly. You can also hire more than one at a time for added redundancy.
There are a shockingly large number of people on this planet who speak (nearly) perfect English, have sharp wits, and are looking for work. If you have tasks that you can pay them a fair wage to solve, you’re helping everyone. And remember, a fair wage in the Philippines (where many people speak English perfectly), is significantly less than in the US. Do your cost of living research and pay fairly and generously!
Accountability and incentives are important. I left Mechanical Turk and instead interviewed and hired individuals off oDesk for data labeling tasks because I received better quality and had more consistency over who I was working with.

Are you a lifesourcer or a micro-outsourcer? What have you learned?

Machine Learning Project Ideas

2012-04-22T15:56:00-07:00

Ryan Stout and I are giving a talk at RailsConf about Machine Learning tomorrow. To go along with the talk, here is a list of project ideas to get your creative juices flowing:

A robust email and mailing address typo corrector for web forms.
A Rickroll detecting browser plugin- it warns you before you follow a link that will likely result in Rickrolling. (Rickroll Protection As A Service?)
A per-user clicktrail analyzer that predicts which links a user is most likely to follow, given their history. Use this to highlight or promote high-likelihood links.
A user info and usage pattern analyzer that classifies users by likelihood of upgrading to a premium plan.
A RubyGem for classifying user generated content into appropriate, inappropriate, spam, NSFW, etc.
Along the same lines: a nudity detector for uploaded images.
A RubyGem for code optimization based on the current backtrace, possibly using reinforcement learning. For example:

  # This probabilistically selects a choice based on the
  # current backtrace and the history of reinforcement signals seen.
  optimize do
    choice do
      # some code path that ultimately triggers a "reward" or "punishment" signal
    end
    choice do
      # some other code path
    end
  end

A story karma predictor that estimates the final score on Hacker News of any article, based on textual content and the poster’s info.
A system that classifies support requests by their estimated severity.
Make things easier for your users:
- given them default settings selected by users similar to themselves
- default to pages they use often; expand modules they interact with frequently

Once you know what’s possible, it’s hard to find a project that wouldn’t benefit from some machine learning.

Have other ideas? Want to discuss these? Post them in the comments and follow @tectonic for updates.

Fixing the Chrome background refresh bug

2012-02-15T11:55:00-08:00

There is a bug in the current version of Chromium (hence Google Chrome) that sometimes fails to redraw CSS background images when they’re hidden and then re-shown. This issue appeared on Mavenlink’s Tour page. Thomas Fuchs discusses some possible solutions, but none of those worked for us. Here is our ugly solution:

Please let me know if you find something better!

Hacking Google for fun and profit

2011-12-14T20:37:00-08:00

At the end of last year, Google announced their Vulnerability Reward Program which rewards security researchers for reported security and privacy holes in Google properties. This sounded like an interesting challenge, and I set out to find security holes. I found three, got paid, and am now in the Google Security Hall of Fame. All in all, a rewarding experience.

Below I describe the three security holes that I found.

Determining if a user has emailed another user

In my opinion, this is the most subtle, but also the most disturbing, of the three bugs. As with the other bugs that I found, this was an example of Cross Site Request Forgery- the practice of convincing a user’s browser to make a request on their behalf to a remote server. This type of attack generally only works when the user is logged in to the remote service. In this case, if a user is already logged into Gmail (and they usually are), a malicious website could make a series of requests for Gmail profile images and, based on the return codes, determine whether or not the visitor had communicated with another Gmail user. This worked because Gmail, as a well-intentioned privacy measure, would only show profile images to a viewer if they had had mutual contact. Here is some example code that worked at the time:

checkUsername

function checkUsername(username, callback) {
    var image = new Image();
    image.onload = function() {
          callback(true);
    };
    image.onerror = function() {
      callback(false);
    };
    image.src = "https://mail.google.com/mail/photos/" + username + "%40gmail.com?1&rp=1&pld=1&r=" + (new Date()).getTime();
}

checkUsername("fbi-reports", function(hasEmailed) {
  alert("The current visitor " + (hasEmailed ? "has" : "has not") + " emailed the FBI.");
});

checkUsername("wikileaks", function(hasEmailed) {
  alert("The current visitor " + (hasEmailed ? "has" : "has not") + " emailed WikiLeaks.");

It should be clear why this is a serious privacy concern. If you suspected someone of being a whistleblower, for example, you could make a page that probed a bunch of revealing email addresses and checked to see if any had been contacted. Luckily, Google reports that they have now fixed this bug. Cross Site Request Forgery attacks can usually be prevented by adding a CSRF token (a unique and user-specific token) to every request.

Identification of a user’s Gmail address

This bug would have allowed a malicious website to determine your Google username if you were simultaneously logged into your Google account and typed anything into a seemingly innocuous web form. One of the fields in the form would actually be an iframe pointing to a public Google Document. When the user typed into the field, they would really be entering text into the Google Document, and what appeared to be their cursor in the field would actually be the Google Document insertion point. When a user typed into the field, the attacker could determine their username (and hence email address) by observing the publicly-displayed list of current document editors.

Again, this is a type of Cross Site Request Forgery, specifically known as Clickjacking, which can be especially hard to prevent. There are many types of Clickjacking, almost all of which use iframes. One approach, which I used here, is to artfully display content from a target site in such a way as to look like it’s part of the current page. Another approach is to hide the iframe invisibly under the user’s cursor, moving it as the cursor moves, and causing the user to click on the other site without realizing it.

Google correctly used the X-XSS-Protection and X-Frame-Options headers, but some browsers do not honor these. The solution to this one is tricky, but it is generally to use frame busting, to provide appropriate headers, to use CSRF tokens, and to not expose any user information without a direct user interaction.

Deletion of all future email

The third bug that I found was a fairly severe security hole that affected a portion of Gmail users. Due to a missing CSRF token during the first step of the filter creation flow in the HTML-only version of Gmail, a malicious site could trick visitors into creating a Gmail filter that would delete all future received email. This worked in the current (at the time) version of Firefox, but not in Chrome or Safari due to their correct handling of the x-frame-options header. I didn’t test it in IE.

This security hole was exploitable via a combination of a classic Cross Site Request Forgery with a Clickjacking attack. First, I discovered that it was possible to submit the first part of the filter creation flow in an iframe using JavaScript because Google had forgotten to include a unique CSRF token in the form.

<form id='form' method='POST' target='iframe' action='https://mail.google.com/mail/h/ignored/?v=prf' enctype='multipart/form-data'>
    <input type=hidden name='cf1_hasnot' value='adfkjhsdf'>
    <input type=hidden name='s' value='z'>
    <input type=hidden name='cf2_tr' value='true'>
    <input type=hidden name='cf1_attach' value='false'>
    <input type=hidden name='nvp_bu_nxsb' value='Next Step'>
    <input type='submit' style='display: none'>
</form>

I then positioned the iframe such that the “Create Filter” button on the subsequent page would fill the frame without showing the button border; only the word “Create” was visible. A fake button was then shown around the iframe with a style that matched the gmail style such that when the user believed they were submitting a form with a submit button entitled “Create,” they were really creating a malicious and destructive filter in Gmail.

Google says this has now been fixed.

Google’s Response

In all three cases, Google responded promptly to my security report and fixed the bug within a reasonable amount of time. I was given two $500 awards for the three bugs. Google generously doubled these amounts when I chose to donate them to charity, so the Athens Conservency and the Buckeye Forest Council, two of my favorite local charities in Athens, OH, received one thousand dollars each, care of Google.

These were subtle bugs. They took trial and error to find. However, in total, I only spent a few spare evenings of my time. If Google’s products- some of the most secure in the world- are susceptible to these sorts of attacks, you can bet many others are as well. Every programer makes these mistakes sometimes. Security is too complicated for anyone to get right all of the time. Check your code!

Take your security into your own hands… or, why you should hack Google too!

Many companies try to silence security bug reporters through legal threats and sometimes even action, driving discoverers of bugs underground and onto the black market where such knowledge can do real harm. Google has set an admirable example by creating a program that is enlightened, responsive, and well-run, and I hope other companies move in the same direction.

I had a great time using jsFiddle to explore and demonstrate bugs. You can do the same– check out their guidelines and do your part to improve the security of products that you love.

Enjoyed this post? You should follow me on Twitter.

How to make your Rails app tweet the Twitter

2011-05-12T17:35:00-07:00

Suppose you want to build a Rails application for tracking popular links, and you want it to post the most popular links to Twitter automatically. This quick tutorial will show you how to do that using the newest version of the Ruby twitter gem. A little while ago I added the ability for Freebies Finder to tweet popular freebies. I recently had to do this for another site and decided that a tutorial was in order.

Setup the accounts

We’ll pretend that our website is called AwesomeLinks.com. Signup for two Twitter accounts, AwesomeLinks and AwesomeLinksDev. We need to create a Twitter application through which our website can post to these accounts. Do this by logging into AwesomeLinks and visiting https://dev.twitter.com/apps/new. Select ‘Client’ as the Application Type, skip the Callback URL, and select Read & Write access. Twitter will give you a OAuth Consumer key and secret, which you will soon need.

Getting your OAuth Token and Secret

Now you need to authorize your new Twitter Application to post on both of your Twitter accounts. For this, we use a script:

(If you used the Twitter gem in the past, you may have used authorize_from_access for this, but that no longer works. We now have to require and use oauth separately.)

Fill in your Twitter Application’s Consumer key and secret and run the script. You will be prompted to visit a URL and then to enter the PIN that Twitter provides. Do this for both of your new Twitter accounts and record the results in config/twitter.yml, like so:

Initializing the Twitter gem

Now, create an initializer in config/initializers/twitter.rb and again include your Twitter App’s key and secret:

(If you want to Tweet to multiple accounts, you can do this differently and instead make separate Twitter::Client objects, each with their own OAuth tokens.)

Tweeting the Twitter

Finally, it’s time to augment our Link model so that it can send tweets. I decided to have it tweet the link’s description and a shortened version of its URL using the following code:

The rest is up to you. You could write a cronjob to automatically call tweet! on every link, or only on those with enough popularity. Have fun!

Why plug computers are a security nightmare

2011-02-28T18:03:00-08:00

The increasing availability of low profile “wall-wart” plug computers like the SheevaPlug can be viewed as an emerging threat to physical network security. For $99, a budding industrial espionagist could buy the SheevaPlug developer kit or the consumer TonidoPlug, install some easily-available network intrusion testing software, and illicitly “test” the security of a competitor’s network.

While many of these techniques have been known for a while, the low form factor of plug computers and consumer netbooks, coupled with their rapidly decreasing price, could enable disposable intrusion tools and open new avenues for attack. The current $99 SheevaPlug has no wireless capability and limited storage, but the manufacture has just announced an expanded model with wifi, bluetooth, and an internal hard drive. Even without these advances, current generation plug computers can easily be expanded with a USB memory stick or external USB hard drive, USB wireless interface, and more. For little more than $100 one could make a practically undetectable wireless bug that can be deployed in seconds.

In fact, soon you may be able to just buy an all-in-one penetration plug computer, the PlugBot.

edit: as was pointed out to me after posting this article, the described device already exists and can tunnel out over 3G. http://pwnieexpress.com/pwnplug3g.html

With two wireless adapters and some simple software, a plug computer becomes a wireless bridge capable of automatically cracking wireless networks in range (both WEP and WPA are vulnerable these days; see aircrack). Most locations have multiple 3rd party networks overlapping their physical space, which, if cracked, could be used as back channels for the plug computer to phone home. The attacker could then tunnel into the company network undetected and completely bypass the company’s external defenses by routing through an available 3rd party wireless network. From the perspective of the attacked network, even if the intrusion is noticed, it appears to come from within their own physical space.

A number of other uses come to mind for such devices:

Passive sniffing of internal network traffic using dsniff and sending it back to an attacker. Many networks aren’t sufficiently secured once you’re past the perimeter firewalls.
Physically connect two ethernet interfaces and use the plug computer as a man-in-the-middle proxy to sniff all traffic entering and leaving a workstation.
Attach a camera or other sensor payload and use as an over-the-internet video bug.

Again, much of this has been possible for a while, but form factor is everything. Also, I haven’t seen people talking about the possibility of bridging multiple available wireless networks together for attack obfuscation and to avoid connecting through a company’s edge network. I don’t think companies pay enough attention to passive physical monitoring and intrusion threats like this, especially given the insecurity of wireless encryption standards. What do you think?

How I do command-line accounting: Ledger and Reckon

2010-11-06T18:03:00-07:00

Ledger is a powerful yet simple double-entry accounting system with a command line interface, making it perfect for those of us who prefer our text editor and flat files over any confusing and inflexible accounting program.

I’ve been using Ledger for about a year now, including for balancing last year’s budget. In the process, I’ve needed both to import CSV files of financial data from various sources and to label that data with the most appropriate account. To do this, I’ve written a Ruby gem called Reckon that can usually guess bank data CSV headings, and can also use Bayesian learning to automatically categorize each entry. I think of this as Mint for the command line, but where you don’t have to trust a 3rd party with your bank account passwords.

Getting started with Ledger is pretty easy. I recommend skimming the official documentation (pdf).

I usually start out the year by making a new year.dat file and entering my business bank account’s starting balance:

2010/01/01 * Checking
    Assets:Bank:Checking            $5.000.00
    Equity:Opening Balances

Next, I run Reckon on a CSV dump file from my bank: reckon -f bank.csv -p

If the output looks reasonable, I convert to ledger format and label everything:

$ reckon -f bank.csv -o output.dat

What is the account name of this bank account in Ledger? |Assets:Bank:Checking|

(Here I accept the default because I want to have all money flowing in and out of my bank account.)

+------------+--------+--------------------------+
| 2010/03/13 |  $100.00 | GOOGLE ADSENSE REVENUE |
+------------+--------+--------------------------+

Which account provided this income? ([account]/[q]uit/[s]kip) Income:Adsense

(I decided to call this Income:Adsense.)

+------------+---------+-----------------------------+
| 2010/03/14 | -$10.00 | FEE: INCOMING DOMESTIC WIRE |
+------------+---------+-----------------------------+

To which account did this money go? ([account]/[q]uit/[s]kip) |Income:Adsense| Expenses:Bank Fees

+------------+-----------+-----------------------+
| 2010/03/14 |  $1000.00 | WIRE TRANSFER DEPOSIT |
+------------+-----------+-----------------------+

Which account provided this income? ([account]/[q]uit/[s]kip) |Expenses:Bank Fees| Income:Various Sales

(Reckon guessed Expenses:Bank Fees, but I decided to call this Income:Blogging.)

+------------+-----------+------------------------+
| 2010/03/28 |  $300.00  | GOOGLE ADSENSE REVENUE |
+------------+-----------+------------------------+

Which account provided this income? ([account]/[q]uit/[s]kip) |Income:Adsense|

(Notice how this time it guessed Income:Adsense correctly.)

...

Now, my output.dat file looks something like:

2010/01/01 * Checking
    Assets:Bank:Checking               $5.000.00
    Equity:Opening Balances

2010/03/13    GOOGLE ADSENSE REVENUE
    Assets:Bank:Checking               $100.00
    Income:Adsense                    -$100.00

2010/03/14    FEE: INCOMING DOMESTIC WIRE
    Expenses:Bank Fees                 $10.00
    Assets:Bank:Checking              -$10.00

2010/03/14    WIRE TRANSFER DEPOSIT
    Assets:Bank:Checking               $1000.00
    Income:Blogging                   -$1000.00

2010/03/28    GOOGLE ADSENSE REVENUE
    Assets:Bank:Checking               $300.00
    Income:Adsense                    -$300.00

And if I run Ledger on it, I can get a breakdown:

$ ledger -f output.dat balance

        $6390.00  Assets
       $-5000.00  Equity
          $10.00  Expenses
       $-1400.00  Income

$ ledger -f output.dat equity

    2010/11/06 Opening Balances
    Assets:Bank:Checking                    $6390.00
    Equity:Opening Balances                $-5000.00
    Expenses:Bank Fees                        $10.00
    Income:Adsense                          $-400.00
    Income:Blogging                        $-1000.00
                                               0

Notice that the income is negative because the money flowed from the income source (Google and Blogging) to the destination (my bank account). This is double entry accounting, so the final sum is always zero, as seen in the last row.

Finally, later in the year when there is new financial data, I want Reckon to learn from my existing .dat file, so I do:

reckon -f bank.csv -l 2010.dat -o output.dat

To install Ledger and Reckon, see their Github pages:

Rails RSpec tests are CPU bound

2010-10-26T18:03:00-07:00

Today I experimented with running a large Rails RSpec test suite on a RAM disk. My hope was that by hosting either the MySQL server or the Rails project directory on the RAM disk, the test execution would be significantly increased. If this were the case, I would feel (more) compelled to buy a certain new device with a solid-state disk drive. Unfortunately, while I now have some slick scripts to bring up a RAM disk with either my Rails project or MySQL running on it, the improvements were on the order of 10 seconds over a 10 minute test run (the load time of Rails). Thus, it is clear that these tests are CPU bound, not disk IO bound, and a SSD wouldn’t help.

If you have an SSD, can you corroborate this?

(Tests were performed on a brand new Quad-Core Intel Core i7 iMac.)

Revolutionary

2010-08-22T18:03:00-07:00

Sometime in the late ’80s I convinced my parents to get an external hard drive for our Mac Plus to augment our extravagant two floppy disk drives. We got a 70MB drive for about $400, which is about $500 in today’s dollars. Now you can get 2TB for $100, or 10TB for $500. This is an increase of about 14 million percent in about 20 years! Talk about technological change! I have no doubt that this growth - that storage is now basically free - can enable incredible new technologies. What can we do with this?

Multiple profiles in Chrome on Ubuntu

2010-03-02T18:03:00-08:00

If you’re running Chrome on Ubuntu (likely elsewhere too), you can run multiple copies of Chrome with fully different profiles by launching it with the --user-data-dir option. For example, I run two copies of Chrome like so:

/opt/google/chrome/google-chrome --user-data-dir=~/.config/google-chrome-work &amp;
/opt/google/chrome/google-chrome --user-data-dir=~/.config/google-chrome-personal &amp;

Easy web scraping: Get the title of any URL with YQL

2010-03-02T18:03:00-08:00

This snippet demonstrates how to get the title from any webpage using a simple YQL query and jQuery. The title is fetched from url and is placed in #page_title.

// query: select * from html where url="http://some.url.com" and xpath='//title' var yql_url = "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22" + encodeURIComponent(url) + "%22%20and%0A%20%20%20%20%20%20xpath%3D'%2F%2Ftitle'&format=json&callback=?"; $.getJSON(yql_url, function(json) { if (json && json.query && json.query.results && json.query.results.title) { $('#page_title').html(json.query.results.title); } });

This was very handy for a project I’m working on.

Using TSort in Ruby for Topological Sorting of ActiveRecord Models

2010-01-29T18:03:00-08:00

Recently, I was building a project list application in Rails and I needed to be sure that sub-projects showed up in the list somewhere below their parents.

A topological sort does just what I needed, and I was pleased to discover that Ruby ships with a TSort module. Here is how I extended Array to allow sorting of Projects:

# Ruby's TSort requires that you implement # tsort_each_node and tsort_each_child. I # extend Array so that it knows how to # TSort instances of Project, which has a # child_project_id pointing to another # Project. require 'tsort' class Array include TSort alias tsort_each_node each def tsort_each_child(node, &block) if node.is_a?(Project) [node.child_project].each(&block) if node.child_project? else node.each(&block) if node end end end # Then I can topologically sort a user's projects like so: sorted_projects = current_user.projects.tsort

(Aside: I later post-processed the list to indent sub-projects and position them just below their parents. This post-processing was made easier by the fact that all sub-projects were already somewhere below their parents in the list.)

Replacement for script onload in IE

2008-11-23T18:03:00-08:00

This is an old post from my last blog.

Firefox and Safari support an onload event for SCRIPT elements. That is, you can dynamically add a new script to a page, set its onload event to fire a callback, and know when the script has been successfully loaded. You would want to do this because when you include a bunch of new SCRIPT tags in a page, there are no guarantees in what order the browser will decide to evaluate them, thus making dependencies among the scripts difficult to resolve. Using onload to chain the script additions is one solution to this, however, Internet Explorer doesn’t seem to support onload in SCRIPT tags.

Update: Owen in the comments says: Thanks for your post, but I have since found a more efficient way to do this and thought I might share with you. As you said you can use an onload event in Firefox, but in IE you can use the onreadystatechange event. Works from at least IE6. Haven’t tested earlier.

My solution:

function importJS(src, look_for, onload) {

   var s = document.createElement('script');

   s.setAttribute('type', 'text/javascript');

   s.setAttribute('src', src);

   if (onload) wait_for_script_load(look_for, onload);

   var head = document.getElementsByTagName('head')[0];

   if (head) {

     head.appendChild(s);

   } else {

     document.body.appendChild(s);

   }

 }

 function importCSS(href, look_for, onload) {

   var s = document.createElement('link');

   s.setAttribute('rel', 'stylesheet');

   s.setAttribute('type', 'text/css');

   s.setAttribute('media', 'screen');

   s.setAttribute('href', href);

   if (onload) wait_for_script_load(look_for, onload);

   var head = document.getElementsByTagName('head')[0];

   if (head) {

     head.appendChild(s);

   } else {

     document.body.appendChild(s);

   }

 }

 function wait_for_script_load(look_for, callback) {

   var interval = setInterval(function() {

     if (eval("typeof " + look_for) != 'undefined') {

       clearInterval(interval);

       callback();

     }

   }, 50);

 }

 (function(){

   importCSS('some_stylesheet.css');

   importJS('jquery-1.2.6.js', 'jQuery', function() {

     importJS('some_script_that_uses_jquery.js');

     importJS('another_one.js', 'SomeClassOrVariableSetByTheScript',

       function() {

         importJS('a_script_that_uses_that_class_or_variable.js');

       });

   });

 })();