footle

First Post!

2025-04-14T10:23:44-07:00

I’m building an iOS app that will make it easy to post to any GitHub-based blog (like this one) from iOS/iPadOS. I’ve finally got the basic pieces together to do an actual post from my phone, and this is it!

My next step is to set up a Sharing Extension so you can write your post in any app (like Apple Notes) and just share to post.

Right now the app just supports Jekyll, but it should be easy to adapt to support any GitHub-based workflow.

Screensaver Spelunking

2025-03-21T11:14:00-07:00

Macos Sonoma brought to the Mac the amazing aerial screensavers/wallpapers that can be found on the AppleTV. (I say screensavers/wallpapers because you can choose to have your wallpaper be a frame of whatever your screeensaver is.)

One thing missing, though, are the descriptions of what is being displayed. If I see some amazing place my first question is “where is that?” On the AppleTV you can view the description by clicking (swiping?) up on the remote. There doesn’t seem to be any secret keys or incantations to get it to display on the Mac, though, even though that information is available. If you go into Settings -> Screen Saver on your Mac, you’ll see descriptions:

I figured it couldn’t be that hard to (a) find out what the current screensaver is, (b) look up the description somewhere, and (c) find easy way to show that information. It turned out to be not easy, but I did figure it out.

The Screensavers

Searching for “macos where are aerial screensavers stored” gives /Library/Application Support/com.apple.idleassetsd/Customer/4KSDR240FPS. If you open that directory in Finder (run open /Library/Application\ Support/com.apple.idleassetsd/Customer/4KSDR240FPS from the Terminal), you’ll see they’re all MOV videos. Apparently idleassetsd is the process that handles the downloading and display of the aerial screensavers.

My first thought was that maybe the descriptions were encoded in the video somehow, but after digging around in them a bit, that turned out not to be the case.

The Screensaver Database

I looked in the /Library/Application\ Support/com.apple.idleassetsd/ directory and lo! there was sqlite database: Aerial.sqlite:

sqlite> .tables
ZASSET             ZUSER              Z_1SUBCATEGORIES   Z_METADATA
ZCATEGORY          Z_1CATEGORIES      Z_2ACTIVEFORUSERS  Z_MODELCACHE
ZSUBCATEGORY       Z_1PINNEDFORUSERS  Z_3ACTIVEFORUSERS  Z_PRIMARYKEY

Exploring these tables, we find some information, but nothing terribly descriptive:

sqlite> select ZACCESSIBILITYLABEL from ZASSET limit 3;
Iran and Afghanistan
Caribbean
Africa and the Middle East

Ripgrep to the Rescue

Since I knew the description strings had to be somewhere on my hard drive, the next obvious move was to just search for them. I picked a word from one of the descriptions that was unlikely to be found elsewhere—”Temblor,” from “California’s Temblor Range”—and fired up ripgrep.

> rg Temblor /Library/Application\ Support/com.apple.idleassetsd/
>

Nothing. Hmm. After a bit of head-scratching and poking around in other places, I realized that ripgrep by default doesn’t search binary files. So:

> rg --binary Temblor /Library/Application\ Support/com.apple.idleassetsd/
/Library/Application Support/com.apple.idleassetsd/Customer/TVIdleScreenStrings.bundle/sv.lproj/Localizable.nocache.strings: binary file matches (found "\0" byte around offset 12)

/Library/Application Support/com.apple.idleassetsd/Customer/TVIdleScreenStrings.bundle/es_419.lproj/Localizable.nocache.strings: binary file matches (found "\0" byte around offset 12)

...

Bingo! A strings bundle is a collection of localized strings. Given a key that uniquely identifies a string and a language, you can look up the translation of that string in that language.

The next piece was to find all the string keys. The above ZASSET table has a ZLOCALIZEDNAMEKEY column. But, while poking around in the com.apple.idleassetsd folder, I found an entries.json file that had everything I needed in one file:

> cat "/Library/Application Support/com.apple.idleassetsd/Customer/entries.json" | jq ".assets[0]"
{
  "id": "009BA758-7060-4479-8EE8-FB9B40C8FB97",
  "showInTopLevel": true,
  "shotID": "GMT026_363A_103NC_E1027_KOREA_JAPAN_NIGHT",
  "localizedNameKey": "GMT026_363A_103NC_E1027_KOREA_JAPAN_NIGHT_NAME",
  "accessibilityLabel": "Korea and Japan Night",
  "pointsOfInterest": {
    "60": "GMT026_363A_103NC_E1027_60",
    "150": "GMT026_363A_103NC_E1027_150",
    "0": "GMT026_363A_103NC_E1027_0",
    "32": "GMT026_363A_103NC_E1027_32",
    "195": "GMT026_363A_103NC_E1027_195",
    "22": "GMT026_363A_103NC_E1027_22",
    "110": "GMT026_363A_103NC_E1027_110",
    "260": "GMT026_363A_103NC_E1027_260",
    "180": "GMT026_363A_103NC_E1027_180"
  },
  "previewImage": "https://sylvan.apple.com/itunes-assets/Aerials126/v4/51/ff/08/51ff0824-8da5-78f0-e218-9e61264965bb/Space_Korea_Japan_01@2x.png",
  "includeInShuffle": true,
  "url-4K-SDR-240FPS": "https://sylvan.apple.com/itunes-assets/Aerials116/v4/cb/5b/50/cb5b5035-6701-619f-9065-3d7d0e5fbef4/comp_GMT026_363A_103NC_E1027_KOREA_JAPAN_NIGHT_v18_SDR_PS_20180907_240fps_0d0095d4-5875-4d43-a1a8-7dc915b11b9dq24_sRGB_tsa.mov",
  "subcategories": [
    "61171241-39F3-4ADE-84AA-9CD4EE4A78DA"
  ],
  "preferredOrder": 9,
  "categories": [
    "55B7C95D-CEAF-4FD8-ADEF-F5BC657D8F6D"
  ]
}

It looks like you can even get different descriptions based on how far along you are in the screensaver video.

The Final Boss: Getting the Active Screensaver

The last bit of information I needed was determining the active (or last active) screensaver. I think I spent the most time on this. I was not able to find anything. Eventually I realized that some process had to have that file open, and if so, lsof would find it.

idleassetsd came up empty, so I looked through my process list for other suspects, and found WallpaperVideoExtension:

> lsof -Fn -p $(pgrep WallpaperVideoExtension) | grep ".mov"
n/Library/Application Support/com.apple.idleassetsd/Customer/4KSDR240FPS/B8F204CE-6024-49AB-85F9-7CA2F6DCD226.mov

Boom!

Putting it all Together

I threw together a little Mac menu bar app, WallpaperInfo. Most of the logic is in Wallpaper.swift:

The result:

Sadly, I wouldn’t be able to put this in the App Store because of Apple’s sandboxing rules, but feel free to download the binary from my repo.

It would be nice to have the description on the screensaver itself, like the AppleTV does, but I don’t think that’s possible short of either editing the videos to add descriptions, or running your own version of the screensaver. But given that Apple could easily add this as a feature (please!), I didn’t want to invest too much more time into this. I considered my itch scratched.

Migrating GitHub Pages to Cloudflare

2025-01-14T07:30:00-08:00

Or, On Becoming a Cloudflare Fanboy.

What’s wrong with GitHub Pages?

I’d been using GitHub Pages for hosting static sites, including this one, for years. It’s easy and free-ish (I believe they’re free for public repos, and if you’re paying $4/mo, private repos as well).

One thing you don’t get, though, is any kind of visitor metrics. I know I can get this if I add something like Google Analytics to my sites, but I’m loath to do that for a number of reasons. For one, to be GDPR compliant, I’d have to add one of those annoying cookie banners to all my sites. Also, ad blockers can block the GA Javascript and skew results. Finally, I don’t want to have to add the GA code to every site.

If I were hosting the sites myself, I could get the information I need from the web server logs, but if GitHub is hosting the content, they have that info—but they aren’t sharing it.

Enter Cloudflare Pages

I’d been using Cloudflare for a while, as early on it was the only way to enable SSL on your GitHub Pages sites (GH now provides SSL for all Pages). But I didn’t know they had their own “Pages” offering.

Turns out it’s super easy to use: just point it to your GitHub repo, tell it what branch you want to deploy from and if there’s any build commands you want to run (plus other optional settings like output directory, environment variables,etc.), and it will publish your site on a pages.dev subdomain. Then you can add your own domain name via a CNAME record to that subdomain. If Cloudflare is hosting your DNS it will even update your records for you. And everything gets SSL.

(Another advantage of Cloudflare’s Pages offering is I can easily set up a dev/staging site by just creating another Pages site and pointing it to a branch in my repo.)

The analytics it gives you is decent: total requests, unique visitors, bandwidth, % cached, requests by country/region, etc. You can even see breakdown by source IP, browsers, OS, desktop vs mobile (although oddly it’s under the Security section, not Analytics).

One thing that seems to be missing is a breakdown of traffic by page. It would be nice to see, for example, if particular blog posts are getting more traffic than others, but I’m not seeing that data anywhere. Update: They do have this, in a separate section (they really need a good UX person to clean up their menus), and it has to be enabled per site. If you are using their DNS proxy, it will automatically inject their Javascript into your site. Otherwise you can manually insert their JS snippet. They don’t use cookies, and it doesn’t look like they are exposing IP addresses anywhere, so no GDPR concerns.

Going All-In

After moving a few of my sites over and being generally impressed with the Cloudflare experience, I decided to go all-in and consolidate all my domains (more than 20…yes, I have a domain problem) under Cloudflare. This meant having them be my registrar, DNS, and, where appropriate, hosting the content. Previously my domains were spread out among a few different registrars, and my DNS split between registrars, Digital Ocean, and Cloudflare. Having everything under one roof made a lot of sense.

The process was pretty straightforward. The only real snag I hit is that by default Cloudflare sets up “Flexible” SSL/TLS, where traffic is encrypted between visitors and Cloudflare, and then connections to your backend are made over HTTP. But if you already have a cert on your backend, and that backend is configured to redirect HTTP traffic to HTTPS, you’ll end up with a redirect loop. The fix is to just go into Cloudflare and change the SSL/TLS mode to “Full”.

So What Does This Cost?

Nothing. It’s all free…which has been bugging me a bit. Other than registrar fees when transferring domains to them, I haven’t paid them anything. They do have paid plans, of course, but the additional features offered—mostly around security—are targeted towards much bigger sites (bot detection, image optimization, cache analytics, web application firewall, etc.). I would actually be happy to pay them something, but I haven’t hit anything yet to warrant it. However, it seems like the paid plans are all per domain, so if I did end up needing to pay for something across all my sites, it would get unreasonably expensive (their cheapest paid plan, “Pro,” is $20/mo/domain).

Fingers Crossed

I do feel a little nervous about going all-in on Cloudflare, as I would going all-in with any company/service, but I’ve been really impressed with them so far. They’ve had some controversies in the past, but generally seem like a pretty good company, both technically and ethically. I’ll certainly keep you posted if anything changes.

A Year of Whatever

2025-01-10T10:48:06-08:00

It’s been a bit over a year since I was laid off from Etsy, where I’d been for 11½ years. Although everyone will tell you not to take a layoff personally, it still stings knowing that someone decided you were not worth keeping around. That said, the layoffs seemed to hit old-timers and remotes particularly hard, so the deck was stacked against me.

It wasn’t hard for me to decide not to go looking for another full-time job. I’d long fantasized about being an indie software developer, working on my own things and setting my own schedule, so I guess that’s what I was now.

I started an LLC for all the amazing apps I was going to start writing, but then some consulting fell into my lap. (Coincidentally, this same thing happened to me the last time I was laid off, nearly 25 years ago. I had been working in San Francisco during the dot-com boom and when the company went under I was all set to move to Bishop and become a climbing bum for a year, when a friend pinged me and I ended up spending the next four years consulting.)

I spent a couple months with Wikimedia Foundation creating data modeling guidelines. I learned that WMF has some really interesting and unique challenges, both technically and politically. It’s a really great group of folks.

That was followed a by a much bigger consulting opportunity, which lead to starting another LLC with some friends. We spend about six months tackling a pretty hairy data engineering challenge. What I learned from that is if you see NetSuite, run.

Around the paid gigs I also spent some time helping out one friend with their startup, another with his app, and wrote a web app to help another friend give away free bikes to kids.

I’ve been keeping myself pretty busy since that last gig ended. I spent a good amount of time doing a deep dive into MacOS aerial screensavers/wallpapers. I wrote a Safari extension to redact news articles. I wrote a little web game. I also spent a good amount of December doing Advent of Code, working on my Swift and SwiftUI skills in the process.

I’ll write separate posts about all of those projects, but I’m currently mulling over what to do next. I recently got a 3D printer, so playing with that and learning a bit of CAD has been taking a lot of my time, but that’s not nearly as fulfilling as a good project is.

So after a year of being “indie,” I’m still pretty sure I’m never going back to a full-time jobby-job. I do miss my colleagues at Etsy, and there’s something to be said about being part of a team and a larger mission, but I do not miss all the overhead that comes with working at a “large” company (“work about work”). I love waking up every day knowing that I have zero meetings and what I want to accomplish that day is entirely up to me. Literally whatever.

Now to just come up with that killer app idea….

Automating STOP to End

2024-03-31T17:48:17-07:00

My friends Sunah Suh, Anthony Hersey, and I finally figured out the magic incantations needed to automate sending “STOP” to political text messages. Setting it up is completely non-obvious, so I’ll walk through it step-by-step.

Note that this is for iOS only. If someone with an Android phone wants to write up their process, I’d be happy to link to it.

Also, this was done on iOS 17.4. Surely this process will change some day. If you notice something is off, let me know and I’ll update it.

First, open the Shortcuts app and tap the Automation tab at the bottom.

Tap the plus (+) in the top right corner.

Start typing "Message" in the search bar, and select Message.

On the When page, hit Choose next to Message Contains.

Type in the text to look for, such as "STOP to end". It appears to be case-insensitive.

Hit Done and back on the When page, select Run Immediately. Then tap Next in the top-right corner.

Now select New Blank Automation.

Under Next Action Suggestions, you should see Send Message. (If not, hit Add Action and search for "Send Message".) Tap Send Message.

Tap on the Message field and type "STOP".

Now comes the first tricky part. Hold down (long press) on Recipients and select Shortcut Input. (If instead it pops open a "To:" page, you didn't hold down long enough. Cancel and try again.)

Now the next tricky bit. Tap the Shortcut Input field and select Sender, then hit Done.

That's it! Now repeat this for different variations of "STOP to...." ("STOP to quit", "STOP2END", etc.)

The Case of the Absurdly Large Numbers

2023-01-21T18:10:08-08:00

One of the engineers on my team at Etsy, who is working on getting our clickstream events into protobuf, noticed that we were getting some values that were too large for protobuf’s largest integer, uint64. E.g: 55340232221128660000.

This value was coming from our frontend performance metrics-logging code, and is supposed to represent the sum of the encodedBodySizes of all the JavaScript or CSS assets on the page. But given that 55340232221128660000 is around 50,000 petabytes, it’s probably not right. There had to be a bug somewhere.

We were only seeing this on a small percentage of visitors to the site—less than 0.1%. I looked at the web browsers logging these huge values and couldn’t see any obvious pattern.

My first thought was that we had code somewhere that was concatenating numbers as strings. In JavaScript, if you start with a string and add numbers to it, you get concatenated numbers. E.g.:

> "" + 123 + 456
< '123456'

But I didn’t see anyway that could happen in our code.

Looking at the different values we were getting, I saw a lot of similar-but-not-quite equal values. For example:

18446744073709552000
18446744073709556000
18446744073709654000
36893488147419220000
36893488147419310000
55340232221128660000
55340232221128750000
55340232221128770000
73786976294838210000
92233720368547760000

The fact that they all end in zeros was curious, but I realized that was because they were overflowing JavaScript’s max safe integer.

I started googling some of the numbers and got a hit. I thought it was very odd that one of our seemingly random numbers was showing up on a StackOverflow post. Then I googled “18446744073709551616”. Any guesses what that is?

2⁶⁴

36893488147419220000? Approximately 2 * 2⁶⁴

55340232221128660000 =~ 3 * 2⁶⁴

etc.

I realized what was probably happening was when we were adding up all the resources on the page, some of them were returning 2⁶⁴-1 (the maximum 64-bit unsigned integer), but some were normal, so that explained the variation.

Now encodedBodySize is an unsigned long long, so my first thought was there is a browser bug that was casting a -1 into the unsigned long long, which would give us 2⁶⁴-1.

I started digging through browser source code. I found that in WebKit, the default value for responseBodyBytesReceived is 2⁶⁴-1, although that value should be caught and 0 returned instead.

Then I went spelunking in the Chromium source, and found this commit, from less than two weeks ago. (Although the bug was reported back in May 2022.) Sure enough, it was a default value of -1 getting cast to an unsigned value. Boom! Mystery solved.

I pushed a fix to our JavaScript to catch these bogus values and hopefully our protos will live happily ever after. Gone are my dreams of finding and fixing a major browser bug, but I’m happy that I can move on with my life.

The Framework for ML Governance

2021-09-18T15:14:51-07:00

The Framework for ML Governance by Kyle Gallatin

ML is becoming commoditized, but there hasn’t been much in the way of scalable frameworks to support the delivery and operation of models in production (of course, Algorithmia has one, hence their sponsoring of this report).

ML needs to follow the same governance standards for software & data as other more traditional software engineering does in your organization. In the development phase, this includes validation and reproducibility of your model, and documentation of your methods and rationale. The delivery and operational phases are much more complex, including things like observability, auditing, cost visibility, versioning, alerting, model cataloging, security, compliance, and much more.

MLOps is the set of best practices and tools that allow you to deliver ML at scale. ML governance is how you manage those practices and tools, democratizing ML across your organization through nonfunctional requirements like observability, auditability, and security. As ML companies mature, neither of these are “features” or “nice-to-haves”—they are hard requirements that are critical to an ML strategy.

Data scientists can’t do this all by themselves. They need support from the larger organization. The value of ML governance needs to be understood at the highest levels. They should involve infrastructure, security, and other domain experts early on, and be sure to be open in their communication with the rest of the company.

Advent of Code 2020: Swift

2020-12-28T08:00:55-08:00

Advent of Code was a fun diversion this year. It seemed a bit easier than some previous years. Perhaps it helped that I picked a language that I had some experience with, Swift. Here are some quick thoughts on Swift as an AoC language, some general AoC tips, and notes on some of the more interesting puzzles.

You can find all my 2020 code here.

Swift for AoC

First, the complaints:

Swift is too pedantic about strings. AoC usually involves loads of string manipulation and parsing, so being able to easily index into strings is super helpful. Swift does not make this easy out-of-the-box. I understand why. Strings are complicated in a language like Swift that supports Unicode well. Indexing into an array is expected to be a constant-time operation, but that’s not possible when you have to look at the preceding bytes in order to figure out what the _i_th character is. Still. You can at least make it easier and less ugly than myString[myString.index(myString.startIndex, offsetBy: i)]. Thankfully Swift is easy to monkey-patch, so I added some useful extensions to make it tolerable.

Regexes are similarly annoying in Swift. Here it just feels like they never properly ported them over from Objective-C. Here’s an example:

// print the animals found in a string
let myString = "The quick brown fox jumped over the lazy dogs."
let animalsRegex = try NSRegularExpression(pattern: #"(fox|dog)"#, options: [])
let matches = animalsRegex.matches(in: myString, options: [], range: NSRange(myString.startIndex..<myString.endIndex, in: myString))
for match in matches {
    for i in 1..<match.numberOfRanges {
        print(NSString(string: myString).substring(with: match.range(at: i)))
    }
}

My colleague Mike Simons gave me a RegEx helper he uses, which reduces the above to:

let animalsRegex = try RegEx(pattern: #"(fox|dog)"#)
let matches = animalsRegex.matchGroups(in: myString)
for match in matches {
    for group in match {
        print(group)
    }
}

Tuples not being hashable or even extendable is a pain for a lot of AoC problems. I resorted to using a custom struct for a bunch of problems, but at some point discovered SIMD vectors, which have the added benefit of having arithmetic operators defined for them.

What I loved about Swift:

Working with a statically-typed language in a proper IDE (Xcode) is such a huge time-saver. Everything is checked as you type (which can be a little annoying when Xcode starts scolding you for not using variables that you’ve defined before you get a chance to write the code that uses them; but you learn to ignore it). You have a full-featured debugger.

Playgrounds are really useful for trying out quick snippets of code.

Swift is generally a pleasant, straight-forward language to write. There’s not all the overhead of thinking about memory and ownership like there is in Rust, for example.

It’s actually pretty performant—at least when not running the debug build (I usually got a 10x speed-up when building for release).

General AoC Tips

Hash maps, hash maps, hash maps. Even if the problem seems array-shaped, there’s a good chance a hash map will be more efficient.
Many of the problems are grounded in some basic CS principle (day 5 this year being a classic example). Think about what that might be before diving in.
Oftentimes part 2 will be part 1 but MORE. Days 15 (Rambunctious Recitation) and 23 (Crab Cups) this year were good examples of that. When coding part 1, think about how your solution might scale if you were doing a million times more, or with a million times more data.

The Puzzles

Ok, on to specific puzzles. I’m just going to talk about some of the more interesting ones. Spoilers follow:

Day 5: Binary Boarding (code)

This is a classic AoC problem in that it disguises basic CS principles. If you were to solve this naively, it might be kind of annoying. but when you realize that the seating strings (e.g. FBFBBFFRLR) are just binary numbers, with (F,L) -> 0 and (B,R) -> 1, it becomes trivial. Particularly helpful here was Swift’s ability to convert a binary string to decimal with Int(binary, radix: 2).

Part 1 just required sorting the list of converted seat numbers and taking the last (highest) one. Part 2, finding the first non-consecutive id, is probably faster just iterating through the sorted list and stopping on the first id that is not one more than the last id (which is what I did originally), but it was more fun and concise to use Gauss’ formula to calculate what the total should be if all the consecutive numbers were present and subtract the actual sum of all numbers.

For kicks, I did a “no code” version in a spreadsheet.

Day 7: Handy Haversacks (code)

This was the first day that required some real thought. I started down the path of having a bidirectional graph with each bag having parents and children, but I got bogged down. I think sometimes creating objects complicates things vs. just slinging around string hash maps.

Day 9: Encoding Error (code)

This is totally an interview question I would have failed. It’s easy enough doing it the “dumb” (O(n^2)) way that I did it, constructing a lookup table of all sums in the preamble, or, for part 2, iterating through every possible sequence of two consecutive numbers, then three, etc.—also O(n^2). After I finished, a friend shared the “inchworm” technique, which is O(n). Neat! I totally failed that interview.

Day 10: Adapter Array (code)

Part 2 was fun to figure out. There were very different ways to approach this. I went with recognizing that we were dealing with power sets. I wrote up my thinking in my code, so I won’t duplicate it here. Some friends solved this using dynamic programming, which I definitely need to level up on.

Day 11: Seating System (code)

Nothing particularly interesting here—the first of three Game of Life-type problems. I just had to share this, though.

Day 12: Rain Risk (code)

Pretty straightforward. Part 2 was fun because I [re-]learned how to rotate a vector.

Day 13: Shuttle Search (code)

This was a fun one. I solved part 1 in bed on my phone with a calculator, and then spent the entire next day working on part 2. I learned about the Extended Euclidean algorithm and the Chinese remainder theorem. I used the existence construction to solve it, but the simpler solution that many used was sieving.

Day 17: Conway Cubes (code)

Game of Life again! Doing it in 3D and then 4D was a fun twist. Nothing particularly tricky. I just have a soft spot for GoL.

Day 18: Operation Order (code)

This was a fun one. I spent a lot of time scratching my head and drawing diagrams, until I remembered it’s a lot easier to parse expressions in postfix, and you can use the Shunting yard algorithm to convert infix to postfix.

Day 19: Monster Messages (code)

Woo, boy! This was a good one. I’d never written a parser before, so this was a learning experience. My part 1 solution involved creating a matcher from the grammar, and it was fast and elegant. Part 2 threw in the wrench of recursive rules, which absolutely do not work when building a matcher. I tried a whole bunch of things like adding recursion limits, but I could not get it to work. I ended up moving on and came back to it only after I’d finished all the other days. By then I had heard the term CYK algorithm bandied about, and so I did a complete rewrite using that. I spent a bunch of time reading about the algorithm and about Chomsky normal form. I “cheated” a bit and didn’t write code to convert the provided rules to Chomsky normal form, as there were only a few rules that needed to be tweaked and it was easy enough to do it by hand:

Original	Chomsky normal form
`107: 18 \| 47`	`107: "b" \| "a"`
`11: 42 31 \| 42 11 31`	`11: 42 31 \| 42 133` `133: 11 31`
`8: 42 \| 42 8`	`8: 47 50 \| 18 4 \| 42 8`

Day 20: Jurassic Jigsaw (code)

This was a favorite. Part 1 was easy once I figured out a way to generate a hash code that uniquely identified a side, irrespective of its orientation:

static func sideToInt(_ side:String) -> Int {
    let binary = side.replacingOccurrences(of: ".", with: "0")
                     .replacingOccurrences(of: "#", with: "1")
    let num = Int(binary, radix: 2)!
    let numReversed = Int(String(binary.reversed()), radix: 2)!
    return num * numReversed * (num ^ numReversed)
}

Once you have that, your corners are the tiles that have two “singleton” sides (i.e. unique to that tile). No assembling necessary…for part 1. Part 2 did require assembling, and that was a bit tedious. It was a cool problem, though.

Day 23: Crab Cups (code)

Classic AoC problem where the simple solution for part 1 utterly fails in part 2. I actually got the answer to part 2 after letting my dumb-but-optimized part 1 solution run overnight, but while it was running, mulling it over in bed, I realized that this was a linked list problem. I fixed it in the morning and the runtime went from ~ 7 hours to 0.2 sec.

Day 24: Lobby Layout (code)

This was surprisingly easy for this late in the calendar, although my first naive attempt trying to map it to a 2D coordinate system failed. But I googled “hexagonal grid coordinate system” and found this cool page. I used the cube coordinate system and it was easy-peasy.

Part 2: Yay, more Game of Life! I’m sorely tempted to write a visualization for this.

Humanize

2020-10-10T18:27:47-07:00

I scratched an itch last week and actually wrote some code (these days my primary IDE is Google Docs). Check out humanize, which replaces any large numbers in text piped into it with shortened versions using SI prefix symbols (K, M, G, T, etc.). So “File size: 972917401236” becomes “File size: 973G”.

This came about because I was debugging an issue with a Parquet file using parquet-tools, whose output looks like:

% parquet-tools meta part-01373-r-01373.gz.parquet | grep "row group"
row group 1:           RC:100 TS:121513011 OFFSET:4 
row group 2:           RC:100 TS:44877183 OFFSET:14293218 
row group 3:           RC:111 TS:28645460 OFFSET:19704281 
row group 4:           RC:168 TS:31794536 OFFSET:23387243 
row group 5:           RC:363 TS:28078938 OFFSET:27531686 
row group 6:           RC:15547 TS:899825873 OFFSET:31242035 
row group 7:           RC:9963 TS:47930700 OFFSET:142353955 
row group 8:           RC:20100 TS:911417774 OFFSET:155302131 
row group 9:           RC:20100 TS:889810662 OFFSET:269749697 
row group 10:          RC:10100 TS:855908709 OFFSET:382090935 
row group 11:          RC:7282 TS:1532744480 OFFSET:486019001 

Tired of squinting and counting digits to see how big some of these numbers were, I figured there’d be some easy bash/sed way to convert these numbers into something more readable. It is pretty easy to do this for isolated numbers using numfmt:

% echo "1234567890" | numfmt --to=si --round=nearest
1.2G

But that doesn’t work on numbers embedded in text:

% echo "This is a big number: 1234567890" | numfmt --to=si --round=nearest
numfmt: invalid number: ‘This’

GNU sed can do a replacement with the result of a function, but it’s rather ugly:

% echo "This is a big number: 1234567890" | sed -re 's/[0-9]+/`numfmt --to=si --round=nearest &`/g;s/.*/echo &/;e'
This is a big number: 1.2G

But I’m doing this work on a Mac, which does not come with GNU sed (or numfmt, for that matter, although you can brew install coreutils to get gnumfmt). I started working on a perl one-liner to do it, but decided this was something that needed to exist on its own, and should be easy for anyone to install.

I chose to write it in Go because it has an easy way for people to install binaries with go get. (Rust has something similar with cargo install, but Go is more common in my workplace.) The problem with go get is that you have to run it again to upgrade, so for my fellow Mac users I added a Homebrew tap.

Long story short:

% parquet-tools meta part-01373-r-01373.gz.parquet | grep "row group" | humanize -binary
row group 1:           RC:100 TS:116Mi OFFSET:4 
row group 2:           RC:100 TS:43Mi OFFSET:14Mi 
row group 3:           RC:111 TS:27Mi OFFSET:19Mi 
row group 4:           RC:168 TS:30Mi OFFSET:22Mi 
row group 5:           RC:363 TS:27Mi OFFSET:26Mi 
row group 6:           RC:15Ki TS:858Mi OFFSET:30Mi 
row group 7:           RC:10Ki TS:46Mi OFFSET:136Mi 
row group 8:           RC:20Ki TS:869Mi OFFSET:148Mi 
row group 9:           RC:20Ki TS:849Mi OFFSET:257Mi 
row group 10:          RC:10Ki TS:816Mi OFFSET:364Mi 
row group 11:          RC:7Ki TS:1Gi OFFSET:464Mi

ReadBot

2018-03-01T18:12:42-08:00

I wrote a Slack bot, ReadBot, to both scratch an itch and check out the current state of Slack integration. As a bonus, I got to spend some time getting familiar with Glitch, a pretty cool live coding platform.

ReadBot lets you connect your Goodreads and Pocket accounts to Slack, and then adding a bookmark (🔖) reactji to a message with an Amazon book link in it will add that book to your Goodreads “to-read” shelf, or any other URL to your Pocket account.

The Slack API and documentation has improved immensely since I last checked it out a couple years ago. I was clearly not the only one to get confused by profusion of integration options (bot? slash command? webhook?) , so they’ve improved their API landing pages to help guide you. There are also plenty of tutorials to get you started.

One of the fears when diving into an API integration is whether that API—or even integrations in general—are going to be supported in the future. Many developers have been burned by neglected, deprecated or even shutdown APIs. Slack, however, seems to be encouraging integrations whole-heartedly with its app directory.

One challenge of creating a new Slack integration has been Slack’s requirement that it be served over SSL. While Let’s Encrypt has made this a lot easier, setting up an SSL-enabled server is still a significant hurdle to clear just to start developing your integration. This is where Glitch comes in.

Glitch is a collaborative live coding environment made by Fog Creek Software. It’s super easy to get started, especially since you can find existing public projects on the site and “remix” them to make them your own. And every project has its own SSL-enabled hostname, right out of the box.

Glitch apps are Node.js-based (they seem to allow Python apps as well, but support is currently unofficial). The app restarts itself on the fly as you edit (although this can be disabled), so your app is always live. Autocomplete package search makes installing new packages a breeze. You can do pretty much everything you need to through the UI, but if you need, you can drop into a shell and do whatever you like to your instance.

There’s no built-in revision/commit history, but you can export your project to GitHub (as well as import from GitHub). Since your app is just a regular Node app, it’s perfectly portable to any other environment.

One general issue with writing Slack bots is that in order to continue working on your app after it is in production, you need to have separate development apps and endpoints. It would be nice if Slack made this a little easier, perhaps by allowing you to specify separate production and development endpoints, slash commands, etc.

I have no plans on publishing ReadBot to the Slack App Directory because I don’t want to have to maintain another production service. If you’re interested in doing so, go for it.

Rescuing Windows with Hammerspoon

2018-01-23T21:02:42-08:00

I recently got a new monitor and it came with a problem. For some reason when I attached or detached my laptop, many of my windows would end up off the edge of the screen, and I’d have to drag them back over one by one. It was annoying enough that I decided I’d write an app to “rescue” them.

I figured it wouldn’t be too hard. Get a list of active windows, see which are off-screen, and move them over. But it was hard. My code only half-worked. And for some reason I couldn’t get Chrome windows to move.

I started looking around and found Swindler. It seemed to be just what I needed and confirmed that I wasn’t completely incompetent:

Writing window managers for macOS is hard. There are a lot of systemic challenges, including limited and poorly-documented APIs. All window managers on macOS must use the C-based accessibility APIs, which are difficult to use and are surprisingly buggy themselves.

But then I saw Hammerspoon listed at the bottom of the Swindler README, under Related Projects. Intrigued, I checked it out. It was just what I needed.

Hammerspoon lets you write Lua scripts to interact with the MacOS system, and it’s pretty powerful. The Getting Started Guide gives a good sense of some of the things you can do with it.

I’d never written Lua before, but with plenty of examples to crib from and good API documentation, it didn’t take me long to come up with a little script that did just what I needed:

I have that file saved at ~/.hammerspoon/rescuewindows.lua, and in ~/.hammerspoon/init.lua I have:

local rescueWindows = require "rescuewindows"
hs.hotkey.bind({"cmd", "alt", "ctrl"}, "R", rescueWindows)

So when it hit ⌃⌥⌘R, all my off-screen windows slide over to my main screen. Nice!

Going "Serverless"

2018-01-15T16:13:13-08:00

When my friend Nelson and I set out to build our location tracking app & accompanying website, Wanderings, we started down the familiar path of choosing our stack. Flask seemed like a reasonable choice. Postgres for the database. SqlAlchemy for the ORM? Sure, I guess. We figured we’d try hosting on GCP, mostly to learn more about the platform. Do we bother with containers? Meh, leave those for the youngsters. And so on.

But the more we thought about hosting our own service, the less enthused we were. The GCP costs were piling up, and we didn’t really want to be responsible for having a copy of our users’ location data. One of the driving ideas behind Wanderings was having a location tracker where you controlled your own data. Why should you trust us with a copy of it?

I should back up and talk a little about what Wanderings is and how it works.

Back in 2011 there was a small uproar when it was revealed that Apple was apparently tracking your every move with your iPhone. Out of that grew a project, OpenPaths, that allowed you to generate your own location data, and have full control over it. The app used very little energy, so you could just start it and forget it.

However, with the release of iOS 11, OpenPaths stopped working, and its accompanying website, openpaths.cc, no longer resolved. It appeared to be no longer maintained. There were some other location tracking apps out there, including Google Timeline, Fog of World, and Strut, but none that had the combination of strong privacy guarantees , simplicity, and low-energy usage that we were looking for. So we decided to make our own.

The Wanderings iPhone app is pretty basic. It uses iOS’ significant location change service to only update your location when you move by a significant amount. It also relies on wifi and cellular for location information rather than power-hungry GPS. It does all this in the background, so you can start it once and forget it.

New location data is uploaded to your private, secured CloudKit database on iCloud, so you are the only one with access to your data.

So back to our server conundrum.

One of our requirements was a website that would render the full heat map of your travels, let you export that data, and eventually let you do a lot of interesting things with your data. Of course, it doesn’t make sense to re-download all your location data every time you visit (especially since CloudKit only lets you pull 200 records per request) so we needed some kind of cache of your data in the app, as in a database.

But we don’t really want your data. And we really don’t want to deal with maintaining a server somewhere, either.

Fortunately, for a few years now browsers have supported IndexedDB, an embedded transactional database with sizable storage limits. We realized if we used this as our cache, the app could be a static site (something Nelson had explored years ago), which we can host cheaply anywhere. In fact, we ended up just using GitHub Pages to host the site right out of our repository.

Using a browser-embedded database has the downside of having to re-download all your data when you go to a new browser or if you’ve cleared your cache, but we think that’s an acceptable trade-off to have complete control of your data.

So check it out and let us know what you think. Send your bugs/feature requests/love to support@wanderin.gs.

Code Signing Will Kill Me Yet

2016-03-31T15:25:19-07:00

This is one of those I-spent-long-enough-stumped-on-this-issue-I-should-write-it-up-for-future-generations posts.

I wrote a little app at work (which I’ll talk about in a future post), and I was having a strange code signing issue. The app would sign just fine:

$ spctl --assess -v Orwell.app
Orwell.app: accepted
source=Developer ID

But then if I zipped it up and sent it to someone, when they unzipped it, OS X would tell them that it was damaged and should be thrown away:

Sure enough, if I zip and unzip the app, and verify the code signing again, I get:

$ spctl --assess -v Orwell.app
Orwell.app: a sealed resource is missing or invalid

After much hair-pulling, I found Technical Note TN2318: Troubleshooting Failed Signature Verification, which says:

The file prefixed with “._” is problematic and a was the result of copying certain Mac OS X files to a non-HFS+ formatted disk. These files are referred to as Dot files, Apple Double files, or resource forks. They are invisible to Finder but can be removed using the dot_clean utility.

Sure enough:

$ find Orwell.app -name "._*"
Orwell.app/Contents/Resources/app/node_modules/applescript/._package.json
Orwell.app/Contents/Resources/app/node_modules/applescript/lib/._applescript.js
Orwell.app/Contents/Resources/app/node_modules/applescript/samples/._execString.js
$ dot_clean Orwell.app
$ find Orwell.app -name "._*"

Running dot_clean before I signed the app fixed the issue.

(Orwell is an Electron app, hence the node_modules dir, in case you were wondering.)

Fixing A Swift Memory Leak

2015-10-10T10:57:00-07:00

I wrote a Mac menu bar app for internal use at Etsy that show the status of a number of internal systems and pops up a notification if any of them change status—for example, if a problem is detected and someone puts a hold on our deployment queue.

Some months ago, a colleague noticed the app was leaking memory. I put off looking at it until recently, when I noticed that my own instance was eating up 3+ GB of memory.

Fortunately, Apple provides some great tools for tracking down leaks. One is Instruments, an incredibly powerful tool that I know thiiis (puts thumb and forefinger a millimeter apart) much about. But I know enough to inspect a process for leaks.

I started Instruments, selected the Leaks profiling template, and attached it to my leaky app, and saw this:

Yup, that’s a leak. You can also drill down to see where the leak is coming from:

So the leak is obviously in the networking code, seemingly in Apple’s part of it.

I should pause here to say a bit about how the app works. It’s pretty simple: every 3 seconds, it fetches an internal status web page, parses out the statuses from an HTML table (writing a JSON endpoint for the status page is on my todo list!), and updates the menu accordingly.

The networking code in the app was pretty simple:

let session = NSURLSession.sharedSession()
let task = session.dataTaskWithURL(url) { data, response, error in
    // check for error, do something with the data
}
task.resume()

NSURLSession was introduced with in Mavericks and iOS 7 as a replacement for NSURLConnection.

I tried setting the cache sizes to 0, and invalidating the session when I was done, as recommended in this article, which seems to describe the same leak:

// THIS CODE DID NOT HELP!
let configuration = NSURLSessionConfiguration.defaultSessionConfiguration()
configuration.URLCache = NSURLCache(memoryCapacity: 0, diskCapacity: 0, diskPath: nil)
let session = NSURLSession(configuration: configuration)
defer {
    session.finishTasksAndInvalidate()
}
let task = session.dataTaskWithURL(url) { data, response, error in
    // ...

But this didn’t help. What did end up working was using an ephemeral session:

let session = NSURLSession(configuration: NSURLSessionConfiguration.ephemeralSessionConfiguration())
let task = session.dataTaskWithURL(url) { data, response, error in
    // ...

With that in place, here’s what Instruments showed me:

Yay! No leaks!

We Don’t Need No Stinkin’ GUI

If you’re more of a command-line gal, there’s another way to investigate leaks built into OS X: the leaks command:

$ leaks EtsyInternalStatus
Process:         EtsyInternalStatus [24147]
Path:            /Applications/EtsyInternalStatus.app/Contents/MacOS/EtsyInternalStatus
Load Address:    0x105eaf000
Identifier:      com.etsy.EtsyInternalStatus
Version:         1.2.3 (1.2.3)
Code Type:       X86-64
Parent Process:  ??? [1]

Date/Time:       2015-09-02 20:33:26.173 -0700
Launch Time:     2015-09-02 20:30:44.447 -0700
OS Version:      Mac OS X 10.10.5 (14F27)
Report Version:  7
Analysis Tool:   /Applications/Xcode-beta.app/Contents/Developer/usr/bin/leaks
Analysis Tool Version:  Xcode 7.0 (7A192o)
----

leaks Report Version:  2.0
Process 24147: 25225 nodes malloced for 15639 KB
Process 24147: 615 leaks for 5321984 total leaked bytes.
Leak: 0x10a02f000  size=135168  zone: MallocHelperZone_0x105f65000  length: 5020
...
<snipped 7800 lines of leaked memory dumps>

Now, the fixed version:

$ leaks EtsyInternalStatus
Process:         EtsyInternalStatus [85066]
Path:            /Applications/EtsyInternalStatus.app/Contents/MacOS/EtsyInternalStatus
Load Address:    0x102d2f000
Identifier:      com.etsy.EtsyInternalStatus
Version:         1.2.4 (1.2.4)
Code Type:       X86-64
Parent Process:  ??? [1]

Date/Time:       2015-10-10 10:27:29.556 -0700
Launch Time:     2015-10-10 10:16:11.476 -0700
OS Version:      Mac OS X 10.10.5 (14F27)
Report Version:  7
Analysis Tool:   /Applications/Xcode.app/Contents/Developer/usr/bin/leaks
Analysis Tool Version:  Xcode 7.0.1 (7A1001)
----

leaks Report Version:  2.0
Process 85066: 62741 nodes malloced for 6598 KB
Process 85066: 0 leaks for 0 total leaked bytes.

The eagled-eyed among you might have noticed that about a month elapsed between me running those checks (life intervened), and there was an Xcode release in the meantime. Turns out that when I ran the leak check again on the leaky version of the app, it was fine! I’m guessing that it was a bug in Apple’s networking code after all, and it got fixed in the latest release.

Try out leaks on some of the other apps you have running. You’ll find that leak-free apps are oddly rare, and some are downright eggregious (my beloved Sublime Text text editor, in which I’m writing this post, shows “577841 leaks for 44554352 total leaked bytes.” Oof).

For more information, see Apple’s documentation on Finding Memory Leaks.

On "Working" Outside of Work Hours

2015-08-08T22:53:58-07:00

Etsy’s culture puts a very strong emphasis on work-life balance. As someone who has mostly worked for startups, this took some adjusting to. The fact that Etsy IRC and email pretty much goes silent outside of normal business hours struck me as odd initially. But, I really appreciate it. Startups are stressful, and they left me feeling that there was always something work-related that I should be doing whenever I had some “free time.”

However, I love my work. I love the people I work with and the projects I work on. I love solving problems. I love coding. I wish I had more time to spend doing it. Having a family puts a massive dent in your free time. Don’t get me wrong, I love my family and am [usually] happy to spend any available time with them, but I so miss having large blocks of time to myself to just get into the zone.

So when I or other colleagues mention coding (or even checking email) after hours, or on vacation, we’re often castigated for doing so. I love that my colleagues are looking out for my well-being, but hey, we’re all individuals. If you have no desire to be anywhere near a computer outside of work hours, that’s awesome. It’s pretty cool to work at a company where that’s not only possible, but encouraged, isn’t it? But please don’t judge me for thinking that my ideal vacation would be go somewhere alone, with my laptop and a fast internet connection, and just code.¹

If you’ll excuse me, I’ve got to get to bed. I like to wake up crazy early so I have a couple hours of alone time with my laptop.

There is an argument that a senior engineer working after-hours sets a bad example for more junior engineers, making them feel like they should be doing the same. This is certainly something to be cognizant of, but so long as the prevailing culture makes it clear that this is in no way expected behavior, it should not be an issue. ↩

Two Tools for Packaging Mac Apps

2015-04-19T06:25:00-07:00

I ran across two apps yesterday that make packaging Mac apps for distribution a breeze:

To easily build a professional DMG-based installer, look no further than DMG Canvas. There’s a free version with some limitations, but honestly $15 is a bargain for such a useful tool.

For more complex installs, you might need to create a .pkg file, in which case grab the free (and ungoogleable) Packages app by Stéphane Sudre.

Write a Mac Menu Bar App in Swift

2015-03-26T06:19:42-07:00

This past week I taught an Etsy School class on how to write an OS X menu bar (aka status bar) app in Swift, and the walk-through is now public. I hope you find it useful.

If you have any suggestions on improving it, let me know. (Or better yet, submit a PR!)

Sublime Text Guide

2014-11-09T07:16:54-08:00

I’m a big fan of the Sublime Text text editor, and I’ve become one of its biggest evangelists at Etsy. A few months ago I gave an internal presentation/tutorial session on it, and used those notes to create a guide on our internal wiki. I’ve ported that guide to my blog so I can spread the Good Word to others. Since the source code for my blog is available publicly, I’m also hoping that people will help make it even better.

So head over to my Sublime Text Guide and let me know what you think.

Speeding up Jekyll's LSI

2014-11-06T08:51:59-08:00

Jekyll’s Latent Semantic Indexing (LSI) indexes your posts to create the “Related Posts” section you see at the bottom of the individual post pages. However, if you have more than a handful of posts, it is very slow, and it seems to get exponentially slower as the number of posts increase. I ran it on this site, with ~140 posts, and I finally killed it after 3 hours.

There’s a fix for this, though, buried in the README for classifier-reborn, the library used to do the post classification. Install GNU GSL and rb-gsl. On OS X with homebrew, it’s as easy as:

brew install gsl
gem install rb-gsl

Indexing time for me went from, well, 3-∞ hours to less than 5 seconds.

Fun with Swift Extensions

2014-11-04T22:04:55-08:00

As promised, a post on extensions in Swift.

This isn’t going to be an in-depth tutorial. Extensions have been covered better elsewhere, most notably in Apple’s excellent documentation. I just want to show a couple extensions I’ve written: Array#find, and comparison operators for NSDate.

Array#find

If you want to find a value in a Swift Array, you can use the global generic find function:

let arr = [4,8,15,16,23,42]
let idx = find(arr, 15)  // Some(2)

However, find only lets you search for a static value, and what I really wanted was the ability to find the first value that satisfied a test function. For example, say I want to find the first odd number in an array. Let’s extend Array so we can do that.

extension Array {
    func find(test: (T) -> Bool) -> Int? {
        for i in 0..<self.count {
            if test(self[i]) {
                return i
            }
        }
        return nil
    }
}

let arr = [4,8,15,16,23,42]
let idx = arr.find { $0 % 2 == 1 }  // Some(2)

The extension defines a method, find, on Array, that takes as an argument a function mapping an element of type T to a Bool, and returns an optional Int. In other words, we pass in a test function, and it returns the index of the first value in the array for which that function is true. If no such values are found, it returns nil.

But wait! you say. Why write an extension when you can just add a new signature to the global find function so that it will take a test function rather than a value? FINE. Be that way.

You can, um, find the method signature of find in Apple’s documentation:

func find<C: CollectionType where C.Generator.Element: Equatable>(domain: C, value: C.Generator.Element) -> C.Index?

That looks a bit complicated, and CollectionTypes and Generators are a topic for another day, but all you really need to know is that in order to conform to the CollectionType protocol, you need to define a startIndex and an endIndex, and be subscriptable. C.Generator.Element is the type of items in the collection. So instead of passing in a value of type C.Generator.Element to find, we want to pass in a test function that takes a C.Generator.Element and returns a Bool. The rest looks very much like our Array extension:

func find<C: CollectionType where C.Generator.Element: Equatable>(domain: C, test: (C.Generator.Element) -> Bool) -> C.Index? {
    for i in domain.startIndex..<domain.endIndex {
        if test(domain[i]) {
            return i
        }
    }
    return nil
}

let arr = [4,8,15,16,23,42]
let idx = find(arr, { $0 % 2 == 1 })  // Some(2)

Comparison Operators for NSDate

NSDate annoyingly does not implement the Comparable protocol, meaning that instead of:

if someDate <= anotherDate { ...

You have to do:

if someDate.compare(anotherDate) == NSComparisonResult.OrderedAscending || someDate.isEqualToDate(anotherDate) { ...

Let’s fix that. According to Apple’s docs, we only need to implement < and == to conform to the Comparable protocol. Pretty clever, since with just those two, you can derive <=, >, >=, and !=.

extension NSDate : Comparable {}

public func < (lhs: NSDate, rhs: NSDate) -> Bool {
    return lhs.compare(rhs) == NSComparisonResult.OrderedAscending &&
        !lhs.isEqualToDate(rhs)
}

public func == (lhs: NSDate, rhs: NSDate) -> Bool {
    return lhs.isEqualToDate(rhs)
}

The first line just says that we are extending NSDate to conform to the Comparable protocol. You can leave this out, but then you’ll have to implement all the other comparison operators.

The rest is pretty straight-forward. The only curious bit is why the operators are implemented in the global scope, rather than within the extension. I.e. you’d think you could do this:

// This does not work!
extension NSDdate : Comparable {
  func < (other:NSDate) -> Bool {
    return self.compare(other) == ...

So why not? Operators aren’t methods in Swift, but global functions. They have precedence over methods, and have varying rules around associativity. You can even make up your own operators if you want. Be sure to read NSHipster’s great article on Swift operators.

Swift Compiler Bugs

2014-11-04T05:27:31-08:00

Swift is a great language, and far more enjoyable to program in than Objective-C, but it is still in its infancy, and bugs abound. One of the more frustrating is compiler errors that are flat-out wrong. Take a look at this (in Xcode 6.1):

“Could not find member ‘date’?” Trust me when I tell you that date is indeed a member of a Todo object. So what’s going on? Let’s tweak it a bit:

Ah, there we go. Turns out you can’t actually compare dates like that (hmm…I smell a follow-up post on writing extensions). Here’s how you do it:

Here’s another one. The following is essentially a no-op:

for i in 0..<0 {
    println(i)
}

That’s just a for loop from zero up to, but not including, zero. And this code is fine, doing what it is supposed to do—nothing.

However, try making the ending value of the loop less than the starting value. This should be equivalent to the above:

for i in 0...(-1) {
    println(i)
}

The ... operator means “up to and including”. So this should also be a no-op (and it is in most programming languages). But:

Even worse, it will actually crash on the first executable line of code:

None of this should scare you away from using Swift. It’s a fun language to work with, and ends up being much more concise than Objective-C. Just keep in mind that there are still bugs to be worked out, and if you’re getting errors that just make no sense, there’s a fair chance you’ve hit one.

The Importance of Sucking

2014-11-03T10:09:52-08:00

One of the defining charactaristics of a successful programmer is being OK with the feeling that you have no idea what you’re doing. I’ve been programming for over 30 years and it still happens to me regularly. The key is pushing on through that dark period, trusting that eventually you’ll catch a glimmer of light and it will slowly start to make sense.

There are a few tricks I’ve picked up to help me get over that initial hump. The first is to find a tutorial someone has written, and just follow along, just to get something working. Once I have something working, no matter how basic, not only do I start to get a better sense of how all the pieces fit together, but I can start tweaking it and learn from how it responds.

Can’t find a good tutorial? Great! Here’s your opportunity to give back, and write your own. There’s really no better way to learn. Start with doing the most basic thing imaginable. Just get something working. Write down your steps as you go. Then add to it, write it down, repeat. You’ll eventually have something coherent that you can post, and you’ll earn a ton of internet karma points.

It also really helps if you have a specific project to work on, otherwise the tutorial can be a bit of a dead end, and your effort can feel purposeless. Don’t have a project, and can’t think of one? Rewrite something you’ve already written. Ask friends and family if there’s an app they wish they had. Write a basic version of an app that you’re too cheap to buy. Write a todo app.

Start small, though; the bigger the project, the less likely you are to finish it. Minimum viable product (MVP) isn’t just Silicon Valley jargon; it applies equally well to personal projects. Do the simplest, dumbest thing possible. It’s easy to iterate and improve once you have something working, and you can always throw it all away and rewrite it once you feel you know what you’re doing.

The feeling of sucking at something and being completely lost is a part of learning. Everyone starts out that way. If you’re actively avoiding sucking, you’re not learning. Get lost. Seriously, find someplace to get yourself lost. You’ll be better for it. Then tell us about it when you inevitably emerge from the darkness.

Prose.io

2013-09-29T00:00:00-07:00

Prose.io is a content editor for GitHub. It supports Jekyll sites (like this one), and provides Markdown previews. I’m writing this post using it.

I’m sure I looked at Prose some time ago, but must have decided something was lacking. But I just took a look at it again, and it’s just what I need. In fact, I had started a similar project myself not long ago, and now I feel relieved that I can officially call it dead and stop fretting about having stalled out on it.

The Future of Programming

2013-07-31T15:42:37-07:00

The most dangerous thought that you can have as a creative person is to think that you know what you’re doing. Because once you think you know what you’re doing, you stop looking around for other ways of doing things.

— Brett Victor, The Future of Programming

He’s posted references and follow-up notes at http://worrydream.com/dbx/.

Why is Git Rejecting Me?

2013-06-26T21:21:21-07:00

Every Git user, from novice to expert, gets this error message at some point, maybe even daily:

$ git push origin main
To git@github.com:bgreenlee/myrepo.git
 ! [rejected]        main -> main (non-fast-forward)
error: failed to push some refs to 'git@github.com:bgreenlee/myrepo.git'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and merge the remote changes
hint: (e.g. 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

There are a couple ways this can happen, but it all boils down to this: the remote repository you’re pushing to has one or more commits that you don’t have in your local repository. Don’t worry, though, the fix is easy, although how to go about it depends on what you’re trying to do.

Refresher: Rebase vs. Merge

Before we dive in, a quick refresher on rebase vs. merge. These are two general strategies for combining two branches.

Consider the following case:

  main: A -> B -> C
my-fix:       \-> D

Here, you created a branch named “my-fix” from main while it was at commit B. While you were working on your fix, someone else pushed commit C to main. You finish your fix and commit it to the my-fix branch. That’s commit D above.

Now you’re ready to merge your my-fix branch back into main. There are two ways to go about this.

Merge

You can do:

git checkout main
git merge my-fix

This will resolve the differences between the two branches by doing a three-way merge using commits C and D and their common ancestor, B. It does this by creating an additional merge commit, which we’ll call M:

main: A -> B -> C -> M(B,C,D)

Rebase

You can also do:

git checkout my-fix
git rebase main
git checkout main
git merge my-fix

The first checkout is just to make sure you’re starting on the right branch. You may already be on the my-fix branch, in which case you can skip it. What git rebase main does is “rewind” your branch to the point at which it diverged from main, B, then apply all the commits from main made after that point (C), then “replay” your commits to my-fix (D). What you get is:

my-fix: A -> B -> C -> D’

Note that I used D’ above. This is because while the content of the commit is the same as your original commit D, it’s technically a different commit, with a different SHA. This will be important later.

The last two commands switch back to main and then merge my-fix with main. Since at this point my-fix has everything that main has, plus one extra commit, the merge is just a “fast-forward”, meaning that all git has to do is move the HEAD pointer to D’—no merge commit is required.

Now, merging vs. rebasing is somewhat of a religious issue, although in my experience, most organizations prefer rebasing, as it keeps the commit history from getting cluttered with merge commits. I am going to go on the assumption that rebasing is the preferred method.

Now, let’s get back to fixing our problem.

Scenario 1: Pull, then Push

This is the most common scenario, and simplest fix. Say you’ve made some changes in the main branch of your local repository, then go to push them to the main branch of the remote repository. If your push is rejected, what has most likey happened is that someone else pushed some changes to the remote main while you were making your changes, and you need to pull them down to your repo before you can push your changes up. So do a ‘git pull –rebase’, then push again.

Scenario 2: Force Push

Let’s say you’ve been working on your own branch, and suddenly git is rejecting you. Now you know that no one else has been pushing commits to that branch, so what happened?

Most likely, you updated your local branch by rebasing against main. Again, this “rewinds” the commits in your branch to the point where they diverged from main, then pulls in the commits from main that you don’t have, then “replays” the commits in your branch that weren’t in main as new commits. So when you go to push those to your remote branch, even though it may look like the commits are the same, the SHAs are different, so git will refuse to update the remote branch.

One way around this is to force push:

git push -f origin my-fix

This tells git that you don’t care about what’s on the remote branch, just update it with what you have locally. This can be dangerous. Don’t do it unless you know what you’re doing. You would only ever want to do this if you are sure that you are the only one working on that branch, and that you have all the commits locally that are on the remote branch (i.e. you didn’t push changes from another machine). If that’s not the case, and, say, a coworker had pushed some of her own changes to your branch, you will overwrite those changes. Now, chances are they will be recoverable (she probably still has them in her local repository), but digging yourself out will be painful.

If you are in a situation where you’ve rebased your branch and meanwhile your colleagues have pushed changes to the remote branch, one way to get back on track is to cherry-pick their commits into your branch, and then force-push your branch (after telling your colleagues to hold off on pushing):

git fetch  # to make sure your repo knows about their commits
git log origin/my-fix  # to see what their commits are, and copy the shas
git cherry-pick <sha1>
git cherry-pick <sha2>
git push -f origin my-fix

Your colleagues will then have to delete their local branch and pull it down again.

I’m sure I’ll get some hate mail for suggesting force push as a viable option, so I’ll reiterate: don’t do this unless you know what you’re doing. Even if you do know what you’re doing, you’re playing with fire, as a developer at my company learned when he accidentally force-pushed to origin main, which brought most of the engineering team to a halt while we figured out what got blown away and reconstructed main. (To git’s credit, it is hard to do real irreversible damage; between the reflog and commits stored in your colleagues’ repos, you can almost always recover. It just may take some time to sort things out.)

I should also note that it is possible to disable force-pushing on a per-repository basis (unfortunately not per-branch, AFAIK, but there are hoops you can jump through).

BetterExplained

2012-12-21T20:25:26-08:00

Excellent explanations of some complex math & programming topics. My favorite: An Interactive Guide To The Fourier Transform.

Snow Fall

2012-12-20T22:32:38-08:00

Amazing multimedia (that word is so 90s, but nothing more appropriate comes to mind) story of a fatal avalanche at Stevens Pass. Just look.

TIL: John Hammond

2012-05-18T16:36:25-07:00

I don’t know how I managed to avoid learning about John H. Hammond until now. A recent Radiolab episode, Crossroads, introduced him to me.

John Hammond was responsible for discovering and promoting some of the greatest musical talent of the 20th century: Benny Goodman, Count Basie, Billie Holiday, Pete Seeger, Aretha Franklin, George Benson, Leonard Cohen, Bob Dylan, Stevie Ray Vaughn, and Bruce Springsteen, among others. He was also responsible for introducing the world to the music of the great blues man Robert Johnson, who was the focus of the Radiolab piece.

On top of that, he was a civil rights pioneer, playing a large role in integrating the music industry.

(Incidentally, he appears to be no relation to Laurens Hammond, the inventor of the Hammond organ.)

A Year with MongoDB

2012-04-13T13:45:22-07:00

Nice summary of issues with running MongoDB at scale.

rise4fun

2012-04-13T13:44:16-07:00

Software engineering tools from Microsoft Research.

The Rise of the New Groupthink

2012-01-14T09:12:36-08:00

“Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption.”

Learning and the Narcissism of Small Differences

2011-12-28T14:41:15-08:00

I started working with python full-time about a year-and-a-half ago. I was coming from having mostly done ruby and javascript for the last few years, and my prior exposure to python was relatively brief.

I fairly quickly found a bunch of annoyances in python that I carped about frequently. Can’t resolve circular references? Ruby has no problem with that. Class and static methods tacked on via decorators? Those should be first-class concepts in any modern programming language. Not to mention that the python style of programming (and the circular reference issue) encouraged having multiple classes in a single file, or even files with bare, classless functions (i.e. modules). And what kind of crap is “if __name__ == '__main__':”? Yeah, I did a lot of grumbling.

But a funny thing happened. I kept working with it, learned to do things the “python way,” and most of these things turned out to be not such a big deal. I found a lot to like about python. The lack of extraneous characters for defining blocks (a.k.a. significant whitespace) was refreshing, and made for more readable code. It has a deep and well-documented standard library, and the quality of external libraries I’ve come across has generally been extraordinarily high. I had come to enjoy writing python.

When learning something new, the natural reaction is to compare it to what we already know. That’s how the brain works, by making associations. The easy trap to fall into, however, is jumping to the conclusion that because it doesn’t quite fit the patterns that you’ve learned–because it is different–it is inferior or broken. This is often compounded by the normal frustrations any beginner encounters, especially when you’re in an environment where you’re expected to hit the ground running.

I believe this is related to what Sigmund Freud termed the narcissism of small differences, the idea that people with relatively minor differences in viewpoint tend to be more combative than those with major differences.

Over time, as you learn and start to see more of the whole picture, those annoying differences become smaller and smaller, often disappearing entirely. In fact, you’ll likely come to appreciate some of those differences.

It takes effort to fight those initial narcissistic tendencies. The next time you’re thrown into a foreign environment, make a conscious effort to keep an open mind, and bite your tongue until you’ve given it some time. After you’ve mastered the subject, then you’re entitled to bitch about it.

Christopher Hitchens, 1949 - 2011

2011-12-16T14:08:40-08:00

“To be the father of growing daughters is to understand something of what Yeats evokes with his imperishable phrase ‘terrible beauty.’ Nothing can make one so happily exhilarated or so frightened: it’s a solid lesson in the limitations of self to realize that your heart is running around inside someone else’s body. It also makes me quite astonishingly calm at the thought of death: I know whom I would die to protect and I also understand that nobody but a lugubrious serf can possibly wish for a father who never goes away.”

– Christopher Hitchens, “Hitch-22”

Apache Considered Harmful

2011-11-23T11:15:52-08:00

A recent article by Mikeal Rogers about the Apache Software Foundation’s outmoded idea of open source contributions struck a chord with me.

Some time ago I submitted a pull request to Flume, making a very minor change to get something to compile again after code reorganization had broken it. About a week later, I got a comment from one of Cloudera’s engineers saying that the patch looked good, but that since they were in the process of moving to Apache Incubator, could I follow some extra steps. The extra steps (create an account on Cloudera’s JIRA issue tracker, create an issue for the bug, generate a patch for my change and attach it to the issue) weren’t terribly onerous, but considering I had already moved on (and in fact had decided not to use Flume for my project), and had real work to get done, I put it on the back burner and eventually forgot all about it.

The genius of git and GitHub is how easy it is to contribute to projects. Fork, fix, submit. I submit patches to projects all the damn time because barrier to contribution is so low. Like most developers, I’ve got a job that keeps me busy, and any roadblocks big enough to take me out of my flow are usually going to stop me.

Someone more patient than I submitted a “proper” patch for my Flume issue about a month later, so my conscience is clear. I hope, however, that the ASF eventually embraces the new open source reality. Their software will be better for it.

Google Analytics A Potential Threat to Anonymous Bloggers

2011-11-17T04:42:05-08:00

Interesting post from Andy Baio about tracking down an anonymous blogger via Google Analytics, and the trickiness of remaining anonymous in general.

In about 30 minutes of searching, using only Google and eWhois, I was able to discover the identities of seven of the anonymous or pseudonymous bloggers, and in two cases, their employers. One blog about Anonymous’ hacking operations could easily be tracked to the founder’s consulting firm, while another tracking Mexican cartels was tied to a second domain with the name and address of a San Diego man.

Cracking Siri

2011-11-15T13:57:43-08:00

Reverse-engineering Siri’s wire protocol.

How Dark Sky Works

2011-11-09T10:13:57-08:00

Interesting approach, showcasing the power of open source machine learning and computer vision libraries.

On Parenthood

2011-10-24T12:40:12-07:00

Someday I’ll write my own take on parenthood, and I hope it’s half as good as Jeff Atwood’s.

As an aside, as the parent of twins, that pie chart is still pretty accurate—only it’s twice as big.

Thump Phreaking

2011-10-19T13:41:28-07:00

Proof-of-concept out of Georgia Tech using the iPhone 4’s accelerometer as a keylogger.

Jewish [Math] Problems

2011-10-10T21:33:31-07:00

This is a special collection of problems that were given to select applicants during oral entrance exams to the math department of Moscow State University. These problems were designed to prevent Jews and other undesirables from getting a passing grade. Among problems that were used by the department to blackball unwanted candidate students, these problems are distinguished by having a simple solution that is difficult to find. Using problems with a simple solution protected the administration from extra complaints and appeals. This collection therefore has mathematical as well as historical value.

via @b6n

Dart: a language for structured web programming

2011-10-10T05:47:26-07:00

Dart is a new web programming language from Google. See also the blog post introducing Dart.

The Man Who Inspired Jobs

2011-10-08T15:12:19-07:00

“Market research is what you do when your product isn’t any good.”

Thank you, Steve

2011-10-05T17:49:37-07:00

My first home computer was an Apple IIe, in 1984. I had already started learning to program (on a TRS-80 Model III), but that IIe was where I really got excited about computers and programming. I wrote utilities and games in BASIC, then 6502 assembly language (I would write the assembler instructions down on paper, then translate those by hand to the byte code), hacked games (Bard’s Tale character editor, anyone?), and even taught programming classes to other kids.

The Steves, Jobs and Wozniak, were always heroes of mine, although it wasn’t until I was older that I truly appreciated Jobs’ brilliance. I always thought that Woz, the engineer, was the one who made the magic happen, but if it weren’t for Jobs, Apple could well have faded with Beagle Bros into obscurity.

Like a majority of people I know, I use Apple products daily, for work, entertainment, and communication. I can’t think of another company that puts as much thought into the quality of their products, both in terms of construction and user experience, and there’s no doubt that Steve Jobs is the one to thank for that.

I still have that IIe, in a suitcase in my garage. I think I’ll boot it up tonight in tribute. Thanks for everything, Steve. Rest in peace.

Update:

Mozilla's Secure Coding Guidelines for Web Applications

2011-10-03T15:49:52-07:00

O'Reilly's Open Feedback Publishing System

2011-09-25T14:46:39-07:00

Mentioned previously, O’Reilly’s Open Feedback Publishing System is a great resource for developers, and a brilliant move my O’Reilly to get early feedback and corrections on their books before they go to print.

Qu

2011-09-23T20:48:00-07:00

Qu is a new Ruby queuing and background job-processing library from Brandon Keepers, one of the maintainers of delayed_job.

Experiment in Extreme Schooling

2011-09-20T14:42:00-07:00

Fascinating article about a New York Times writer who moves to Russia with his family, and immerses his kids in a Russian school.

What really resonated with me is just how resilient kids are. I often catch myself thinking that my own girls aren’t ready for something, that it will be too hard for them, but that’s a dangerous attitude to have. Much better to believe that they can do anything, because chances are, they can.

Storm

2011-09-20T14:04:40-07:00

Storm is a distributed “realtime” computation system from Twitter.

The Kid Should See This

2011-08-28T14:20:00-07:00

A great collection of “off the grid-for-little-kids” videos. “It’s most likely not stuff that was made for them…But we don’t underestimate kids around here.”

Time

2011-07-31T14:33:15-07:00

Must-read–especially for programmers–notes about time.

HBase: The Definitive Guide

2011-07-24T14:39:41-07:00

Great guide to HBase, available free (for the time being) online via O’Reilly’s Open Feedback Publishing System.

Pandoc

2011-07-17T21:13:57-07:00

Pandoc is a universal document converter. Incredibly useful, especially for converting Markdown to HTML, PDF, etc.

PatrickJMT: Just Math Tutorials

2011-06-17T21:23:16-07:00

Free video tutorials on hundreds of math topics.

NASA - The Frontier Is Everywhere

2011-01-17T08:07:55-08:00

My Milkshake

2011-01-15T04:07:00-08:00

Seen at Trueburger, Oakland

Yahoo Account Hacked

2011-01-14T01:43:23-08:00

My Yahoo account was hacked this morning. The first sign that something was up was an email from Yahoo saying that an email address was added to my account. The address was “bradley_greenlee@yahoo.com”. Tricky. It definitely gave me pause. That’s my name, but I didn’t think I had that account. I went in and deleted it.

Suddenly Adium went nuts, popping up 20+ dialogs saying “<username> has (retroactively) denied your request to add them to your list.” Huh. Then came the first of a number of emails from friends (basically, Y! IM friends) saying that they had gotten an email from me saying that I was stuck in Wales without my creditcard and passport yada yada. Yup, hacked.

I was already on my way to changing my password. Unfortunately, my Yahoo account was still tied to my old AT&T DSL account, and I had to remember the username for that before I got in (this actually probably saved my butt, preventing the attacker from changing my password). I did eventually get in, though, and changed it to a nice, long random string.

So, how did this happen? Well, Yahoo was my spam-bucket account–I used it on a lot of throwaway sites that needed an email address. Unfortunately, I never changed the password from a common one I use on a lot of those same throwaway sites (for sites that I actually care about, I generate a random password, which I store in 1Password). Stupid. I’m sure one of those sites got compromised and were storing their passwords in plaintext.

I don’t think much damage was done, other than spamming a lot of Y! IM friends. The upside was that I got back in touch with some friends I hadn’t talked to in ages.

The lesson here is to never trust any site to keep your password secure. Count on it getting compromised at some point. Use a random password for each one, and a tool like 1Password or LastPass to keep track of them.

Most important, never use the same password on your email accounts that you do on any other site. If your email account gets compromised, the attacker can use that account to gain access (via “forgot password”) to any other sites that use that email address.

Digital Publishing Isn't Dead Yet

2010-12-29T07:05:25-08:00

ReadWriteWeb posted a short article today, The iPad Not a Savior of Magazines, as Digital Sales Continue to Fall. Looking at the current sales trend is pretty silly, however. Of course you’re going to see a big drop-off. Early adopters are eager to check out The Future, but people aren’t going to continue to pay newsstand prices for digital editions of magazines, especially if they’re already paying for a print subscription.

I don’t think we’ll see much improvement until both Apple and the publishers get over themselves and come to an agreement on subscription pricing. At issue is how much access the publishers get to their subscribers’ data. Traditionally, what keeps print publications alive is their ability to resell their subscribers’ information to advertisers. But Apple isn’t about to give that up. So either publishers need to accept that they’re going to have to give up some control in order to play in Apple’s ecosystem, or Apple needs to loosen its grip a bit (less likely).

Still, I’d wager that in 5 years, digital subscription sales will surpass print.

RSA Animate - The Secret Powers of Time

2010-10-08T03:59:35-07:00

The Swell Season covers Neutral Milk Hotel

2010-08-13T02:41:12-07:00

Installing Ruby 1.9.2 via rvm on OS X

2010-07-29T13:34:10-07:00

It took me a bit of digging to figure this out (it kept failing with readline errors, and then iconv went missing), so I thought I’d share:

rvm package install readline
rvm package install iconv
rvm install 1.9.2 -C --enable-shared,--with-iconv-dir=$HOME/.rvm/usr,\
--with-readline-dir=$HOME/.rvm/usr

Update: The package command has been renamed pkg in newer versions of rvm.

For Jessica

2010-07-29T01:58:15-07:00

A couple of weeks ago, a friend of mine and I were talking about a study she’d just read, which concluded that people without children were happier than people with children; or, to put it more precisely, despite what conventional wisdom holds, the study found that having children did not increase anyone’s happiness.

At which all I could do was burst out laughing. Because, well. Duh.

Only an academic would undertake a study like this, defining happiness as something along the lines of “satisfaction with life” and “feeling rewarded by your work.” If there’s an occupation more likely to make you feel incompetent and unrewarded than being a parent, I have never heard of it.

The Top Idea

2010-07-21T06:01:50-07:00

The American Scholar: Solitude and Leadership

2010-07-10T09:52:12-07:00

If you want others to follow, learn to be alone with your thoughts

Michigan Rain

2010-07-05T11:10:06-07:00

[Flash 9 is required to listen to audio.]

OS X Screen Sharing over SSH

2010-06-18T07:27:15-07:00

I recently set up a Mac Mini at home, and have found using Screen Sharing to access it invaluable. Not wanting to open up anything other than SSH on my home router, I created the following script to open up a Screen Sharing session over SSH:

Peter Callesen - A Single Sheet of Paper

2010-06-03T01:18:32-07:00

Raganwald: Optimism

2010-05-18T07:10:22-07:00

Zeldman: Life is Beautiful

2010-05-11T13:53:05-07:00

"Oh Mary" by theresa andersson

2010-05-07T14:17:59-07:00

Richard Feynman and The Connection Machine

2010-03-22T08:46:05-07:00

I 💖 Feynman

Panic Cakes

2010-03-19T14:43:49-07:00

via this isn’t happiness

New York Times: Christoph Niemann's Abstract City

2010-03-11T14:44:00-08:00

Out Of A Forest

2010-03-07T15:48:51-08:00

This was the first time I had heard The National. I immediately went out and bought Boxer.

Pivot

2010-02-09T02:49:00-08:00

Crash

2010-02-05T02:16:32-08:00

</param> </param> </param> </param></embed>

Opera 10 Anchor Tag Rendering Bug

2010-02-04T14:59:05-08:00

For some time now, Wesabe’s transaction edit dialog has been broken in Opera (10.10 as of this writing). Since it was fine in all other major browsers, and none of us were Opera users, fixing it was never high priority. I thought I’d take a look at it again tonight, though, and just when I was about to give up and go to bed, I managed to track it down.

If you have an anchor (<a>) tag that contains block-level elements (which is technically not allowed, but all other browsers seem to be ok with it), such as  or <div>, the anchor will close itself before the elements it is supposed to contain. E.g.: <a> hello </a> Will render as: <a> </a> hello

If you give the anchor an href, though, even if it is just a blank string (<a href="">...), it renders fine.

Here’s a page that demonstrates the bug: http://footle.org/misc/opera-bug.html.

Anyway, I filed a bug with the Opera folks. Time for bed.

Better Dictionary: Wordnik

2009-12-22T02:53:39-08:00

Ok, thanks to Google for putting their dictionary out there, which I wrote about a couple of posts ago, but the quality of the results pales in comparison to Wordnik, which I’ve just discovered. It won’t do language translation for you, but as far as the English language is concerned, this is pure word porn. So here’s an updated bookmarklet for you:

wordnik

Just drag that to your bookmarks bar, select a word or phrase on the page, and click. Happy bonus: just clicking on the bookmarklet without highlighting a word first will give you a random word (thanks to Wordnik, not me). Cool!

Chrome Frame's Shifty User-Agent

2009-12-15T06:49:48-08:00

One of the security measures we have in place at Wesabe is to invalidate a session and log the user out if their User-Agent or IP address subnet changes between requests. However, we recently had a number of complaints that users were getting logged out prematurely. Looking at one user’s session in the logs, I noticed that their user agent string had “chromeframe/4.0” appended for some requests, but not others. It turns out that Google’s Chrome Frame only modifies the user agent for top-level requests, and not for subsequent requests sent as that page loads:

It was a compromise to keep code complexity down. In order to tag every request from IE...we have to have hooks in place at many more places (not to mention supporting the different things IE6, 7 and 8 do). So, as a compromise we decided to keep it down to only the single hook that allows us to tag top level requests.

This fix on our side was to just strip the chromeframe identifier from the user agent string, although it makes me wonder what other browser extensions cause similar issues and whether invalidating a session when the user agent changes is even a worthwhile security measure.

Google Dictionary Bookmarklet

2009-12-04T07:52:25-08:00

Google recently launched a decent dictionary, so for you logophiles, here’s a bookmarklet that will bring up the definition of whatever text you have selected on the page: gDefine. Just drag it to your bookmarks toolbar, then double-click a word somewhere and click the button to get a definition.

It is set to use the English dictionary, but if you want to use it in a different language (or even translate between languages), just change the langpair parameter from en|en to, say, fr|fr for French, de|de for German, etc. (try a search on the Google Dictionary site in the language you want and look to see what the langpair parameter is in the url). Also, the hl parameter controls the language of the results page, so if you want your results to be in that language as well (as opposed to English), change that accordingly.

Here are a few to get you started:

gDefine (French) gDefine (Spanish) gDefine (German)

And translations:

gTranslate (French>English) gTranslate (English>French)

Enjoy!

Bill Cosby on Drums

2009-11-12T03:45:50-08:00

Wanted: 7 Fearless Engineers!

2009-11-10T22:32:34-08:00

(via Rich Coad)

CYOA

2009-11-10T14:16:20-08:00

Impressive analysis of Choose Your Own Adventure books, a fixture of my childhood.

Cell Size and Scale

2009-10-29T06:49:54-07:00

We Could Use More of This

2009-10-28T14:39:00-07:00

There's a weekly Tuesday evening Ultimate Frisbee game at a field near my house (sadly, I haven't gotten out to play in ages). Now that it's getting dark early, they need the lights on in order to play. The city is usually pretty stingy with the lights, but this just came through on the mailing list:

So here's the deal with the lights for the lower field...The lady (I forget her name) wants us to help out with getting supplies for their cooking class: taco stuff, corn dogs and fries (maybe potatoes), chicken and rice, stuff like that to teach cooking with. The reason is, the state is cutting back on funding and she asks us for donations of food to help out with the class. In turn she said we could have the field and lights as long as we turn off the lights when we're done and if the soccer teams try to take over she will deal with them for us. Not a bad deal, plus we're able to help the families and do a barter.

So: Bring food and play Tuesday night!

I love it.

A literary appreciation of the Olson/Zoneinfo/tz database - Jon Udell

2009-10-25T02:13:34-07:00

Bill Sosin | Siong Chin

2009-09-30T13:41:39-07:00

The Billion Dollar Gram | Information Is Beautiful

2009-09-30T13:40:28-07:00

Billions spent on this. Billions spent on that. It’s all relative, right?

Muni platform, Embarcadero Station

2009-08-12T03:16:00-07:00

Muni platform, Embarcadero Station

MySQL-Style Output for the Rails Console

2009-03-01T10:42:47-08:00

While poking around the database in my rails console, I often found myself jumping to the mysql console just so I could get an easier-to-digest view of the data. I finally had enough of this silliness and wrote up a quick function to dump of set of ActiveRecord objects in the mysql report format.

>> report(records, :id, :amount, :created_at)
+------+-----------+--------------------------------+
| id   | amount    | created_at                     |
+------+-----------+--------------------------------+
| 8301 | $12.40    | Sat Feb 28 09:20:47 -0800 2009 |
| 6060 | $39.62    | Sun Feb 15 14:45:38 -0800 2009 |
| 6061 | $167.52   | Sun Feb 15 14:45:38 -0800 2009 |
| 6067 | $12.00    | Sun Feb 15 14:45:40 -0800 2009 |
| 6059 | $1,000.00 | Sun Feb 15 14:45:38 -0800 2009 |
+------+-----------+--------------------------------+
5 rows in set

Grab it from GitHub and stick it in your .irbrc.

Checking for STDIN in ruby

2008-08-21T10:24:06-07:00

A colleague was asking today how he could see if there’s data waiting on STDIN in ruby (on Linux). On OS X, this is pretty straightforward:

if $stdin.stat.size > 0
  puts "got something: #{STDIN.read}"
else
  puts "nada"
end

That doesn’t work in Linux, though. After some digging, I found this post, which lead to:

require 'fcntl'

flags = STDIN.fcntl(Fcntl::F_GETFL, 0)
flags |= Fcntl::O_NONBLOCK
STDIN.fcntl(Fcntl::F_SETFL, flags)

begin
  puts "got something: #{STDIN.read}"
rescue Errno::EAGAIN
  puts "nada"
end

Which works on both platforms, but is (a) ugly (catching an exception), and (b) requires you to actually try to read from STDIN.

In playing around with that, though, I noticed that STDIN.fcntl(Fcntl::F_GETFL, 0) returned 0 if there was something on STDIN, and something non-zero (2 in OSX and 32770 in Linux) if STDIN was empty. So now the code is simple again:

require 'fcntl'

if STDIN.fcntl(Fcntl::F_GETFL, 0) == 0
  puts "got something: #{STDIN.read}"
else
  puts "nada"
end

I’m not sure how reliable that is on other platforms (it won’t work on Windows—fcntl is a Unix-y thing). If I’m understanding fcntl.h correctly, a return value of 2/32770 (0x8002) seems to indicate that the io stream in question is open for writing. I’m not sure if that is intended or just a side-effect. Anyone know?

Leopard External Monitor Annoyance Solved...aka Hide All Apps on a Mac

2008-03-21T11:20:40-07:00

One of the more annoying things about Leopard is that when you plug in an external monitor that is set up to be your primary, your open applications don’t move over to the monitor. I was having to manually drag everything over. Then I realized that if I hid all my apps before plugging in the monitor, when I brought them up again, they’d be in the right place.

So now the question was: is there an easy way to hide all your apps without cycling through them all? It took a bit of digging, but it turns out there is. Cmd-option-click on an app in the Dock and it will hide all other apps. So I tried Cmd-option-click on the desktop itself and…voilà!

Update: Technically, you’re Cmd-option-clicking on the Finder, so if you have any Finder windows open, they won’t hide. Bummer.

ConditionsConstructor

2008-02-27T00:04:59-08:00

I hadn’t posted this before because it seemed kind of trivial and silly, but I used it again the other day and decided maybe I should share the joy after all.

Far too often I find myself jumping through hoops building ActiveRecord find conditions arrays dynamically depending on what arguments were passed in to the method. So I cooked up a simple little class to clean that up a bit:

# class to construct conditions for AR queries

# cc = ConditionsConstructor.new('foobar &gt; ?', 7)

# cc.add('blah is not null')

# cc.add('baz in (?)', [1,2,3])

# cc.conditions

# # => ["foobar &gt; ? and blah is not null and baz in (?)", 7, [1, 2, 3]]

class ConditionsConstructor
  def initialize(str = nil, *args)
    @conditions_strs = str ? [str] : []
    @conditions_args = args
  end
  
  def add(str, *args)
    @conditions_strs &lt;&lt; str
    @conditions_args += args
  end
  
  def conditions
    [@conditions_strs.join(' and ')] + @conditions_args
  end
end

Tracking the 123 Meme

2008-02-16T04:06:24-08:00

I'm pretty late to the party, but I just ran across this on Chuck Hoffman's blog: "Grab the nearest book, open to page 123, go down to the 5th sentence, and type up the 3 following sentences."

Before I do that, I thought it would be interesting to try to track this meme back as far as possible. For the sake of my sanity, I'm not following multiple paths from posts that link to more than one source--I'll just pick the link that seems to be the primary source. So here goes:

nothing happens linked to Knowing and Doing linked to Exploration Through Example linked to a thaumaturgical compendium linked to michaelzimmer.org linked to Chronicles of Dissent linked to Fergie's Tech Blog linked to Threat Level linked to Danger Room linked to Abu Muqawama linked to Afganistanica: Mountain Tourism in Afghanistan linked to [My] State Failure Blog linked to Kings of War linked to MountainRunner linked to zenpundit.com linked to the glittering eye who gave a bad link that I had to track down to Independent Liberal linked to Donklephant linked to Stubborn Facts linked to another post on the same blog, linked to Sideways Mencken, who did not link, but referenced Internet Ronin linked to AmbivaBlog [and now I'm starting to tire of this game] linked to The Anchoress linked [not directly, had to dig it up *sigh*] to Some Have Hats linked to [again not directly...grr] Church of the Masses linked [incorrectly] to The Daily Grotto [warning! annoying chanting] linked to Aussie Coffee Shop linked to Where Angels Go which mentioned, but failed to link to the person who tagged them, so it's a dead end.

Whew. This was getting a bit tedious. Not to mention that it seemed to have gotten firmly entrenched in religious/pro-life blogs, which were getting on my nerves.

Just as well. Turns out Google returns almost half a million hits for "page 123" sentence, and there doesn't seem to be a way to sort results by date (the daterange operator failed me completely). The oldest post I could find on Google Blog search was this one, from November 24, 2004 (albeit for a slight variation on the meme--just the fifth sentence, not the three following it), but which clearly isn't the origin. Seems it was a meme floating around on LiveJournal around then. If anyone has better search-fu and can find an earlier post, let me know.

At this point, I've made myself pretty sick of this meme. Often it appears to be a means by which people can brag about whatever erudite text they may or may not actually be reading. Even so, I can't have wasted this much time and not play along. I've got two books nearly equidistant from my keyboard now. One is, at least in my field, too mundane to bother with: the Pickaxe book. The other sports the following 123/5/3 sentences:

The latissimus dorsi muscles have a major role in the deadlift: from the floor, the lat pulls back on the humerus to keep the arms from swinging forward away from the shins, and acts as an anchor on the upper part of the humerus to maintain the position of the bar directly under the shoulder blades until the bar crosses above the knees. The lats act in an essentially isometric way from the floor to the point where hip extension allows the arms to become vertical. At this point tension comes off the lats, and as the back becomes vertical, the arms drag the bar into the thighs as they assume an angle behind the vertical, opposite the starting position.

That's from Starting Strength by Rippetoe and Kilgore, which is a great book for learning how to do basic barbell lifts correctly (something we do a fair bit of in Crossfit). Coincidentally, that's just a couple pages past where I was actually reading about the deadlift. (Most interesting tidbit I've learned so far from the book: the importance of a solid grip when doing the dealift. If your hands start slipping, proprioceptive (love that word) feedback tells the back that what you're trying to lift is too heavy, and it basically gives up.)

I'm not going to inflict this on anyone else, but if you're reading this, feel free to prate on about whatever book you have near you.

Cheating for Fun

2007-12-05T01:16:58-08:00

I recently checked out the Scramble application on Facebook, which is a knock-off of Boggle. It’s a fun diversion, even though I usually get my ass kicked. There’s an annoying chat window there, too, and invariably someone will accuse someone else of cheating, at which point a number of people will pipe up with “how would you cheat?”

This got me thinking…how would you cheat? You’d just have to write a Scramble puzzle solver that spit out all the words in on the board. That sounded like a fun problem. I hadn’t done anything with recursion in a while, so I gave it a go. Below is what I came up with. You’ll need a word list to run it. I originally used /usr/share/dict/words, but it was missing a lot of valid words (I think it only has base words, no plurals or tenses). So I stole the scrabble word list from the Debian scribble package, which worked much better.

Anyway, here you go:

I did try this out a couple times on Scramble and won handily. There’s really not much fun in that, though.

Note: This was really just an exercise in coming up with a fun algorithm, and is primarily intended for programmers. If you’re just looking to cheat at Boggle/Scramble, there are plenty of web sites that will help you with that, and with a lot less effort.

Lesser-Known MySQL Performance Tips

2007-11-28T10:02:33-08:00

"dolphins" by talkrhubarb

I thought I knew a fair bit about MySQL before I started working on Wesabe, but I was wrong. Once your tables start to get really massive, all sorts of fun issues come out of the woodwork. There are a thousand articles out there on the basics of database optimization (use indices, denormalize when necessary, etc.), so I'm not going to cover those again. Here are some tips--most specific to MySQL--that I've learned on the job:

MySQL only uses one index per table when doing a query, and the query optimizer doesn't always pick the best column, so indexing everything is not necessarily a great idea. If you're always selecting on a certain set of columns, create a single multi-column index
related to the above, don't put indices on columns with a low cardinality. You can check the cardinality of a column by doing SHOW INDEX FROM table_name
if any text or blob fields are part of your SELECT query, MySQL will write any temp tables to disk rather than memory. So where possible, use large VARCHAR fields (on 5.0.3 or higher) instead. See: http://railspikes.com/2007/9/14/one-reason-why-MySQL-sucks
if you need to do case-sensitive matching, declare your column as BINARY; don't use LIKE BINARY in your queries to cast a non-binary column. If you do, MySQL won't use any indexes on that column.
SELECT ... LIMIT offset, limit, where offset is a large number == bad, at least with InnoDB tables, as MySQL actually counts out every record until it gets to the offset. Ouch. Instead, assuming you have an auto-incremented id field, SELECT * FROM foo WHERE id >= offset AND id < offset+limit (props to Coda for that one).

Update:

If you have a query in which you're looking for the most recent n items of something—e.g. SELECT * FROM foo WHERE bar = ? ORDER BY created_at DESC LIMIT 10)—your query will be significantly faster if you create a multi-column index on whichever field(s) you're selecting on plus the timestamp field that you're ordering by (in this case, CREATE INDEX idx_foo_bar_created_at ON foo (bar,created_at)). This allows MySQL to look up the records directly from the index, instead of pulling all the records where bar = ?, sorting them (probably first writing them to a tempfile), and then taking the last 10. EXPLAIN is your friend. If you're seeing "Using temporary table, Using filesort" in the "extra" section of EXPLAIN, that's bad.

I’ll add more as we come across them. If you have any other non-obvious tips, leave them in the comments or email me (brad@ this domain) and I’ll add them too.

By the way, an invaluable book for learning how to squeeze the most out of MySQL is High Performance MySQL, by Jeremy Zawodny and Derek Balling.

Tumblr

2007-11-05T01:02:32-08:00

I decided to give tumblogging a try. I’m hoping that ease of use and lowered expectations will get me to post more often, and I’m liking it so far. It covers the large middle ground between blogging and twittering. I can put up something interesting without feeling like I need to construct a coherent post around it, and I don’t feel like I’m pestering people with twitters (not to mention that it allows for richer media types).

I realize I could just do the same thing on this blog if I wanted to, but there’s something about the minimalist formatting and ease-of-use that just seems to work. So I’ll continue to post longer pieces here (I’ve got a number that I need to just sit down and write), and I’ll keep posting inane comments to Twitter, but I imagine that the majority of interesting things I find will end up on my tumblelog.

Social Graphs Must Die

2007-09-22T16:24:15-07:00

Chris Wanstrath made a perfectly reasonable request earlier today, so I came up with the following Greasemonkey script:

// ==UserScript==

// @name           Social Graphs Must Die

// @namespace      Footle

// @description    Redact all mentions of "social graphs";

// ==/UserScript==

document.body.innerHTML = document.body.innerHTML.replace(/(social\s+graph(s?))/ig,"<span style='background:#000;color:#000'>$1</span>");

Actually closing a tab (that wasn’t opened by Javascript) in Firefox 2 isn’t possible AFAICT, but this is more fun anyway.

I Get No Spam

2007-08-14T11:26:00-07:00

After hearing a couple complaints from friends about the amount of spam they’re getting, I decided to take a quick look to see where I stood. At the risk of setting myself up, I should say that I get no spam (channeling John C. Dvorak). Or at least very little–maybe one every other day gets through to my inbox.

I think the trick is just to have multiple levels of filtering. I use a Gmail address for most online transactions (for online transactions that I really don’t care about–if I’m doing a one-time registration just to get a trial key, for example–I use my Yahoo address, which is nothing but a spam bucket). My Gmail address gets forwarded to my personal email address (@footle.org), which has SpamAssassin running on the server. If it makes it past SA to Mail.app, it gets inspected by SpamSieve, an awesome Bayesian spam filter for OS X mail clients.

Anyway, my stats for the last 24 hours:

Server-level filters (spam caught):
- SpamAssassin: 281
- Gmail: 32
Client-level filter (spam caught):
- SpamSieve: 17
Spam getting to my inbox: 0
Non-spam messages: 52
False positives: 1 (just Flavorpill, so not a big deal)

(Wow…330 pieces of spam in a single day. That seems way worse than when I last checked.)

Idea-That-I-Don't-Have-Time-To-Work-On of the Week

2007-08-09T01:47:48-07:00

Bonjour-enabled Git. (Although Coda tells me that Dave F. at Powerset already approached him with this idea. Neither of us have time to work on it.)

I started using Git recently and although it’s a bit confounding at times, I love how cheap and easy branching is, and that you can commit incremental changes to your local repository, then later commit them in batch to subversion. Git allows for distributed development, but unless you’re on the same server as other developers, or you go through the trouble of mounting their repository over the network, you can’t check out their repository. Seems like it shouldn’t be too difficult to give Git Bonjour powers.

Update: Chad Fowler has started such a project.

Binary multipart POSTs in Javascript

2007-07-31T15:32:57-07:00

We recently released a very slick Firefox extension at Wesabe. It was written by my colleague Tim Mason, but I helped figure out one small piece of it—namely, how to do binary multipart POSTs in Javascript—and since it involved many hours of hair-pulling for both of us, I thought I’d share the knowledge. (Tim says I should note that “this probably only works in Firefox version 2.0 and greater and that it uses Mozilla specific calls that are only allowed for privileged javascript—basically only for extensions.”)

One of the cool features of the plugin is the ability to take a snapshot of a full browser page and either save the snapshot to disk or upload it to Wesabe (so you can, for example, save the receipt for a web purchase along with that transaction in your account). The snapshot is uploaded to Wesabe via a standard multipart POST, the same way that a file is uploaded via a web form.

Tim was having trouble getting the POST to work with binary data at first, and he had other things to finish, so he wanted to just base-64-encode it and be done with it. I was reluctant to do that, as the size of the upload would be significantly larger (about 137% of the original). Also, Rails didn’t automatically decode base-64-encoded file attachments. But Tim had other bugs to fix, so I submitted a patch to Rails to do the base-64 decoding. I was pretty proud of this patch until it was pointed out to me that RFC 2616 specifically disallows the use of Content-Transfer-Encoding in HTTP. Doh. They also realized that it is a colossal waste of bandwidth.

Since Tim was cramming to meet a hard(-ish) deadline set for the release of the plugin, I offered to lend my eyeballs to the binary post problem. This could be a very long story, but I’ll just get to the point: you can read binary data in to a Javascript string and dump it right out to a file just fine, but if you try to do any concatenation with that string, Javascript ends up munging it mercilessly. I’m not sure whether it is trying to interpret it as UTF8 or if it terminates it as soon as it hits a null byte (which is what seemed to be happening), but regardless, doing "some string" + binaryData + "another string", as is necessary when putting together a mutipart post, just does not work.

The answer required employing Rube Goldbergian system of input and output streams. The seed of the solution was found on this post, although that didn’t explain how to mix in all of the strings needed for the post and MIME envelope. So here it is, in all it’s goriness:

Update: Spaces removed from multipart boundary per Gijsbert’s suggestion in the comments (thanks!).

Reloading Ruby Classes in Rails

2007-05-13T12:58:54-07:00

I ran across a bug in Date::Format today, and after spending a few hours hacking away at a fix (the date/format.rb code is uuuuugly and sloooow…someone should really rewrite that. Better yet, rewrite it in C), I thought I’d submit a patch. So I grabbed the ruby_1_8 branch and lo and behold, my issue had already been fixed!

So the question now was how to monkeypatch the entire Date::Format module? Simply require-ing it as a plugin doesn’t work, since Date::Format is already loaded at that point. The trick then is to use load instead of require.

First, I tried this:

load 'date/format.rb'

(note: load needs the actual filename; it doesn’t have the magic that require does) but that gave me an error:

in `load': wrong number of arguments (1 for 0) (ArgumentError)

Turns out Rails::Plugin::Loader defines its own load, so:

Kernel::load 'date/format.rb'

et voila!

Stop iPhoto from Launching when you Connect your Camera

2007-05-13T03:19:53-07:00

It took me a bit of digging to find this, so I thought I may as well spread the word here. I’ve started using Adobe Lightroom for managing my photos, but every time I connect my camera or card reader to import photos, iPhoto would launch, and I couldn’t figure out where I could change that. I finally discovered that the setting is in the Image Capture application. Run that, then go to Preferences and you’ll find it there. Why they wouldn’t also put it in iPhoto, Lightroom, or even on a preferences panel is beyond me.

Web 2.0 and Coming Soon...

2007-04-17T23:49:02-07:00

Marc and I presented our Super Ninja Privacy Techniques for Web App Developers talk again yesterday at Web 2.0 (we also did it at ETech last month), and it seemed well received. I still need to work on my presentation skills; I’m not an old hand at this like Marc is.

I’m working on a follow-up to my privacy wall post which will describe a much better way to go about keeping a user’s private data private, using an encrypted “Locker”. I’ll also go into detail about how we deal with password recovery.

Hopefully by preannouncing this, I’ll force myself to get off my duff and finish the post. So stay tuned, and bug me if I don’t have it up soon.

Google Takes Steps Towards Greater Privacy

2007-03-14T10:09:49-07:00

Google recently announced that it will soon start anonymizing search logs older than 18-24 months. Full details can be found in their Log Retention Policy FAQ (PDF). This is a heartening step back towards their “Don’t Be Evil” corporate philosophy, which some think has been largely abandoned.

I’ve just recently started using Scroogle as a way of defeating their tracking of my every search (their site is awful; Wikipedia has more readable information about the project), although the motives of the man behind it, Daniel Brandt, who also runs the Google Watch site, may be questionable. Still, he doesn’t have much incentive for keeping a log of queries and IP addresses, and if he did, since he’s not giving me a cookie, he can’t tie all my searches together.

Do Passwords Right With bcrypt-ruby

2007-02-28T10:25:53-08:00

My colleague Coda Hale just released a sweet bcrypt-ruby gem that does password hashing right. It also provides for future-proofing by enabling you to assign a computational cost for generating a password hash and letting you version your password hashes. You have no more excuses.

Protecting Your Users' Data with a Privacy Wall

2007-02-22T02:40:08-08:00

Just Another Brick In The Wall?
by Iain Cuthbertson

We deal with a lot of very private data at Wesabe, so security and privacy are our top concerns. In this post I will describe one of our primary means for assuring privacy, a technique that is general enough that any site can use it. Our creative name for this technique is the privacy wall. Later, I'll go on to tell you ways to hack the wall, just so you don't get too comfortable.

The Privacy Wall

The idea is simple: don't have any direct links in your database between your users' "public" data and their private data. Instead of linking tables directly via a foreign key, use a cryptographic hash [1] that is based on at least one piece of data that only the user knows—such as their password. The user's private data can be looked up when the user logs in, but otherwise it is completely anonymous. Let's go through a simple example.

Let's say we're designing an application that lets members keep a list of their deepest, darkest secrets. We need a database with at least two tables: 'users' and 'secrets'. The first pass database model looks like this:

The problem with this schema is that anyone with access to the database can easily find out all the secrets of a given user. With one small change, however, we can make this extremely difficult, if not impossible:

The special sauce is the 'secret_key', which is nothing more than a cryptographic hash of the user's username and their password [2]. When the user logs in, we can generate the hash and store it in the session [3]. Whenever we need to query the user's secrets, we use that key to look them up instead of the user id. Now, if some baddie gets ahold of the database, they will still be able to read everyone's secrets, but they won't know which secret belongs to which user, and there's no way to look up the secrets of a given user.

Update: A commenter on the Wesabe blog brought up the important point of what you do if the user forgets their password. The recovery method we came up with was to store a copy of their secret key, encrypted with the answers to their security questions (which aren't stored anywhere in our database, of course). Assuming that the user hasn't forgotten those as well, you can easily find their account data and "move it over" when they reset their password (don't forget to update the encrypted secret key); if they do forget them, well, there's a problem.

Attacking the Wall

I mentioned earlier that you store the secret key in the user's session. If you're storing your session data in the database and your db is hacked, any users that are logged in (or whose sessions haven't yet be deleted) can be compromised. The same is true if sessions are stored on the filesystem. Keeping session data in memory is better, although it is still hackable (the swapfile is one obvious target). However you're storing your session data, keeping your sessions reasonably short and deleting them when they expire is wise. You could also store the secret key separately in a cookie on the user's computer, although then you'd better make damn sure you don't have any cross-site scripting (XSS) vulnerabilities that would allow a hacker to harvest your user's cookies.

Other holes can be found if your system is sufficiently complex and an attacker can find a path from User to Secret through other tables in the database, so it's important to trace out those paths and make sure that the secret key is used somewhere in each chain.

A harder problem to solve is when the secrets themselves may contain enough information to identify the user, and with the above scheme, if one secret is traced back to a user, all of that user's secrets are compromised. It might not be possible or practical to scrub or encrypt the data, but you can limit the damage of a secret being compromised. My colleague and security guru Sam Quiqley suggests the following as an extra layer of security: add a counter to the data being hashed to generate the secret key:

secret key 1 = Hash(salt + password + '1')
secret key 2 = Hash(salt + password + '2')
...
secret key n = Hash(salt + password + '&lt;n&gt;')

Getting a list of all the secrets for a given user when they log in is going to be a lot less efficient, of course; you have to keep generating hashes and doing queries until no secret with that hash is found, and deleting secrets may require special handling. But it may be a small price to pay for the extra privacy.

Finally, log files can be a gold mine for attackers. There's a very good chance you're logging queries, debug statements, or exception reports that link users to their keys or directly to their secrets. You should scrub any identifying information before it gets written to the log file.

So That's It, Right?

The privacy wall is far from a silver bullet. Privacy and security are hard—really hard—particularly so if your app is taking private data and extracting information out of it for public consumption, like we are at Wesabe. The privacy wall is one of a number of methods we're using to insure that our users' private data stays that way. If you're lucky enough to be going to ETech next month, definitely check out Marc's session on Super Ninja Privacy Techniques for Web App Developers.

I hope you found this helpful. Let me know what you think; I appreciate any and all feedback. And if you've got any cool privacy techniques up your sleeve, share the knowledge!

[1] A cryptographic hash is way of mapping any amount of plain text to a fixed-length “fingerprint” such that the same text always maps to the same hash, and given a hash, it is impossible to generate the text from which it was derived. Hashes are wonderful things with many uses. If you’re a developer, and you didn’t already know this, stop reading now and go here or here, and learn how to generate a SHA1/2 hash in your programming language of choice. Come back when you’re ready. I’ll wait.

[2] You can throw in a salt too, to be safe; just make sure that you’re not using the same hash that you’re using for checking the user’s password. You are smart enough not to store passwords in plaintext in the database, aren’t you?

[3] Danger, Will Robinson! Keep reading.

Campfire Hacks: Uploading Screenshots with Quicksilver and Pyro

2007-02-04T06:39:02-08:00

We live in Campfire at Wesabe, so naturally we've developed a number of tools and hacks around it. One of these days I'll get around to writing about the Campfire bot framework we've developed, but right now I'll start with something simpler: a way to upload screenshots to Campfire with only a few keystrokes (riffing of my colleague Coda Hale's desire to eliminate the need for a mouse). Note that this is for Mac users only, so if you're one of the poor souls who hasn't seen the light yet, you can stop reading now.

Two prerequisites (other than a Mac): Quicksilver and Pyro. Quicksilver should be familiar to most Mac users already. It is an incredibly powerful application that can be quite obtuse but can save you gobs of time if you learn how to use it. Pyro is a client for Campfire. Why do you need a client for a browser-based chat app? Getting message notifications in your dock is one reason, but the biggest reason is that there seems to be a memory leak in Firefox that causes it to grind to a halt if you've had Campfire up for too long. But I digress.

You first need to enable the Screen Capture Actions plugin in Quicksilver (go to Plugins -> Recommended, and check Screen Capture Actions). Then to go Catalog -> Quicksilver and make sure that Internal Commands is checked). This should give you Capture Region, Window, and Screen commands.

Next, the secret sauce: a bit of AppleScript that uploads a file to Campfire via Pyro:

on open theFile
  tell application "Pyro"
    upload theFile to room "[your campfire room name]" in campfire "[your campfire].campfirenow.com"
  end tell
end open

Paste that into Script Editor and save it as PyroUpload in ~/Library/Application Support/Quicksilver/Actions (if this folder doesn't exist, create it), and restart Quicksilver.

Now you can upload a screenshot with just Cmd - Space - [Capture Window|Region|Screen] - Return - [take your screenshot] - PY - Return.

Have fun!

Must...stop...looking at...stats...

2006-11-17T15:19:12-08:00

I’ve been too tired today to do any actual work, so I’ve spent much of the day camping out on the Wesabe site stats. It’s terribly exciting having so many people hitting your site and so many signing up (almost 1/3 of our unique visitors have created accounts). The site has been humming along beautifully, too–major props goes out my colleague Coda Hale for his Apache/Mongrel/Pen prowess.

Anyway, I just wanted to share one of the more interesting stats from our analytics (Mint–very tasty):

That’s a lovely thing to see. Granted, these are largely very tech-savvy, early-adopter people at this stage, but it’s heartening to see IE getting the beatdown.

Wesabe Launches

2006-11-16T17:41:09-08:00

The reason I’ve had my head in the sand for the last six months is that I’ve been working on a new startup, and we’ve finally launched publicly. Check us out. Here’s a screencast demo of the service:

</param> </embed>

Let me know what you think. (Or better yet, send your thoughts to support@wesabe.com.)

Still here

2006-07-22T17:25:50-07:00

No, I haven’t abandoned this site. I’ve just been working my butt off at a new startup. I’ll try to get some content up soon, but in the meantime, enjoy some pictures of the newest member of our family.

Productivity in Java vs. Rails

2006-05-31T03:57:35-07:00

I am far more productive when writing Rails code than when writing Java. I just realized that one of the reasons for my lower productivity in Java is the need to recompile every time a make a change to a page on the site. In the 15 seconds or so it takes to recompile and redeploy to Tomcat, I get bored and am apt to go check my new favorite news site, popurls, or my RSS feeds, or (less likely) post to my blog. Suddenly those 15 seconds have become 5 minutes. And this happens many times throughout the day.

With Rails, I make a change, refresh my browser, and there it is. On to the next step.

HTTP Authorization with Apache/FastCGI

2006-05-24T10:56:12-07:00

It took me forever to figure this one out, but if you want HTTP Authentication to work with Apache 2 and mod_fastcgi, you need this in your apache conf file:

FastCgiConfig -pass-header Authorization

FastCGI doesn’t pass the Authorization header by default for some reason.

Processes, Threads, and Ruby

2006-05-23T06:50:40-07:00

While researching the best way to handle calling an external program from Ruby (and capturing stdout & stderr), I came across this post, which is a good review of how processes and threads work:

http://www.ruby-forum.com/topic/65155#75363

I still haven’t figured out exactly how I’m going to do this, but I’ll post it here when I figure it out. Ruby has a few different ways of opening and communicating with processes, but all seem to be lacking in some way or another. IO.popen lets you write to the process’ stdin, and read from its stdout, but you can’t get stderr without jumping through serious hoops (like redirecting stderr to a file and then reading the file…ugh). Open3.popen3 (brilliant naming) gives you stdin, stdout, and stderr, but the subprocess runs as a grandchild, so there seems to be no way to wait for it to finish.

Database War Stories

2006-04-30T01:46:58-07:00

There’s an interesting series of posts over at O’Reilly Radar. Tim O’Reilly asked people how they were using databases in their “Web 2.0” applications (although I think the Web 2.0 part of it is for the most part irrelevant). The responses so far have made for interesting reading. So far we’ve heard from Second Life, Bloglines and Memeorandum, Flickr, NASA World Wind, and craigslist. One of the lessons learned is that with a high-traffic site, at some point you have to break your database up so that the “hot” data is spread across a number of boxes. This got me thinking. It should be possible to build a tool that analyzes your database usage and, given a number of slave boxes to configure as it sees fit, automatically configures masters and slaves and distributes your data across those boxes as necessary. This would not be a one-time only process either; it would continue to monitor usage and performance and adjust accordingly. Certainly not an easy task, but should be doable.

Bad UI in iTunes

2006-04-25T08:58:27-07:00

I use iTunes both on a Mac and Windows. In OS X, Command-H hides an app, something I do so often I just do it without thinking. However, in iTunes on Windows, Control-H (aka backspace) deletes whatever object you’re on, with no undo. I just deleted a playlist I had spent a long time working on and I see no way to get it back.

Why don’t I at least have the option to undo? Terrible UI. Surprising that Apple of all companies made such a basic mistake.

Amazon's new S3 Storage Service

2006-03-15T03:10:05-08:00

Amazon just launched a new service, S3 - Simple Storage Service. It is a web service that allows you to store as much data as you like, with file sizes up to 5GB, and you just pay for the storage you use and the data transferred. Rates are very reasonable, too – $0.15/GB/month of storage, and $0.20/GB in data transferred.

This is pretty interesting. It gives developers the ability to create applications requiring significant storage space without having to make a huge upfront investment in equipment and expertise. Want to write your own Flickr? Go for it. Granted, it’s risky relying on a third party for a core part of your business, but you only need them until you get your million users and can get enough funding to build your own storage backend.

Google is apparently working on their own storage backend, Google Drive. It will be interesting to see how this plays out. Nothing but good news for aspiring entrepreneurs, though.

via TechCrunch

Multi-Touch Touch Screen

2006-02-27T02:45:23-08:00

One of the most exciting new technologies I’ve seen in a long time: a multi-touch touch screen. The video says it all.

</embed></param>

Your next ISP: Google

2006-02-08T10:13:10-08:00

John C. Dvorak has a good piece at pcmag.com about speculations that Google is going to be creating their own network (see the “Google is the Internet” scenario from the article linked in my previous post). I certainly hope they do, as telcos have been dragging their feet on broadband for a long time, and acting like the Mafia whenever someone encroaches on “their” territory.

Imagining the Google Future

2006-02-02T03:54:11-08:00

Great article from Business 2.0 describing four future scenarios for Google:

Imagining the Google Future

RIP IM Smarter

2006-01-30T02:16:55-08:00

My imsmarter proxy stopped working last week, and I just got around to going to their site to see what was up. Looks like they’ve shut down. A bit of a pity; I thought it was a useful service. I use IM on four different machines and it made finding something from a past conversation a lot easier. Actually, though, what I used most often was its reminder feature. I could send it an IM saying “Remind me in 2 hours to check the car” and it would do just that, saving me many parking tickets. I imagine there are other services like that out there; I should check around. Actually, that would be a pretty trival thing to implement myself. Hmmm.

Your Dwindling Privacy

2006-01-09T04:47:50-08:00

Your phone records are for sale
“The Chicago Police Department is warning officers their cell phone records are available to anyone – for a price. Dozens of online services are selling lists of cell phone calls, raising security concerns among law enforcement and privacy experts.”

Data Mining 101: Finding Subversives with Amazon Wishlists
Although this raises some serious privacy concerns, what I found more interesting about this article was how relatively easy it is now to take data from all kinds of publicly available sources and use them to produce something like a map showing the locations of all the readers (or in this case, desire-ers) of a particular book.

(Both of these links via Bruce Schneier’s Blog.)

Business Blogging

2006-01-01T01:53:37-08:00

Chris Anderson of Wired / The Long Tail has started a wiki page to track public blogs by Fortune 500 companies. The list isn’t terribly long yet, but I’m sure it will be growing, both as more people discover existing company blogs and as more companies jump on the bandwagon.

Speaking of which, we’ve jumped on the bandwagon ourselves at Triporama. The Triporama Blog isn’t yet linked in from the main site (it will be soon), but Wendell has already posted a great piece about the origins of Triporama.

Happy New Year!

New Photos

2005-12-17T11:19:08-08:00

Bunch of new photos on up on Flickr, btw.

Triporama Launches

2005-12-15T04:02:29-08:00

I’ve been pretty busy lately, if the infrequency of my posts is any indicator, but it’s paid off: Triporama officially launched yesterday. We sent out some 300 emails and then immediately left to go to the bar. Fortunately, the site held up, with only one serious bug so far, which I fixed last night.

Not that I can slack off now…we’ve got a mile-long list of features we’d like to implement. It’s great to finally get it out there, though.

Interesting Links

2005-12-15T02:40:42-08:00

This:

Top 10 Innovative Web 2.0 Applications of 2005

led me to this:

Good Reading

2005-11-23T01:59:01-08:00

Clay Shirky has a great piece about ontologies, categorization, and how they don’t really make sense for most of the Web today: Ontology is Overrated

Paul Graham’s new essay tries to make some sense of everyone’s favorite buzzword: Web 2.0

Happy Thanksgiving!

Utter Insanity

2005-11-02T04:02:40-08:00

Video of Dan Osman speed-soloing up Bear’s Reach at Lover’s Leap (near South Lake Tahoe, CA). Need I say don’t try this at home? http://www.ebaumsworld.com/video/watch/1220/

(I’ve done this route, but it took me more like 2 hours–not 4 minutes and 25 seconds.)

I’m not sure when this video was shot, but I think it was not long before his death on November 23, 1998. He was always a huge risk-taker, and one of his hobbies was taking very long falls on climbing ropes. He died attempting a 925-foot fall on rigging that he had left up in Yosemite in the rain and snow for over a month. Pretty stupid.

Here’s a good article about Dan: http://www.outsideonline.com/outdoor-adventure/Terminal-Velocity.html

Flock

2005-10-20T16:11:00-07:00

I'm posting this entry from within Flock, a new browser based on Mozilla, the codebase which underlies Firefox. You might think the world needs a new browser like it needs a hole in the head, and I more or less agree with you, but Flock has some really cool features:

You can link your bookmarks to your del.icio.us account so that they're always available when you move between computers.

It integrates with a number of blogging services and software to let you blog directly from the browser. You can even highlight a chunk of text on the page, right-click on it and select "Blog This", and it opens up a blog post window with the text inserted and quoted, and the site name linked. (I'm finding that its blogging editor is incredibly annoying when you're trying to do bulleted lists)

flickr integration, which seems to me to be a bit gratuitous, but it is kind of cool to get access to your photos in a toolbar (sorry, "topbar").

Integrated RSS feed reader, which right now has a really clunky interface. But this is just a developer preview release, so I'm sure that will improve.

"The Shelf"...kind of a clipboard for web content. You can drag-and-drop links, chunks of text, or pictures on to it and later drag-and-drop them into your blog posts (it seems to be made pretty specifically for blogging; you can't double-click on items in your Shelf to open them up, and although you can drop them onto any application that accepts clipboard data, what you get is an HTML-wrapped version of your content).

All in all, very impressive. I hope that some of these features (especially the social bookmarking integration) make their way over to Firefox, but I'll be keeping an eye on Flock's progress.

The Future of Video

2005-10-12T07:16:40-07:00

Apple just released a new video-capable iPod, as was expected. You can also download music videos from the iTunes Music Store, at $1.99 a shot. That's cool. Yawn.

But wait...what's really interesting is that you can also download episodes from some TV shows, like Lost or Desperate Housewives--also for $1.99. This is the start of a major paradigm shift. I'm not a huge TV watcher, but there are a few good shows out there that we've gotten hooked on (Lost, Smallville, Arrested Development, Battlestar Galactica). Not having cable and Tivo, though, I can't always catch my shows. I've always said I'd be happy to pay a small fee ($1.99 is about right) to be able to download an episode legally (it's not hard to find and download them "illegally" online). Now I can do that. Go Apple. The networks certainly weren't going to get their shit together to do it themselves.

It's still got a long way to go. Only a few shows are apparently available. Actually, I don't know exactly what's available because the only links to videos on their "Music" store home page are for "Music Videos" and a big "Lost" graphic. Is it so hard to just have a separate video section in your store? I'm guessing that Apple tried to push this out the door quickly, before properly redesigning their store interface to support the videos. Or maybe not. It's not that hard to do. Regardless, I'm sure they'll sort it out soon enough.

They're going to have to rename the Music Store, though. Media Store? Actually, my guess is that once they pad out their offerings a bit, they'll have a separate Video Store.

An Apple Video Store will be huge for everyone involved. The networks will be able to squeeze new money out of shows that have long since been written off. You'll be able to relive your youth by downloading some Miami Vice episodes or "owning" your own copy (as much as Apple's DRM lets you own) of the classic "Master of my Domain" Seinfeld episode, not to mention getting access to all kinds of old shows that will never make it to DVD. And of course Apple get its cut.

Go Apple.

Shot of the Week

2005-10-11T15:45:05-07:00

Link Roundup

2005-10-11T02:17:20-07:00

In lieu of anything else interesting to post, I'll clear out some of the links I've been saving:

Writeboard
I haven't had the opportunity to really use this yet, but it looks like a pretty cool collaboration tool. No more sending Word docs back and forth. It's also packaged with their equally cool Backpack app.

Aurgasm
I'm a bit late in the game here, but I've just started discovering the wonderful world of MP3 blogs. All sorts of great, eclectic music, for free. There are more MP3 blogs out there than you can shake your booty at, but this is one of my favorites.

Improving Link Display for Print
Absolutely ingenious. Using CSS & Javascript, you can automatically list all links on the print version of your web page as footnotes. I'm going to have to give it a shot on this site.

Fog Creek Copilot
For anyone who is, like I am, the person family and friends turn to when they have computer problems, this looks like a godsend for those trickier issues. Yes, computer remote control software is nothing new, but the really ingenious thing here is how easy it is to get someone set up. You just sign up with the site and give the person you're helping a code. They go to the site, enter the code, download the necessary software, and you're connected.

CSS Layouts: I Give Up

2005-09-27T09:57:00-07:00

We're in the middle of a fairly major site redesign at Triporama, and I was doing my best to have the layout be as purely CSS as possible, replacing the previous table-heavy layout. It's turned out to be a major headache, and I'm taking a step back now and reimplementing parts of the layout with tables.

One problem with CSS today is that every different browser has different levels of support for the different versions of CSS, whereas tables have been around for so long that most browsers in use today handle them fairly similarly. The really big problem with CSS, though, is IE. Internet Explorer is a CSS bug fest. It would be great to say "fuck IE," (actually, I say that almost every day) but it is unfortunately still the most widely used browser.

Yes, I know--tables aren't meant to be used for layout; they're not "semantic", they're slower (not sure about that one--maybe if you have many levels of nested tables). And yes, clever designers have come up with all sorts of tricks to get CSS layouts to work across different browsers. But you end up spending hours or days getting your layout to work on various browsers and what you have in the end is a confusing morass of fragile code. It's just not worth it. I'll wait for IE 7 plus a few years (although my parents will probably still be using AOL 8.0 on a dialup line) before attempting a pure-CSS site again.

So, to summarize my new web design philosophy: use CSS for various individual elements/components on the site, but if you're doing a multi-column layout, use a table for the main layout (actually, I still have my header and footer positioned with CSS, as I spent far too long figuring out how to get the footer to stay glued to the bottom of the page if you have less than a page worth of content, but to flow after the main content if you have more...and the solution I found seems to work on all modern browsers). The layout for the whole site is in a single file, so making site-wide changes (another argument for using CSS) is still easy.

Slowlight

2005-09-19T16:16:11-07:00

Spotlight was a great addition to OS X, and it works great for finding things buried in your filesystem. But for anyone used to Quicksilver or LaunchBar, it is painfully slow as an application launcher, which is really all I need most of the time. I recently reinstalled Quicksilver and I don't think I've touched Spotlight since then.

Link Roundup

2005-09-19T02:08:09-07:00

Some random interesting stories/links I’ve run across in the last week or so:

Photos from this year's Burning Man in the form of a graphic novel. Very nicely done.

With an audio recording of you typing away at your computer for 10 minutes, it's possible to figure out everything you typed. (via Crypto-Gram)

Tongue-eating bug found in fish. So gross, but so cool. (via Boing-Boing)

Word Pads, a very addictive word game. My high score so far is ~~only around 5000~~ 18575.

I didn't really discover this in the last week, but it's too cool not to get a mention: the Gmaps Pedometer (the link centers on a more interesting location than the default of Hoboken, NJ). It's a testament to the brilliance of the Google folks that people have been able to come up with great hacks like this.

Finally, I'm going to try to play in the upcoming Ultimate Frisbee regionals, despite not having played since last winter, mostly due to my crappy back. I think I can hold it together for a weekend. Or at least a day. But this has inspired me: a 67-year old man comes out of retirement to play for his soccer team in their time of need. (via Fark)

LINQ

2005-09-17T02:49:05-07:00

I'm not a C#/.Net programmer, but LINQ is too cool. You can basically use SQL-like querying on just about any collection, whether it be an array in memory, a database, and XML structure. This is one of those "why didn't anyone think of this before?" things. Yes, there have been similar tools for querying objects for a long time, but nothing this elegant. Watch the video for a quick (well, 30 minute) intro, or, if you prefer text, read the overview document.

New Pics

2005-09-01T04:38:22-07:00

Finally got off my butt and posted some more pics to Flickr.

Spam Filtering Idea

2005-08-30T03:13:36-07:00

I share a hosted server with a few friends. We host a number of sites (including this one) and our own email. We have SpamAssassin set up and tuned pretty nicely (thanks, Jelo). Unfortunately, it's killing our server. SpamAssassin can suck up a lot of resources. Ideally we'd have a server dedicated to processing mail, but we can't afford that.

So here's my idea: a spam filtering cluster. Have a bunch of [trusted] home-based servers set up with SpamAssassin and dynamic DNS. A mail comes into the main mailserver and if it isn't on a whitelist, it gets shipped out to one of the filter machines. The filter machine runs SA on it and lets the main server know the result (it doesn't need to sent the mail back, so no worries about low upload speeds). Emails over a certain size would probably just be processed on the main server to avoid the bandwidth and time required to send it out (large emails are rarely spam anyway). Finally, in order for the Bayesian filtering to function correctly, you'd have to periodically sync the data from the Bayesian learner out to the filter machines.

I've looked at SpamAssassin a bit and I think it wouldn't be too hard. SA comes with a client and a server, spamc and spamd. Incoming mail gets piped through spamc, which takes care of shipping it off to spamd for processing. Spamd can live on any other host. It can also just report back whether the email was spam or not, rather than returning the entire email. So the only two things left to do are:

Write a small client (spamb?) that maintains a list of hosts running spamd. When a message comes in, pick the next hostname in the list, and pass it on to spamc. Spamc sends it to the appropriate spamd host, and receives the response--either yes/no or in our case, the full SA report, which gets attached to the message headers. If we get the full message back, that means that spamc timed out trying to contact that spamd host. In this case, mark that host as being down, along with a timestamp, and take it out of the rotation for a certain period of time.
A way to distribute the users' bayesian data files and prefs to the remote systems. Apparently spamd can read user from a SQL database, although I haven't looked into it to see if the bayesian learner data can be stored in a database. If so, that's an easy solution to the problem. Otherwise, you could just write a script that checks for changes in any of the user files and if it sees them, rsyncs them to each of the hosts.

I'll let you know what happens if I get around to trying this out.

Ruby on Rails Problem with OS X

2005-08-28T04:41:22-07:00

I've been playing around with Ruby and Ruby on Rails lately. I'm not that far into it yet, but they both look interesting. I'll write more about my experiences later.

In the meantime, if you're setting this up on a Mac (running Tiger) like I am, do yourself a favor and before you start getting into any of the Rails tutorials (like Rolling With Ruby on Rails), go here first:

Ruby On Rails, Mysql, and OSX Tiger Woes

Do what it says (including the fix it mentions), and you'll save yourself a lot of grief. Note that as one of the commentors mentions, it works just fine with mysql-ruby-2.7 also.

Back to my tutorial...

What do ants smell like?

2005-08-26T11:08:37-07:00

Just got an appointment reminder from Kaiser. Must remember not to put on my Parfum de la Fourmi.

Sage Advice of the Day

2005-08-24T03:02:53-07:00

Some insight to be gleaned here:

10 Steps to a Hugely Successful Web 2.0 Company

I don't completely agree with all of his "steps", particularly the examples he provides. Take #7--Get people hooked on free--for example. Yes, free will always win you fans, and a large user base has intrinsic value, but if you're providing a service that people are willing to pay for, by all means let them pay. (Regarding his Thefacebook vs. Match.com example--which one is actually making money?) Still, there are some good ones. My favorite: 6. Be mindnumbingly simple. Then again, this is a pretty age-old design mantra.

Useful Sites of the Week

2005-08-15T03:56:11-07:00

I've had my web designer hat on lately. I've always been a much better judge of good design than a creator of good design, so I need all the help I can get. One thing that can make a huge difference in the quality of your site is color selection. Here are some sites I've been going to lately for chromatic inspiration.

The Return of Design - Web Color Schemes
A slew of pre-packaged palettes, all using web-safe colors (i.e. 256-color palette), each with a number of variations.

ColorBlender
Pick your starting color and ColorBlender will show you a 6-color palette for that color, and allow you to easily play with variations or customize them. You can also save them [to a cookie] so you have your personal blends available to you whenever you visit the site. You can also email direct links to your blend. Very cool.

Color Scheme Generator 2
The ultimate color-generating tool. If you can't come up with a color scheme you like with this tool, you really are a lost cause. You can even see how your scheme looks to people with any one of eight different types of colorblindness.

CSS Not Working? Don't Forget Your Doctype

2005-08-08T11:54:22-07:00

I just spend *hours* trying to get some CSS working only to discover that the reason it wasn't working was because I didn't have the DOCTYPE declared at the top of my html:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
         "DTD/xhtml1-strict.dtd">

The Butler in the Pantry with the Candlestick

2005-08-04T08:32:26-07:00

Today's diversion: Whose Fish?

Took me about 45-50 minutes, the first 30 minutes of which were spent on a false start. After starting over with a better system for keeping track of everything, it wasn't that bad.

I really doubt that this was Einstein's puzzle, and certainly the 2% figure is pulled out of someone's ass, but it's still good for some brain calisthenics.

Secret to cheaper flights?

2005-08-04T06:33:26-07:00

Last night Jacqueline was looking around online for flights to Seattle. She ended up falling asleep before she could book anything, but when she went to check again in the morning, she found that the same flights were about $30 cheaper.

Perhaps it was just a coincidence, but my theory is that people/agents put holds on flights throughout the day, and those holds are released at the end of the day (assumedly midnight, although I don't know which timezone), freeing up seats and thus reducing prices. Has anyone else seen this? Or is this common knowledge and I've just been clueless?

Netflix

2005-08-02T17:05:18-07:00

I finally caved in and signed up for Netflix, as you'll see if you look on the bottom of the sidebar. I've been a bit of Luddite, preferring to get my movies from one of the great local video stores. But we recently moved and now our closest video store is about a mile away--as opposed to about 200 feet away--so our movie-watching has dropped dramatically (while our watching of mind-numbing reality shows has increased accordingly).

It seems to me that the queue is the key feature of Netflix. I can't count the number of times we've seen a preview for a movie and thought "We should rent that." Then you get to the video store and you spend half an hour walking up and down the Recent Releases trying to figure out what to get. Then again, sometimes you're in the mood for a particular kind of movie, and it might not be what's next in your queue.

Hmmm...here's a business idea to help out all those struggling video stores in the new Netflix world: make an online service (let's call it FooFlix) where people can browse movies, get recommendations, and add movies to a queue, much like Netflix. But then install terminals in video stores and tie member's FooFlix accounts to their video store account. Then they scan/swipe/enter their store card/code in the terminal and it brings up the movies in their queue and highlights the ones that the video store has available. The stores would also be able to see what their customers (or an aggregate of all FooFlix customers) have in their queues, so they know what they should be ordering.

As for a revenue model, you probably couldn't get away with charging customers for the service, but you could charge video stores to rent the terminal, and perhaps for customer queue data. You could probably do some pretty targeted advertising on the site, too.

Probably not going to make me a billionare. If anyone steals the idea, at least send me a postcard or something.

Thank God I wasn't OS/2 Warp

2005-07-29T17:03:16-07:00

Which OS are You?

ISO Good-enough Idea

2005-07-27T17:13:43-07:00

Joel Spolsky's latest article is largely a rehash of one of his earliest articles, whereby he expounds on his belief (which I share) that there's no substitute for top-notch programmers. The thing that got me thinking was his "shaky claim that most people are wrong in thinking you need an idea to make a successful software company," and this chart:

Best Working Conditions

Best Programmers

Best Software

Profit!

The problem I have with this is that you still need a pretty good idea to start with, or you're never going to get the best programmers in the first place. Great programmers aren't in it for the money; they want to work on stuff that interests them. Cool stuff. Sexy stuff. Having a great working environment with lots of other great programmers around is a big draw too, but how are you going to achieve that without the great idea in the first place?

This hits a chord with me because I know a lot of great programmers with whom I would love to start a company, but I've been waiting for the perfect idea to come along and it hasn't. But perhaps the trick is to just come up with a good-enough idea -- good enough to get one or two other great programmers to dive into it with you -- and just take it from there.

So...who's got a good-enough idea?

There's Sex In My Violence!

2005-07-22T03:22:22-07:00

Another great column by Mark Morford. Complaining about a bit of hidden sex in a game that glorifies sex, drugs, and violence does seem a bit misguided. Yeah, Puritan roots!

Falling

2005-07-13T16:16:52-07:00

Cool and a bit disturbing, and strangely addictive. Use your mouse to move her around. Pretty nice bit of programming.

Embrace Your Geekness

2005-07-13T02:29:38-07:00

Somebody has deemed today Embrace Your Geekness Day. So what better way to celebrate than post to my blog.

I searched around for information about the origins of this holiday, but found nothing other than lists of other obscure days (July 15th is Respect Canada Day!). How does one get a day proclaimed anyway? There are some openings left; one calendar I looked at had the 29th and 31st of this month still open. How about a Wear Flip-Flops to Work Day? (Googling for that came up with a single hit.)

Belated Happy Birthday

2005-07-11T07:33:11-07:00

The July 4th New Yorker cover was classic:

Scrumptions Muffins

2005-07-09T19:39:57-07:00

From a cookbook by the Ladies Auxillary of some hospital in Rhode Island:

Scrumptions Muffins

1 cup softened vanilla ice cream 1 cup sifted self-rising flour

Mix well. Pour into a muffin tin lined with paper cups, about 3/4 full. Bake at 350 F for 20 min.

You know you want to make them. I know I will.

Update: Made them. I had to improvise a bit, as I didn’t have self-rising flour. I added about 1/2 tsp. of baking powder instead. They were pretty bad. Very dense. Must have been the flour.

Always check your logs

2005-07-08T13:30:50-07:00

Brad’s #1 rule of debugging web apps: always check your log files. All of them.

A friend of mine just ran into a problem where a php file upload script that had been working fine for months suddenly started barfing on large files. He was getting a “partial upload” error. I told him to check his web server error logs and his system message log. He did the former, and after hours of pounding his head against the wall, I finally went in and poked around myself. I actually found the problem before looking in the log files, but there it was in /var/log/messages:

Jul 7 15:27:42 xxxx /kernel: pid 46891 (httpd), uid 80 on /var: file system full
Jul 7 15:30:02 xxxx /kernel: pid 46875 (httpd), uid 80 on /var: file system full
Jul 7 15:49:07 xxxx /kernel: pid 46872 (httpd), uid 80 on /var: file system full

The temp directory that php was using for file uploads was /var/tmp, and /var only had 216K left. This is why small file uploads were working, but larger ones weren’t. (As an aside, be careful of what you’re storing in /var. On some systems, log files are kept in /var/log, and /var is often set up as a separate partition, and often relatively small at that. In his case, the whole partition was only 252M, and /var/log was eating up 195M of that…and growing. We moved /var/log to /usr/var/log and created a symlink.)

Original	Chomsky normal form
`107: 18 \| 47`	`107: "b" \| "a"`
`11: 42 31 \| 42 11 31`	`11: 42 31 \| 42 133` `133: 11 31`
`8: 42 \| 42 8`	`8: 47 50 \| 18 4 \| 42 8`