Gregable

Backups with a ReadyNAS Ultra 4 Plus and CrashPlan

2013-03-17T11:58:00.000-07:00

After a long vacation full of photography, you load all of your photos up in Picasa. You spend hours tweaking colors, cropping, tagging faces, and plopping down geo-tags onto a big map. The very next day and your computer won't start. The drive is dead.

Despite the fact that the value of those photos and all of your other digital memories are priceless to you, you have never spent the effort to set up a reasonable backup. It was something you planned to do, but never got around to. Three years of photos are now locked away in a lifeless hard drive.

You consult your friends. You get 5 different recommendations for 5 different pieces of software that will attempt to recover the disk. The platters won't spin though, so no software will help. Next you try to find an exact duplicate of that disk and swap the platters. This isn't remotely easy and has a very low chance of success. Maybe you care about your photos enough that you even send the drive off to a professional drive recovery service. Perhaps that works and 30% of your files are recovered, but the cost ends up sky high.

Most of my geek friends at least claim to have a backup system in place for the files that they care the most about. These systems usually have one or more of the following flaws:

Technically complex, ie: cron jobs, command lines, shell scripts. (guilty)
Cost fairly large sums of money (the cheapest online backups I see usually start at $5/mo)
Require regular human action (swap out thumb drives, burn a CD).

These flaws generally aren't fatal for geeks, but they are for non-geeks (ie: family). So, how do geeks approach the family tech support backup problem? I'll share my solution with you, though I'm sure there are alternatives.

For software, we are going to install CrashPlan. CrashPlan is a client/server backup system with a number of really handy features that we want:

Unless you want to back up to CrashPlan's servers, it's free to use.
It's a Java app that runs on multiple systems (windows, mac, linux).
It has a decent graphical UI that is non-technical.
You can offer to be a backup destination for a friend. The process is very simple for both of you, Crashplan gives you a 6 character code. If the friend enters that code in their client, they can backup to you. Firewalls, dynamic ip addresses, etc are all negotiated for you keeping things simple. Backups are encrypted before being sent, so there is no privacy risk.

This last feature is what I use for my family backups. However, on my side things get a little more geeky/technical.

I don't like leaving my machines on when not in use, due to power consumption. However, by default this would make backups challenging as there will only be transfer when both me and my family member's machines are up.

Instead, I use a Network Attached Storage device (NAS) to store my backups. It's low-power-ish and always on, which saves me money over leaving a energy hungry computer on all the time. For CrashPlan, you'll need a NAS with an x86 processor and which allows you to run software on it. I use the ReadyNAS Ultra 4 Plus.

On my NAS, I install the Community Plugin that enables Root SSH Access and reboot. Now, I have root access to my NAS with the admin password used to setup the NAS. Simply ssh root@nas

Now I need to add to /etc/apt/sources.list a new source: deb http://archive.debian.org/debian-backports etch-backports main non-free by adding that source line to the end of the file.

Next update our package list: apt-get update

Next we need to install Java. We first reconfigure our dialog so we can accept the terms and conditions: dpkg-reconfigure debconf
Select [1] for dialog and [3] for medium

Install Java: apt-get install sun-java6-jre
Select 'yes' for everything

You can reconfigure again (optional): dpkg-reconfigure debconf

Select [6] for noninteractive and [3] for medium.

Finally, we can install Crashplan on the ReadyNAS.

wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.2.1_Linux.tgz

tar -xvf CrashPlan_3.2.1_Linux.tgztar -xvf CrashPlan_3.2.1_Linux.tgz

cd CrashPlan-install

./install.sh

Defaults work for most questions except backup location. I used /backup/crashplan.

Once installed, you can log out of your SSH connection. Crashplan is running as a server.
Of course, you still need to do some configuration which can only be done from the Crashplan client UI. From your computer, install Crashplan and follow these instructions for connecting to your server's headless client:
http://support.crashplan.com/doku.php/how_to/configure_a_headless_client

Once you have that set up, you'll be able to generate a Crashplan backup code, something like FJSW3X. Send this to your family, ask them to install Crashplan and use your backup code. The first backup may take awhile, but after that Crashplan should keep up to date incrementally with no intervention or hassle from your family.

Software Development Books

2013-01-13T19:56:00.002-08:00

I'm often looking to improve the software that I write. If you are in the same boat, here's a few books that I felt have helped me. This is not exhaustive, but some of the ones I could think of off the top of my head that I'd recommend. Please share others that have been good for you too, I'm always looking for more.

Note, there are affiliate codes in these links, though feel free to not use them, I don't really care. If you are in the Bay Area, I would plug my favorite bookstore which frequently has some of this kind of stuff in stock (BookBuyers)

Algorithms on Strings, Trees, and Sequences: Very likely the best book on string algorithms (and trees/sequences). It's references computational biology, but you need not know a tree from a frog to get a ton of value out of this book. Invariably, one of my Google coworkers is always borrowing this book. If you are interested in more about the wonderful world of strings, this book will get you pretty far.
Refactoring: This was a very useful read when I read it a few years ago. It came at the right time in my programming development. This almost has less to do with the mechanics how to refactor and more with how to structure code in the first place. The examples are easy enough, but seeing them and the reasons why they reduce complexity helped a ton.
Design Patterns: I don't get as much value out of this as most people. I don't find myself implementing the "X pattern" so much as perusing patterns has occasionally tipped off a light bulb in my head on how to structure things. I feel like there is more I can learn from this still and intend to revisit.
Coders at Work: A collection of interviews with some of the big software developers in the field. Full of lots of nuts and bolts insights and opinions on software development. This isn't so much about software engineering, but about everything that goes on around it. Unlike the above three books whose hardbacks are high-quality productions with diagrams, this one is a cheap paperback book with only text - there is no reason not to just grab the kindle version. Note I also read Founders at Work, but found it to concentrate more on things like fundraising / making deals - Coders was more relevant.
Javascript the good parts(O Reilly): More than a few people have mentioned that they never could wrap their brain around Javascript until Crockford's book. I found myself in the same position. I've forgotten too much from this book as I don't use Javascript frequently enough, but this is a great place to start if you want to understand it. There is also an @Google Tech Talk from Crockford on the same subject that might give you a flavor.
Wireless Nation: The Frenzied Launch of the Cellular Revolution: A little off-topic, but this is a fascinating book that takes a look into how the cellular industry got started in the US. It helps you to understand clearly how we got to where we are now, such as why the standards are so fragmented. It's also a delightfully fun read.

2012, Looking back

2013-01-01T23:23:00.003-08:00

2012 was an abysmal year for the Gregable blog. Only 4 posts! They were decent, but not great. Google Plus has taken some of my steam for short form postings, but really the blame lies on my shoulders.

Anyway, 2013 should be better. With this post, I'll already be caught up to March of 2012's volume. Dear Gregable readers, what would you like to know more about? Help me break out of my writer's block.

And... Happy New Year to you and yours!

rel=canonical as a browser feature

2012-08-22T20:08:00.000-07:00

I informally propose that rel=canonical become a tag that not only search engines respect, but also browsers.

For a little while now HTML5 browsers seem to have a feature where javascript can modify the displayed URL of the page using window.history.pushState. The changes are of course subject to same origin policy rules (ie: the protocol, hostname, and port cannot be modified, only the path and parameters).

Originally javascript folks hacked this in with older browsers by shoving text after the "#" symbol in the URL. Even though there are a number of problems with this, it was useful enough that it became somewhat widely used. With modern browsers this is no longer required.

To see what I mean, click on this little demo: http://kurtly.tumblr.com/sticky-history and look at the URL bar. If you aren't running an outdated browser, you should see the URL changing every few hundred msec. The page is not being re-fetched from the server.

The rel=canonical link tag has been telling search engines basically "I know you are fetching the URL http://gregable.com/foo, but I'd suggest you should pretend this URL is http://gregable.com/bar in your search index". Basically the same idea as the window.history.pushState functionality, only for search engines.

I propose that a rel=canonical link tag on any HTML page which satisfies the same origin policy should visibly change the URL in the browser. All the same motivations exist for this as they do in the browser. If a user copy/pastes the displayed URL, they'll get a more satisfying experience. If the user mis-types an URL (ie: .html vs .htm), sending them to the correct one generally requires a 301 redirect which adds latency. The javascript solution is less reliable as users sometimes surf with javascript off, and the javascript may not execute until the page has finished loading either.

Are there any obvious reasons I'm missing why this is a horrible idea?

Are there any regular Gregable readers who work on browser standards and might want to propose this more formally?

LED Bulbs

2012-05-08T23:00:00.001-07:00

I've just been trying out some LED light bulbs and they seem to have progressed a great deal since the last time I played with them. For recessed fixtures that have a narrow angle of lighting, they seem to be a pretty good deal.

Previous generations of LED light bulbs had problems:

Blueish color of light
Delay after turning on the wall switch
Wouldn't work with dimmer controls
Not as many lumens (brightness) as desired.

I've bought a couple different bulbs off of Amazon and tried them out. I ended up really liking these ecoBrites: http://www.amazon.com/gp/product/B003THZHOU. No affiliation / kickbacks for me at all, I'm sure there are other great options out there too.

They seem to solve all of the above problems, though they do look a little bit different than regular bulbs if you look at the bulb when it's turned off.

The key is to look for bulbs of a certain "color temperature". The blue or "cool" colors are a higher temperature (around 4000-5000k) whereas the yellowish incandescents tend to be a warmer color around 2500-3000k. CFLs are usually a higher temperature too though not usually blue, so they look very white.

My PG&E rates are tiered - 12.8c/kWh for the baseline, then it goes up to 14.6c/kWh for the next chunk and I'm actually bumping a small amount in to 30c/kWh rate lately. So, my incremental cost of shaving off power usage is 30c/kWh initially and if I can get it down enough, probably 14.6c/kWh.

The above bulbs are 7W and replace 60W incandescents. So, I'm saving 53W/h while these run. They cost $39/bulb though. Very conservatively, let's go with the 14.6c/kWh rate. That's .7c/hr savings. Assume I run each bulb for only 2 hrs per day. To save $39, it'll take 6.9 yrs to breakeven. That's the conservative number.

If you assume only that:

I'm replacing a bulb, so would have to pay $7 anyway, the breakeven is 5.6 years
I need to buy a new incandescent bulb every ~750 hrs, the breakeven is 4.4 years
the extra 53W of heat an incandescent bulb generates needs to be matched by at least 53W of air conditioning work (likely far more due to inefficiency), the breakeven is 3.4 years
I'm actually reducing my bill by the 30c/kWh rate, the breakeven is 3.4 years
If I'm using the bulb for 3 hrs / day, the breakeven is 4.6 years

If you assume all of the above, my breakeven becomes only 11 months.

In practice, the real story is probably somewhere in the middle. I do need to buy incandescent replacements periodically, I sometimes need to use air conditioning, but certainly not always, and my savings is probably a mix between the 30c and 14.6c rates once all is said and done. So maybe the breakeven is 2-3 years, so roughly a 26% return. That still seems like a very good investment these days.

Cloud Storage Price Comparison

2012-04-24T20:47:00.001-07:00

Google Drive launched today right on the heels of Microsoft's Skydrive earlier in the week. It seems the cloud storage revolution is heating up, with other big competitors including Amazon's S3 and Dropbox.

Of course, each of these services has different sets of features. Amazon S3 is more of a bare-bones backend for developers to build on top. Dropbox has linux support, yay. Google Drive has some amazing appstore integration and rich Google Docs interface. Microsoft's Skydrive presumably has the deepest integration with Office software (although Drive does have an Office plugin called Cloud Connect).

While it's sometimes hard to compare long lists of features, it's pretty easy to compare a number. Today, Gregable readers, that number is GB and monthly cost. Here's what the 4 offerings above look like when stacked up against each other (please correct my math if I missed something):

Microsoft Skydrive and Dropbox don't publish rates beyond their 100GB additions and may not even offer them. Google Drive keeps the same ratio past 1TB, and amazon actually gets a tiny bit cheaper. The interesting stuff to most people is in the left part of that graph, so let me blow that part up for you:

That's a simple view, but I think it's roughly accurate. Some other caveats just to the pricing to keep in mind:

Amazon S3 charges separately for upload and download bandwidth where the other solutions include bandwidth in the price, so Amazon S3 really costs more than what's shown above in practice.
Amazon S3 gives you 5GB free only for the first year. The other services' freebie quotas are permanent. Dropbox gives you 2GB free, Google gives you 5GB free and Skydrive gives you a nice round 7GB free.
Microsoft Skydrive charges you annually rather than monthly, so I converted all of their prices to monthly prices. Of course, this means less flexibility as well as the fact that you have to pay everything upfront.
Sharing in dropbox counts against all of those users' quota, while sharing in Google Drive counts only against the person who shared. It isn't clear about Microsoft Skydrive's policy on this.

Also, I imagine that it'll be very interesting to take a look back at this post in a couple years and see how this has all changed.

Mega Millions

2012-03-24T16:50:00.000-07:00

I for one think the lottery is a tax on people who aren't very good at math. If I wanted to gamble, I'd buy a casino.

Mega Millions is the big multi-state lottery that has in recent days grown it's jackpot to the point where the payout could be worth the risk. The cash payout is currently at $255 million (the $356M number is an annuity) and growing. To win, you buy a $1 ticket where you must pick 5 random numbers correctly out of a pool of 56 and 1 random number correctly out of a pool of 46. The odds of a correct pick are 1 in ₅₆C₅ x 46 or 1 in 175,711,536.

Is this a "good bet"?

For the sake of simplicity, let's ignore taxes as well as the possibility that there is more than one winner (thus splitting the payout). Also, most people's utility for money is non-linear (ie: beyond a certain point, more money doesn't matter as much any more). Those are important issues to consider in real life as they have a fairly large effect.

To estimate the expected net returns, P(win) x jackpot - cost =

= $0.451.

45% return on your dollar in just a few days sounds like a great investment. Mortgage your house, max out your credit lines, sell your stocks, and invest in tickets!

The problem with bets is that even if you have the odds in your favor, you can still lose everything. How do you choose what to bet then? Bet too little on a good gamble and you'll leave money on the table. Bet too much and the losses will keep wiping too much of your winnings out. It turns out that there is formula that predicts the optimal size in a series of bets which will maximize your winnings in the long run. This is called the Kelly criterion. It determines a bet size based on your odds and your current bankroll available to wager. As your bankroll grows, you bet more.

Let's say that you were only allowed to buy one mega-millions ticket per round with the 1 to 175,711,536 odds of winning $255M. Is this one ticket a sound investment? The Kelly criterion tells you the fraction of your bankroll you should invest in this bet. The formula is simple enough, divide the expected net returns ($0.451) by the net winnings if you win ($255M). The result is 1.77x10-9 or 1 in $565,410,199.

If and only if you have at least $565,410,199 to invest, buying a $1 ticket is a mathematically sound investment.

Buying two tickets in the same lottery round is a slightly different gamble than buying 1 ticket each round. It's a little better odds. Taken to the extreme, buying 175,711,536 tickets guarantees a win whereas buying 1 ticket in each of 175,711,536 rounds does not. A modified Kelley criterion can evaluate the case where you buy multiple tickets too, and it's going to be more favorable. Unfortunately, I don't have the time at the moment to add that to the post. If there is interest, perhaps I'll return and see if I can work through that math.

Etsy

2011-10-28T19:12:00.000-07:00

Cristin, my better half, has been making jewelry for several years now and selling it in a store in her hometown in Virginia. She just set up an etsy store and listed some of her first jewelry on it for sale. Just a few pieces for now, but I think you should check it out:

strangeterrain.etsy.com

Yosemite High Sierras

2011-09-06T09:51:00.000-07:00

I just returned from a 6-day, 50 mile hike between Yosemite's High Sierra camps.

Most people who visit Yosemite visit the iconic Yosemite valley, which is truly beautiful, but teeming with people. A smaller fraction turn north as they are entering the Park and head up to the higher altitude area of Tuolomne Meadows and Tioga Pass. Equally beautiful, although less iconic, this area of Yosemite has more limited amenities and is far less busy even in the summer months.

Fewer people still are even aware of Yosemite's High Sierra Camps. These are 5 staffed camps (beds, cooked meals, running water, showers) that cannot be reached by road. You can only reach these camps by trails, either on foot, or by mule. Few people I've talked to have ever heard of these camps. Still, availability is extremely limited so reservations are made a year in advance by lottery. The season for some of the camps this year was less than 1 month, and they could only handle about 40 people per night.

The camps are arranged in a loop, and all of the camps can be reached via a long day hike from the road. The most remote camp, at Merced Lake, is a 12 mile hike from the valley. We decided to visit all of the camps along the loop in clockwise order (apparently counterclockwise is more common). The route is below.

View High Sierras in a larger map

The days ranged from 6 - 10 miles, some of the days up days, some down. The most elevation change was ~3000 ft. The experience was different every day: granite, forests, meadows, waterfalls, streams, lakes, vistas, sunsets, stars, and wildlife. I've detailed more of the trip below for those interested, but feel free to skip.

Day 1: Tuolomne Meadows to Vogelsang

The first day's issue is dealing with the altitude as we aren't yet acclimated. We climb a modest 1,400 ft over 7 miles from Tuolomne Meadows to Vogelsang camp. The climb is relatively steady uphill the whole way with forest cover some of the way and the most amazing meadows especially as you near the end of the trip. Wildflowers, pikas running around, streams with fish, and granite walls on either side of the valley you are walking through. A quick climb at the end over a lip and you are at the base of Fletcher peak at Vogelsang camp, the highest altitude camp of the 5.

Vogelsang camp sits a little above 10,000 ft on a ledge above a granite valley below with Fletcher peak above and Vogelsang Peak in the distance. A few hundred feet away, a few of us took a brief dip in Fletcher lake, which is just above freezing even in August. You can watch the snow melt immediately above the lake while you are swimming. As it turns out Fletcher is a vast aquifer. We are told that Vogelsang has the only legal non-chlorinated drinking water in California. I didn't notice the difference.

After dinner, the stars at Vogelsang are fantastic. Our trip chanced to schedule Vogelsang on a new moon, so it was very dark. The milky way was quite bright and visible.

Day 2: Vogelsang to Merced Lake

We opted for the slightly shorter / easier route of the two options. This turned out to be one of the prettiest stretches on the trip so we were happy with the decision, although who knows what the other option had in store. This day was the biggest elevation change of the hike, dropping ~3,000 ft over 7.6 miles, but it wasn't a smooth gradual drop. Instead it was long switchbacks alongside waterfalls interspersed with more beautiful meadows. Very difficult on the knees. My favorite stretch along this hike was what seemed like a half mile long water slide that just kept going forever. The image below shows a tiny section of it, but can't really do it justice:

Merced Lake camp used to be a military post. The tents are arranged in a half circle around a central campfire which probably was once a flagpole. This is the most comfortable of all of the camps, with 8 showers, a warmer swimming hole in the stream, washbasins, and lots of shade. Not much for a view though as you are at the base of a valley and deep in trees. It's also the most remote - the nearest road is in the valley, 12 miles away. As with the other camps, all supplies are brought in by mules.

Day 3: Merced Lake to Sunrise

Day 3 started with a few hundred feet of drop, then climbing back up 2,300 ft. All over 10 miles. Here again the elevation is not evenly distributed, with several switchback sections. Only this time they are up. It was a long day, easily the hardest of the 6, and probably the least scenic of the trail stretches, although that's relative - it still packed some amazing views.

Sunrise camp on the other hand was likely my favorite, and I spoke with several people who would agree. The camp is perched on a ledge above an alpine meadow at ~9,400 ft. The meadow then drops off and you can see incredible views of several mountain ranges in the distance. The view at sunset was amazing. Sunrise in the morning was even more so as the meadow had frosted over during the night and would sparkle in the sun. The camp also had showers, and the staff formed a one-song band at dinner, which was a fun experience.

The photo below is from the meadow below sunrise camp. Sunrise Camp itself is a little higher, affording good views of the meadow as well as the mountain ranges behind the trees in this photo.

Day 4: Sunrise to May Lake

Day 4 was an 8 mile hike that was fairly easy except for a painful ~2 mile stretch of rapid descent along switchbacks above Tenaya Lake. You begin by climbing over the ridge behind sunrise camp and dropping down gradually past several photoworthy "sunrise lakes" that might have been great for swimming had we had the time. Immediately before the painful descent is a trail marker for Clouds Rest. At the recommendation of someone at Sunrise, we took a side trip out a few hundred feet along this trail to be rewarded with a fantastic view down into Yosemite Valley, including the back side of Half Dome.

The switchbacks were panoramic as well, although overlooking the Tuolomne area and Tenaya lake rather than the valley. At about the 6 mile mark, we reached our strategically placed car at Tenaya Lake, the only road crossing for the hike. We switched out for some fresh gear and finished the rest of the hike uphill to May Lake.

May Lake is nestled halfway up Mt Hoffman, which a staff member from Sunrise mentioned is the geographic center of Yosemite. The camp lies between the lake and a short granite lip. Scrambling up the lip affords a panoramic view of the ridge you just climbed down as well as many other peaks, as seen in the photo below. May lake is pretty comfortable too. The manager, Brian, treated us to some colorful historical stories of the camp as we enjoyed the delicious salmon dinner.

Day 5: May Lake to Glen Aulin

Day 5 was pretty mellow - a gradual downhill, mostly flat, over 8 miles. We got into camp fairly early as a result. Most of the day is within forested areas, so there is little to see. The only challenge was a few patches of mosquitos and flies. We donned some mosquito nets and moved on - I've seen much worse. The only real vista was early in the morning, but it was fantastic - you could almost see all the way to the next camp, below is a fraction of that view. Glen Aulin is just behind that small mountain almost in the center of the photo, right behind the first forested saddle.

Glen Aulin itself is in an amazing location. It sits directly beside a large waterfall on the Tuolomne River. You can barely see some green benches and one or two of the white tents on the opposite side of the river in this photo.

Downstream a short bit, we took a swim in the river, below yet another waterfall. Still cold, but much warmer than Vogelsang. Glen Aulin has no showers, so this was the only option for washing off.

Day 6: Glen Aulin back to Tuolomne Meadows

The final day is also pretty easy, you follow the Tuolomne river upstream for a little under 6 miles back to parking lots at Tuolomne Meadows. For the first few miles, it's literally one waterfall after another and progress is slow due to enjoying the views and taking photos. After a while, the river slows down, flattens out, and widens into a Tuolomne Meadows. A different type of amazing view, more relaxed. Overall, this hike is just a wonderful finale.

I'll eventually upload some more photos and I'll post a link on Google+ if you want to follow along there.

I also thought I'd share a few boring notes on logistics, as I had a little bit of a hard time figuring out much of this online. Only read the remainder if you are planning on making this trip yourself and have questions about the camps and what you need to bring:

The camps are tent cabins. 4 (usually) spring cots with matresses. You'll share with strangers depending on your party size. Each bed has 2 army blankets and a heavy comforter. This was plenty to keep me warm at night and it got below freezing at least one night. The only other thing you'll need to bring is a sleep sack or sheets, no bed linens are provided. The beds also have a pillow and a pillowcase that gets washed. Don't bring pads/sleeping bags/pillows.
Except for Merced Lake which is lower elevation and warmer, every tent has a wood burning stove and you'll be provided with wood, starters, matches, candles. Everything you'd need to start a fire.
As for temperature, it seemed plenty warm as long as the sun was out. After sunset but before you climb into bed, you'll be chilly and the morning will be chilly. Still, all I needed was a light jacket. Unless you plan on staying up late and watching stars, my experience was that lots of heavy clothing was unnecessary. The Yosemite packing list suggested a down jacket, fleece, mittens and thermal underwear. It was nowhere near that cold, but check your weather forecast I guess.
Speaking of weather forecast, use NOAA and click on the map where your hike is - the nearest station which is what weather.com and others use is too far away to be useful.
Merced Lake, Sunrise, and May Lake have showers. You will want to bring a towel and soap/shampoo, although you may be able to buy a towel from the camp store if you forget. A small washcloth is provided at every camp, but that's it. Other than sunrise, all of the camps have great nearby swimming options which might cover for a shower depending on your preferences. Best to bring some kind of shoes you can wear into the water though as the rocks can be a little annoying. Flip flops worked for me. You might need these for water crossings on the trail anyway.
Either carry lots of water (4 liters) for the day or carry a filter. We toted a filter and were rarely far from water, so we didn't need to carry as much water weight.
On food, breakfasts and dinners are plenty of food and wonderful. Best backpacking food ever. Bag lunches include a decent sandwich and optionally fruit, trail mix, cookies, and a fruit drink. You can order it all or a la carte. You order at the camps the night before, not much advanced planning required. Credit Cards are accepted if you don't want to tote cash. Basically, you don't need to bring any food, or just some snacks if you would like.

Canyon Creek Backpacking, Trinity Alps

2011-08-15T13:59:00.000-07:00

Last weekend I got the chance to go hiking in Trinity Alps Wilderness in Northern CA. It's an area I've been wanting to visit for some time, but logistics are challenging - it's a ~6hr drive from the Bay Area and there is only a narrow season when hiking wouldn't require snow traversal.

The Trinity Alps area is very large, we hiked only one valley. The trail starts down the valley shown in the upper left side of the above photo, climbs slowly along Canyon Creek, and then there is a short scramble up to these lakes: Lower and Upper Canyon Creek Lakes at around the 8 mile mark. The trail then cuts between the two lakes and after a short stream fording wraps around to the northern side of the upper creek (right side of photo). Most folks stop and camp around here.

We cut east (camera in the above photo is facing west) climbing at a 40-60% grade for the another mile up to L Lake (some maps show "el" or "ell"). We camped on the granite domes above and to the south. That last mile was quite tiring, but rewarding. We were sitting in a bowl with a great view of the surrounding area and night sky. We enjoyed the Perseid meteor shower until around 11pm when the full moon rose above the nearby ridges and the whole valley was illuminated.

The map below shows the rough trace of the trail, although I just drew this by hand from memory rather than GPS, so it's not entirely accurate.

View Trinity Alps, Canyon Creek in a larger map

Can we build communities with Google+ ?

2011-08-04T22:40:00.000-07:00

Apologies, this might be a bit long. I want to discuss an idea for Google+.

Progression of a Community
Imagine a mailing list about a somewhat broad topic, rock climbing. This could apply to other social software systems (bbs, forum, social news), but I’ll use the mailing list for my example.

Initially the list grows out of a small group of contributors, likely folks who know each other IRL. For these seed members, the list is just a convenient way to communicate out-of-bandf, organize trips, share local news, etc. It’s a shorthand version of a cc list in email.

Over time, people invite their friends and the list starts to grow. Geographical differences may begin separate the members. If you could draw the social graph of the members, it would have sub-clumps. At this stage, not everyone knows each other any longer.

As the list continues to grow, at some point the signal to noise starts to creep up on the list. Reading a discussion on a local rock face in another time zone is minimally relevant. More annoying is that user on the list trying to sell a new belay device to this targeted rock climbing audience. At this stage, a community can collapse under it's own weight.

Scaling Patterns
Strong communities usually attempt to solve the above problems. The attempts often fall into one or more of several patterns:

Individuals who are least similar to the “list personality” unsubscribe or are forced out.
A subgroup of “community administrators” creates strict rules defining acceptable content or acceptable users, removing content or approving it. This administrator group can eventually have scaling problems too. Wikipedia is a classic example of this approach.
Formation of fractured smaller lists to discuss more specific topics. Reddit is a classic example with its numerous subreddits.
Using algorithms / voting to promote the more interesting discussions to help users filter content from noise. Digg or the reddit home page are classic examples here.
I don't think that these patterns are full solutions. Some seem to allow for a larger scale without fully solving the scaling problem. Some succeed at promoting content that is most interesting to the lowest common denominator, but fail at promoting what’s most interesting to each individual.

Communities in Google+
I wonder if + Circles with some tweaks could work better as a community model. I’m interested in your ideas, but here is my strawman:

Anyone can form a “public” rock climbing circle, but it doesn’t contain people, it contains comments, so let’s call it a rock climbing square rather than a circle.
I can post to that square. Those posts don’t show up in the streams of people who follow me.
All posts to the square are visible, but it’s liable to get noisy and unlikely anyone will really pay attention to the full stream for long.
I can follow other people strictly within the context of the rock climbing square. I won’t see their public posts, just their rock climbing posts, but it’s public in a sense as they aren’t required to follow me in return.
I can reshare strictly within the context of the rock climbing square. If Bob is following Alice, Alice is following Charlie, but Bob is not following Charlie - then Alice resharing a post from Charlie will display it to Bob, thus giving the square friend of a friend semantics with a filtering component.

In this sense, the community becomes a very weakly defined notion. It’s not a fully connected graph like facebook, forums, or mailing lists. It’s not disconnected subtopics either, like subreddits. Members become filters for other members, and I can pick and choose my filters as their interests match my interests.

It has many similarities to twitter, but one important difference that I see (other than the character limit). I can follow someone’s rock climbing posts without following their underwater basketweaving posts.

What do you think? How could this be improved?

PS: I work for Google Search, but have no visibility into what the Google+ team’s plans are. As a result I assume it’s safe to write about this kind of thing, but do remember - these are just my personal thoughts.

My Circles

2011-07-08T14:23:00.000-07:00

Google+ seems well received so far. Most of the people I've talked to really enjoy the "Circles" feature and feel that it fits their mental model well. I thought it would be fun to do a little data dive on my facebook graph to see how my relationships were actually interconnected and whether the data matched my perceptions of how I was planning my circles. First the graph, then the explanation:

Gregable's Facebook Social Graph - click through see a larger version.

Each vertex is a friend of mine on facebook. There is an edge between two friends if they are friends with each other. I'm not actually in this graph, otherwise there would be a vertex with an edge to every other vertex.

I called out my wife, Cristin, who bridges several regions of my graph. Even without me calling out that node, it would have been easy to guess.

The green region is high school. The blue region includes Google employees, which is naturally over-represented. College gets broken up a bit, and I wasn't really using facebook much for keeping in touch there. You can see some other dense clusters in here as well, but I'd didn't bother labeling them.

To me, this reinforces the Circles concept pretty well!

Technical Details:
I installed a firefox plugin that saved a copy of every page I visited. I then turned off javascript and clicked through to all of my friends to extract the data. I'm sure there is any easier way to do this, but I would have spent just as long figuring it out for this one-off.

Parsing the pages was some hacky python and regex. Some of the templates weren't parsed as easily as others, so a few friends just got dropped due to laziness. Any friend who didn't have any mutual friends also got dropped. However, I didn't require a connected graph. That just happened on it's own, to my surprise.

The graph was laid out using neato, and then I added the colored boxes on top by hand using Gimp.

With neato, I can change a line and get the nodes as text boxes with names instead, which is fascinating to look through, but not something I feel comfortable sharing publicly.

Also note that I don't do a good job of maintaining facebook, and have had purge cycles many times in the past. A larger graph with a bigger picture would be even more fascinating I think.

Find me over on Google+

2011-07-03T16:24:00.000-07:00

Don't worry, I've been blogging for 9 years and no social network is going to stop me any time soon.

That said, Google+ is a social network that I can share with a wider audience. Lots of folks tend to read my blog: techies, SEOs, etc. I didn't want the same large crowd seeing all of my personal facebook content that I was sharing with close friends and families.

Google+ lets me filter who sees each post using the "circles" interface. So, if you find this blog mildly interesting, I encourage you to follow me on Google+ as well: Gregable on Google+.

Dilbert on SEO

2011-06-24T08:30:00.000-07:00

Today Dilbert learns that only organic linking works. Links such as: Dung for Brains

The Nap Pod Awakens: Sweet Dreams

2011-04-27T12:52:00.000-07:00

If you've ever toured one of the Google Campuses, you may have seen one of our "Nap Pods" designed to let folks get a short refresher nap if they are feeling the need. There happens to be one such pod right outside my office. With someone's legs sticking out, it often looks an awful lot like pacman just ate an employee.

Several of the search folks got together yesterday and we brought our little nap pod to life. It's a bit cheesy largely because we were limited to the primary colors of electrical tape we had on hand. I still think it turned out great. Either way, the process was definitely a lot of fun. What do you think?

Sweet Dreams

Secure Snapshotted Backups with python, encfs, and rsync

2011-03-05T23:29:00.000-08:00

I recently made a very dumb mistake and wiped out /home/greg/ on my personal desktop. It wasn't a hardware failure, it was user error. I had some manual backups, it wasn't catastrophic. I had never really set up a good system though, and there were some annoying losses. After restoring my sanity I decided it was time to set up better backups. I run a NAS with ~3T of usable disk space and part of the reason for running this was backups. My failure to set up a backup system was the only thing missing.

Obviously lots of people have solved this problem before, but my situation was unique enough that many options were off the table. I had a few requirements:

My home directory (primaries) is encrypted with Ubuntu's TrueCrypt setup, the documents I want to back up the most are financial in nature, and so I wanted my backups always encrypted on disk.
I want snapshotted backups so that I was resilient against hardware failures, but also from rm -rf stupidity.
However, I did not want to simply store a series of diffs as that would make recovery more complex and I want recovery to be simple.
Still, I wanted efficiency and speed so I wasn't choking my internal network at various points in the day.

Most importantly though, I wanted to understand exactly how my backup system works and what it's doing. Rather than trust some other code that I didn't understand and couldn't tweak, I wanted to roll my own. Most folks do this with shell scripts, cron, and rsync. I wanted to do something similar, but since my shell-foo is abysmal, I decided on python.

If this is useful to anyone else, I've shared my code. The script has two modes controlled by arguments: backup and snapshot

Backup:

Optionally tries to mount a path which should be set up in /etc/fstab. In my case, this is NFS.
Mounts an encrypted filesystem at /mnt/.../current/
Rsyncs a series of files and paths to /mnt/.../current/
Optionally unmounts the encrypted filesytem.

Snapshot:

Makes up to N periodic snapshots of the encrypted files at one of several frequencies. For example, it might be configured to keep 24 hourly snapshots, 7 daily snapshots, 4 weekly snapshots, and 3 monthly snapshots. Any number of snapshots can be kept at any frequency.

The snapshots are taken of the encrypted files, not from the decrypted filesystem. As a result, you can run the snapshots directly on the remote backup system, I run it on my NAS. It works just fine if you run it locally as well.

Both modes are managed using a .backuprc file in the user's home directory. For example, mine looks something like this:

# Optional, log all events
LOG_FILE /home/greg/logs/backup.log

# Optional, we try to mount this path first.  Failures halt execution.
PRE_MOUNT /mnt/backup/

# Required, password and mount point for encrypted/decrypted file 
# systems. The password can be in plaintext since this file is stored
# on an encrypted filesystem anyway.  We aren't going for paranoid.
ENCFS_PASSWORD AddYourOwnPasswordHere
ENCRYPTED_MOUNTPOINT /mnt/backup/desktop/
# This is where we will write files unencrypted.  Must be empty, must
# not be mounted already.
DECRYPTED_MOUNTPOINT /mnt/encryptedbackup/

# Required, rsync flags.
RSYNC_FLAGS -CRa --delete

# Number of snapshots.  Format: [type=,...] e.g. hourly=12,daily=7
SNAPSHOTS hourly=12,daily=7,weekly=2,monthly=1

# List of file paths to rsync.  Any line that doesn't contain a space is
# a file path.  Paths can be filenames or directories.  This is simply
# the argument passed to rsync.  As a result, you can use rsync features
# like adding a "./" directory to tell rsync which components of the
# path to sync over.
/home/greg/./.heartbeat
/home/greg/./src/
/home/greg/./financial/
/home/greg/./picasa/

The python source, an example .backuprc and an example crontab are all found over here on github.

Some other helpful resources I came across while putting this together:
ReadyNas Root Access Add On
Install Python2.6 on a ReadyNas NV
RBackup - Diff based backups with python
The ultimate guide to rsync backups
How to set up encfs for use with rsync

Favorite Chrome Browser Extensions

2011-02-14T11:03:00.000-08:00

Everyone eventually does one of these posts right? If you aren't running the Chrome Web Browser, you should try it (about 1/4 of my readers do). If you have tried it and still aren't running it, ignore the rest of this post.

AutoPatchWork: for paginated web pages, this magically makes the page into an infinite scroll, preloading the next page as you get close to the bottom. Great for reading stupid NyTimes articles that get paginated into 8 pages. Also forum threads, search results, etc. I've never had it do anything to screw up a page, although it has occasionally failed to trigger, but that's no worse than what I was previously doing. This extension has simply saved me a ton of time and helps me use the mouse less.

PDF/Powerpoint viewer: Converts all links to a .pdf or .ppt to link instead to google docs' inline viewer. I hate waiting to load a large application every time I click on these links, this extension makes my experience much better. Doesn't work if the document can't be fetched from the web (password protected, internal network, etc), but in those cases you can right click and just load the large application instead.

Google Translate: Auto-detects if the page you are looking at is in a different language and offers to translate the page to your language of choice (english for me).

Pandora: - Not the official Pandora extension, that one is pretty worthless. This one integrates your pandora controls into your browser. Only useful if you use Pandora of course.

Favorite Doodle: Change google's homepage doodle to your favorite on days when there isn't a special one running. I wrote about this in 2009.

Secbrowsing: Chrome itself is known to be very secure, but many of it's plugins (PDF, Flash, Java, etc) aren't as secure. This extension will monitor to make sure that you've got the most secure versions of all of these plugins and helps you upgrade them if you don't.

Goo.gl Url Shortener: Click button, get short URL. From the fastest/most reliable url shortening service. I'm biased though.

Facebook Disconnect: Blocks facebook from tracking you when you are not on facebook.com (ie: from like buttons or other javascript widgets). Make sure to go into your extensions and enable this when incognito browsing too.

23++: If you use 23andMe, adds some much needed enhancements to the website like telling you automatically which surnames match for relatives.

What are some other good ones I haven't tried? I'm always looking for good extensions, especially those that operate in the background making my life better without me having to be aware of them.

Bulk Cookie Pricing

2011-02-07T18:42:00.000-08:00

I spotted this gem at Subway this afternoon. How much am I saving again if I buy 3?

Referrer Stripping

2011-01-27T13:51:00.000-08:00

Inspired by Gabriel Weinberg's strange fixation on stripping search queries from HTTP referrers, I thought it would be interesting to describe how to actually strip referrers from an HTTP Request. I don't think I've seen any good information on this elsewhere on the web. It's a neat trick, albeit marginally useful.

First, let me explain what Gabriel is doing so as to compare. If you do a query on Duck Duck Go and click a result, javascript on that page intercepts the click (in a sense) and sends you to http://duckduckgo.com/post.html instead. http://duckduckgo.com/post.html is just static HTML with some more javascript that then re-sends you on your way to the actual destination page you had in mind, so the referrer sent to the destination is http://duckduckgo.com/post.html. Duck Duck Go prefetches this URL before you click on any URL and your browser caches it, so there is only a tiny latency hit from running the javascript, and no network delay. Reasonably clever.

This doesn't in fact strip referrers at all, it just changes them so that the referrer sent to the destination page is simply http://duckduckgo.com/post.html. The query is gone, but the fact that you did a query and that you did so on Duck Duck Go isn't. I'd honestly suspect the fact that one uses Duck Duck Go to be more revealing about demographics than the query itself, but I digress.

You as a user can strip referrers by modifying your browser: [Chrome, Firefox, Opera, IE?, Safari?] What if you as a webmaster really want to strip the referrer for your users clicking on your links? That's a browser feature and as a webmaster you can't change your users' settings, you can't, right? Turns out that you can, but it's a pain in the ass because you need to do different things in different browsers.

With later versions of Webkit and hence Safari and Chrome, it's well known that attaching rel=noreferrer to an anchor will successfully tell the browser to not send the referrer on that request. If you want to do this in javascript instead of through a plain anchor, you can make it happen by creating an anchor element in the DOM and then simulating a click event via event.initMouseEvent. Older versions of Safari and Chrome don't work here, and I don't know a workaround, but these browsers auto-update, so it's not common to see really old versions.

Firefox and IE don't support rel=noreferrer as far as I know. However for as far back as I've tested them, a web page that performs a meta-refresh to a destination URL, even a 0-second refresh, will not send any referrer to that destination URL. This doesn't work in Webkit - it passes a referrer, but for some reason it works in Firefox and IE. It's undefined behavior, nowhere in the spec, so it could change in later versions of these browsers, buyer beware. However rel=noreferrer is part of HTML5, so later versions of these browsers will probably eventually work with rel=noreferrer. Want to do this with a plain link or javascript? Simply stick an intermediate page with the meta refresh as an in-between URL like Gabriel does with javascript, and you'll have the same effect.

Konqueror and Opera don't allow any of these tricks last I checked (it's been awhile), and for your various other browsers (phones mostly), all bets are off.

I don't really know why you would want to strip referrers as a webmaster. It literally is "breaking" the way browsers and the internet are supposed to work. I've used it occasionally for internal systems (like a control panel) where you don't want a referrer to expose the existence of an URL not usually accessible, but one can use obfuscation with a different URL in this case, as Gabriel does. Obfuscation is guaranteed to work in all browsers, and so is simpler to implement and maintain.

Preserving Digital Remains

2011-01-02T22:27:00.000-08:00

Lately, I've started researching my genealogical history a little bit. Intrigued by discovering previously unknown relatives within 23andMe, I've spent some time combing through public records, bought a couple more 23andMe kits for living ancestors, and started talking to my family to see what they remember while they are still alive. It's a fascinating puzzle problem.

One thing that I've started wondering about is the future of the content I'm creating during my lifetime. This goes beyond backups: I 'm reasonably skilled at protecting my data while I'm still alive. I'd like to have some of my data survive me. I want my story to be immortal. I have no clue how I would go about making such guarantees. Lots of organizations are trying very hard to bring data from the past into the present, but how would one go about pushing data from the present into the future?

This isn't just motivated by ego, although that's partially true. For example, it would be useful for my future relatives to be able to know my medical history to understand their own risks. Sshould I become famous after my death for some odd reason, it would be historically interesting to have more details on my life story. As someone living in my ancestor's distant future, I would absolutely love it if I could comb through the digital remains of my great great great grandparents who lived during the civil war.

It's an interesting problem. How can I assure that the content in my blog posts will be immortal and searchable for all time. Can I safely assume that blogger will keep my blog running forever? Yahoo closing Geocities makes it pretty clear that there is no guarantee of persistence in free services. Lots of companies have appeared that allow me to send data to family immediately after my passing, but what if the future historian who is interested in my data hasn't yet been born, and family that outlives me doesn't take much care to preserve my data?

One approach would be to create a company designed for this purpose that would attempt to outlive it's employees. That company would require a large upfront payment for storage, for example: $100/GB. The price would be calculated such that interest on a safe investment of that size could safely expect to cover ongoing archival costs for eternity along with some profit.

Even this though seems risky. What if interest rates decline and storage costs increase (peak oil causing a rise in electricity cost for example). What if the company changes hands and the new owners decide that the best option for shareholders is to delete all old data and cash out the interest bearing bank account? Maybe the executives decide to invest in something a little risky to improve profits and the company goes bankrupt.

Maybe we could back this organization by some large government, such as the US. Costs are still paid as a large upfront chunk, so it would require no taxpayer burden unless there is a gravely bad estimation of the ongoing costs. Presumably it would be politically unpopular to risk losing this data, so there would be more pressure felt from historians or the like than if the organization was profit motivated. Governments have historically done at least a reasonable job of preserving records, such as Census data or birth certificates.

I think lots of people would pay a reasonable cost for this service. Perhaps most wouldn't while they were living, but as part of a funeral service, a mortuary could accept a box of writing or a CD or whatever and scan/upload all of the data found for the deceased. Compared to the rest of the costs of funerals, this line item would be pretty modest. If there were privacy issues, you just put some kind of digital seal on the data: do not open for 100 years. Future Historians would presumably find this to be a very valuable trove of information.

Network Aware Sheetfed Scanner

2010-12-01T21:43:00.000-08:00

I'm hoping someone out there has done something like this before and can help me out. I want to set up a sheetfed (where I drop a stack of papers in at once) scanner at home. I want to have that scanner either connected to the home network (ethernet or wireless) or connected to a headless linux box on the network (via usb presumably).

I want it set up so that I drop in a multipage document, hit some single button - ideally a button on the scanner, but I'd accept something I could trigger over the network on the linux box. At this point, the scanner goes to town, makes a pdf (or any image) with all the pages currently in the tray, and then drops this PDF on the network via NFS / Samba / FTP / whatever I can run from linux. I'll come along later and organize / rename this file, but until then I don't want to have to go to a computer, open up some application, scan, preview, save, ...

I'm willing to buy whatever scanner and/or other devices, up to a few hundred dollars. I'm happy to write a little code, install libraries, etc. I don't need anything fancy like OCR.

I've heard of some products that do stuff like this when hooked up directly to a windows / mac machine that is already running, but I don't leave my windows/mac machines running. I do however have a small linux machine that is always on. I've also seen a scanner that will do this completely over the network, with a fancy touchscreen interface and such, but it was north of $1,500 which is out of my price range for this.

Any ideas? Pointers? I've searched around on the Google for some sage advice, but I think I'm running the wrong queries.

Squanto: The original American Badass

2010-11-24T21:39:00.000-08:00

Thanksgiving is probably one of my favorite American holidays. For most people, it means a day or two off work and some awesome food with some family and friends.

Everyone knows the basic Thanksgiving myth - the Pilgrims landed in Plymouth after enduring a tortuous journey only to face a really bad winter storm. They met the Native American locals, including some guy named Squanto, who taught them how to fish, farm, and essentially survive. They all had a big feast together, and lived happily ever after. This all mythically represents the seeds of a new country, the good old US-of-A.

The true story in these things is sometimes dull and always full of facts and dates. In this case though there is a quite interesting true thanksgiving story within the real Plymouth Pilgrim landing. I'm referring to Squanto's story, whose tale blows the Illiad out of the water. Squanto was the original American Badass.

When the Pilgrims arrived in Plymouth, they explored the area before settling down. They choose Plymouth because of it's empty fields, freshwater brooks, pre-built homes, etc. Hardly starting from scratch, they had pitched camp right in the middle of Squanto's lovely village of Patuxet, appropriating native american cornfields and homes for themselves. One colonist noted "In this bay wherein we live, in former time hath lived about two thousand Indians.". Not only the cornfields, but they also set to work robbing native american homes and graves. In their weak defense, much of Patuxet's population had recently succumbed to a plague, probably smallpox, that had arrived just ahead of the Pilgrims.

Despite taking over an existing settlement, over half of the colonists died in the first winter. It was their superb luck that the next spring, a local walked into Plymouth and said in perfectly accented English "How can I help you gentlemen?". This local was Squanto and Patuxet (Plymouth) was his hometown. Of course, the Pilgrims didn't expect a local to speak English. They had no way of knowing that Squanto had just recently returned home from Europe.

Let's go backwards in time. In or around 1605, the Englishman Capt. George Weymouth was exploring the Maine and Massachussets area. Capt. Weymouth decided to capture 5 natives and take them back to England, fairly brutally. Weymouth wrote, "For they were strong and so naked as our best hold was by their long hair on their heads.". One of these natives is believed to have been Squanto, at the time referred to as Tisquantum. Weymouth gave the natives to Sir Ferdinando Gorges, an investor. Gorges presumably trained Squanto as a guide for sea captains exploring the New England coast, teaching him English along the way.

We don't know much about what happened to Squanto from 1605 until 1614, for all we know, he made several expeditions for Gorges during this period.

In 1614, Squanto joined a small fleet (2 ships) with Capt. John Smith where he was promised to return to his people in Patuxet in exchange for his help as a guide. At some point the fleet reached Patuxet and separated. Squanto remained in Patuxet with the captain of the 2nd ship (Thomas Hunt), and Capt. Smith headed north. Hunt was supposed to fill his ship with beaver skins and head home. Instead, Hunt kidnapped 20 natives including Squanto, and took them back to Spain to sell as slaves.

When Squanto arrived in Spain, some local monks discovered the plan, didn't like it, and "rescued" the surviving Indians from Hunt "to instruct them in the Christian faith". Squanto lived with the monks for a year or two, but eventually traveled back to England.

In England, Squanto arranged to travel on another expedition to Newfoundland. In Newfoundland Squanto was recognized as belonging to Sir Ferdinando Gorges, was recaptured, and taken back to England again. From there, Sir Ferdinando Gorges launched another expedition back to New England with Squanto. Their stops included dropping anchor in Plymouth Harbor in 1619, about a year before the Pilgrims arrived. This time Squanto found that every single man, woman, and child that he had once known in his home of Patuxet had disappeared in the intervening 5 years. They had either been wiped out by the plague or were so afraid of the plague that they had fled to escape it. Squanto, who must have been heartbroken, decided to join the Pokanokets, in Rhode Island.

Squanto later received news of the Pilgrims settling in Patuxet and he travelled to visit them. He joined their society as a guide, never to leave them again until his death. During this period, he commanded great power in both Native American and New England societies because of his unique familiarity with both worlds.

3 years later, in 1623, the Pilgrims had another poor corn harvest and were searching for corn to get them through the winter. Squanto was able to lead a trading expedition with local indians that successfully secured for the Pilgrims the corn and beans that the needed. During that expedition however, Squanto came down with a fever and died suddenly. Before his death, he talked with Governor Bradford and asked him "to pray for him, that he might go to the Englishmen's God in heaven, and bequeathed sundry of his things to sundry of his English friends as remembrances of his love, of whom they had a great loss".

A little more interesting than some local who taught the Pilgrims to grow corn, eh? Happy Squanto Day.

Google Birthday Doodle

2010-10-17T15:38:00.000-07:00

A friend of mine just launched a neat easter egg on Google earlier this week. If you have specified a birthday in your Google Profile (inexplicably under the Contact Info section), we'll show a special doodle to you on the front page of Google on your birthday. The short webcast video above is from Danny Brown who stumbled across this fun easter egg accidentally.

My birthday is right around the corner, I'm looking forward to seeing the doodle. Although I will admit that I cheated on the launch date and set my birthday to that day instead. For testing purposes only. Yeah, that's it.

Goo.gl URL Shortener

2010-09-30T14:51:00.000-07:00

Friends and coworkers of mine just launched Goo.gl, an URL Shortener run on Google Infrastructure.

To me, stability is the biggest selling point of Goo.gl over alternatives - it's run by a company that isn't likely to disappear next year and is known for having some of the most scalable systems in the world. I know the team has worked hard on making this as scalable and reliable as most anything at Google, including search. I think it's safe to say Goo.gl links are any one's best bet as far as future-proofing your shortened URLs. Also, because of Google's obsession with end-user speed, this is likely the fastest URL shortener you are going to ever find. You don't have to take my word from it, see this post from TechCrunch a few months ago that shows Goo.gl as the fastest and most reliable from two different third-party analyses.

Reliability, Greener is Better

Speed, Smaller is Better

Of course, I'm the first one to admit that I'm disappointed that we even need URL Shorteners, but if you assume that you indeed do, this is a great choice.

Why you should learn just a little Awk - An Awk Tutorial by Example

2010-09-29T01:24:00.000-07:00

In grad school, I once saw a prof I was working with grab a text file and in seconds manipulate it into little pieces so deftly it blew my mind. I immediately decided it was time for me to learn awk, which he had so clearly mastered.

To this day, 90% of the programmers I talk to have never used awk. Knowing 10% of awk's already small syntax, which you can pick up in just a few minutes, will dramatically increase your ability to quickly manipulate data in text files. Below I'll teach you the most useful stuff - not the "fundamentals", but the 5 minutes worth of practical stuff that will get you most of what I think is interesting in this little language.

Awk is a fun little programming language. It is designed for processing input strings. A (different) prof once asked my networking class to implement code that would take a spec for an RPC service and generate stubs for the client and the server. This professor made the mistake of telling us we could implement this in any language. I decided to write the generator in Awk, mostly as an excuse to learn more Awk. Surprisingly to me, the code ended up much shorter and much simpler than it would have been in any other language I've ever used (Python, C++, Java, ...). There is enough to learn about Awk to fill half a book, and I've read that book, but you're unlikely to be writing a full-fledged spec parser in Awk. Instead, you just want to do things like find all of your log lines that come from ip addresses whose components sum up to 666, for kicks and grins. Read on!

For our examples, assume we have a little file (logs.txt) that looks like the one below. If it wraps in your browser, this is just 2 lines of logs each staring with an ip address.

07.46.199.184 [28/Sep/2010:04:08:20] "GET /robots.txt HTTP/1.1" 200 0 "msnbot"
123.125.71.19 [28/Sep/2010:04:20:11] "GET / HTTP/1.1" 304 - "Baiduspider"

These are just two log records generated by Apache, slightly simplified, showing Bing and Baidu wandering around on my site yesterday.

Awk works like anything else (ie: grep) on the command line. It reads from stdin and writes to stdout. It's easy to pipe stuff in and out of it. The command line syntax you care about is just the command awk followed by a string that contains your program.

awk '{print $0}'

Most Awk programs will start with a "{" and end with a "}". Everything in between there gets run once on each line of input. Most awk programs will print something. The program above will print the entire line that it just read, print appends a newline for free. $0 is the entire line. So this program is an identity operation - it copies the input to the output without changing it.

Awk parses the line in to fields for you automatically, using any whitespace (space, tab) as a delimiter, merging consecutive delimiters. Those fields are available to you as the variables $1, $2, $3, etc.

echo 'this is a test' | awk '{print $3}' // prints 'a'
awk '{print $1}' logs.txt

Output:
07.46.199.184
123.125.71.19

Easy so far, and already useful. Sometimes I need to print from the end of the string though instead. The special variable, NF, contains the number of fields in the current line. I can print the last field by printing the field $NF or I can just manipulate that value to identify a field based on it's position from the last. I can also print multiple values simultaneously in the same print statement.

echo 'this is a test' | awk '{print $NF}' // prints "test"
awk '{print $1, $(NF-2) }' logs.txt

Output:
07.46.199.184 200
123.125.71.19 304

More progress - you can see how, in moments, you could strip this log file to just the fields you are interested in. Another cool variable is NR, which is the row number being currently processed. While demonstrating NR, let me also show you how to format a little bit of output using print. Commas between arguments in a print statement put spaces between them, but I can leave out the comma and no spaces are inserted.

awk '{print NR ") " $1 " -> " $(NF-2)}' logs.txt

Output:
1) 07.46.199.184 -> 200
2) 123.125.71.19 -> 304

Powerful, but nothing hard yet, I hope. By the way, there is also a printf function that works much the way you'd expect if you prefer that form of formatting. Now, not all files have fields that are separated with whitespace. Let's look at the date field:

$ awk '{print $2}' logs.txt

Output:
[28/Sep/2010:04:08:20]
[28/Sep/2010:04:20:11]

The date field is separated by "/" and ":" characters. I can do the following within one awk program, but I want to teach you simple things that you can string together using more familiar unix piping because it's quicker to pick up a small syntax. What I'm going to do is pipe the output of the above command through another awk program that splits on the colon. To do this, my second program needs two {} components. I don't want to go into what these mean, just to show you how to use them for splitting on a different delimiter.

$ awk '{print $2}' logs.txt | awk 'BEGIN{FS=":"}{print $1}'

Output:
[28/Sep/2010
[28/Sep/2010

I just specified that I wanted a different FS (field separator) of ":" and that I wanted to then print the first field. No more time, just dates! The simplest way to get rid of that prefix [ character is with sed, which you are likely already familiar with:

$ awk '{print $2}' logs.txt | awk 'BEGIN{FS=":"}{print $1}' | sed 's/\[//'

Output:
28/Sep/2010
28/Sep/2010

I can further split this on the "/" character if I want using the exact same trick, but I think you get the point. Next, lets learn just a tiny bit of logic. If I want to return only the 200 status lines, I could use grep, but I might end up with an ip address that contains 200, or a date from year 2000. I could first grab the 200 field with Awk and then grep, but then I lose the whole line's context. Awk supports basic if statements. Lets see how I might use one:

$ awk '{if ($(NF-2) == "200") {print $0}}' logs.txt

Output:
07.46.199.184 [28/Sep/2010:04:08:20] "GET /robots.txt HTTP/1.1" 200 0 "msnbot"

There we go, returning only the lines (in this case only one) with a 200 status. The if syntax should be very familiar and require no explanation. Let me finish up by showing you one stupid example of awk code that maintains state across multiple lines. Lets say I want to sum up all of the status fields in this file. I can't think of a reason I'd want to do this for statuses in a log file, but it makes a lot of sense in other cases like summing up the total bytes returned across all of the logs in a day or something. To do this, I just create a variable which automatically will persist across multiple lines:

$ awk '{a+=$(NF-2); print "Total so far:", a}' logs.txt

Output:
Total so far: 200
Total so far: 504

Nothing doing. Obviously in most cases, I'm not interested in cumulative values but only the final value. I can of course just use tail -n1, but I can also print stuff after processing the final line using an END clause:

$ awk '{a+=$(NF-2)}END{print "Total:", a}' logs.txt

Output:
Total: 504

If you want to read more about awk, there are several good books and plenty of online references. You can learn just about everything there is to know about awk in a day with some time to spare. Getting used to it is a bit more of a challenge as it really is a little bit different of a way to code - you are essentially writing only the inner part of a for loop. Come to think of it, this is a lot like how MapReduce feels, which is also initially disorienting.

I hope some of that was useful. If you found it to be so, leave a comment to let me know, I enjoy the feedback if nothing else.

Update Sep 30, 2010: There are some great comments elsewhere in addition to here. I wish they would end up in one place, but the best I can do currently is to link to them:

Update Jan 2, 2011: This post caught the interest of Hacker Monthly who republished it in issue #8. You can grab the pdf version of this article courtesy of Lim Cheng Soon, Hacker News' Founder.

Birds of a Feather

If you are the type of person interested in Awk, you are probably the type of person I'd like to see working with me at Google. If you send me your resume (ggrothau@gmail.com), I can make sure it gets in front of the right recruiters and watch to make sure that it doesn't get lost in the pile that we get every day.