Gregable.

Battery Voltage Regulation Prediction Software

2010-02-26T13:12:00.000-08:00

It's rare that I can talk about green energy and software issues in one post. Here's some software that someone needs to write.

Over at "This week in batteries", Venkat Srinivasan explains the tradeoffs in modern battery charging. Essentially, the problem is that the higher voltage you charge a battery to, the bigger capacity it has in terms of discharging. However, higher voltages at the same time reduce the lifespan of the battery so that the next charge will have a little less capacity. Virtually all battery chargers choose a fixed voltage to charge up to and then stop charging, and this chosen voltage is a number that gives a good balance between capacity and lifespan, but doesn't really maximize either. From the article:

Operating to 4.1V makes things better and extends the life, 4.0 V is even better and so on. So why don’t battery manufacturers cut the voltage off at, say, 4 V to get better battery life? Because every time you cut this voltage down you decrease the capacity of the battery and its run time. The 4.2V cutoff is a compromise between good run time and decent (read “not pathetic”) life.

Venkat goes on to suggest that this means you should charge your devices up to the maximum limit and then unplug them, so that they slowly drop a little below the max, to extend lifetime. He also mentions that Lenovo laptops even let you set the charge (as a percentage, not a voltage) if you want to play with these settings yourself.

This is a problem not just for laptops, but also for cell phones, plug-in cars, or any device that uses a battery. I'm certainly not a battery expert, but it seems to me that there is a better solution out there.

Most devices using an expensive battery have some amount of processing power and some amount of long-term memory (flash drive, hard drive, etc). Laptop, Cell Phone, and Vehicles certainly fall in to this category. If the device was able to predict roughly how long it would be used before the next recharge, then it would be able to charge to the lowest voltage required to stay alive until the next recharge.

Humans are creatures of habit, we often keep roughly the same schedule from day to day or week to week. If the device kept a history of these habits, it should be able to predict usage. For example, storing data like:

Plug in timestamp
Unplug timestamp
Running out of juice timestamp
Day of week, time of day
which user is logged in, or how much they weigh on the driver seat sensor
ip address the device is allocated to (or nearby wireless APs)
GPS location
applications that were run

Using a little bit of simple machine learning, the software could predict how much capacity is needed for the next unplug and chage accordingly. Thus if the laptop is always plugged in at a desk except on the weekend, a low voltage charge could be maintained and then ramped up all the way saturday morning. Or if the car is used for short trips to/from work, but on Friday night is driven to the next city for happy hour with friends, you can keep the voltage high for happy hour and low for the commute. If the device sees a completely new scenario (suddenly turned on in Japan at 3am on a different network), then it might switch temporarily to a high voltage/capacity until enough data comes in to know what to expect in this new scenario.

This approach would maximize both battery life and capacity while keeping the user blissfully unaware.

My take on Buzz

2010-02-15T16:21:00.000-08:00

Imagine that to read this blog, you had to first sign into your Blogger account. And that to read a Wikipedia article, you first had to sign into your Wikipedia account. And, when you want to read your email from your AOL friends, you have to first sign into your AOL account, but to read your email from GMail friends required first signing into your GMail account. And imagine this was the case everywhere. You could even take this further and imagine that to read CNN, you first had to load the CNN browser.

To various degrees this is the current state of microblogging services (or whatever you call them). I can't read my friend's Facebook updates in google reader. I can't read the Boston Big Picture in Twitter. Even if my friends wire up their accounts so that their tweets end up in Facebook, my Facebook replies don't appear in their Twitter stream. It's a big bucket of suckage.

As a user, I don't care about how the electricity gets to my house, but I really really care that all my appliances can use that electricity, not just the ones that PG&E made. So I care that someone else cares about the how and standardizes this. As a publisher, I want to be able to pick my publishing platforms - maybe more than just one (Flickr+Twitter+Blogger) - based on how well their features meet my style. I don't want to have to choose my platform based on where my readers might be. As a reader, I want to pick my reader application(s), like Google Reader, Bloglines, or Facebook, but only have to pick one or maybe two and to pick them based on features and usefulness to me, not based on where the people I'm interested in reading publish their content.

Switch gears. Google Buzz launched last week. We've been trialling it internally at Google for some time now, but I had pretty much tuned it out. It was just one more social network with mostly inconsequential status updates of which I was overwhelmed with already. Now though, I realize that I had duped myself. Buzz is not necessarily just another walled garden (although, there is still that risk).

Time out. I am an employee of Google. I don't work on Buzz, heck I have less to do with it at the time of writing this than most Google employees having largely ignored it for so long. The opinions in this post, as well as elsewhere in this blog, are my own and not those of my employer. Seriously. Nuff Said.

Buzz could be a move to a more open standard for microblogging. This ReadWriteWeb article on how Buzz's API supports Open Data Standards says it better than I. I spent some time in the last few days reading up on PubSubHubBub, Salmon, WebFinger, etc. I'm no expert, but it's exciting. Whether Buzz succeeds or not matters little to me, but if this kind of open social networking standard takes off, I think we're all going to be a little bit saner in managing our digital lives.

Swoopo and stable equilibriums

2010-02-05T11:10:00.000-08:00

If you've never heard of swoopo.com, it's an "auction" site that allows you to bid on items in 1c increments. Usually the items sell for much much less than they are worth. For example, this 1oz gold bar (worth around $1,400) sold for $216.98. The trick is that you can only bid exactly 1c more than the last bid and simply making a bid costs you $0.60. So, 21,698 bids were made on that gold bar by various people which cost them over $13,018 in total. The winner spent almost $700 in bids themselves. Most people who play this game are losing lots of money.

A similar (simpler) game is described as an auction for a $20 bill. The rule is that you pay your most recent bid, regardless of whether or not you win. You can argue that it always makes sense for you to bid. Initially, you are bidding $1 for a $20 bill, how awesome is that? However your opponent in the auction can now bid $2 for a $20 bill, which he will definitely do - why wouldn't he? At some point you have bid $19 for this bill, but your opponent recently just bid $18. If he doesn't bid $20, he loses $18 - if he does bid $20 and wins, he comes out even. So bidding $20 seems like the right choice. Unfortunately now you lose $19 bid lost if you don't bid and you lose only $1 if you do bid (bid = $21, but you win $20). So it makes sense for you to actually bid $21 for a $20 bill. This continues forever with the risk that both participants bid hundreds of dollars for this $20 bill.

I was having a lunch conversation with some folks the other day who argued that the best choice at any turn is to always bid, but that this is a paradox because you will lose your shirt playing this game. I think the paradox is easily resolved. The thing that causes the paradox is the assumption that the best decision is to always bid, which seems on the surface to be the right strategy but in fact is not. If you assume this is the right strategy, this game has no stable nash equilibrium, but I claim that the game does in fact have such an equilibrium strategy.

Imagine your opponent in this game (simplification - the game is a 2 player version of the $20 auction) is one instance of a set of players in the universe of possible players. As such a player, there is a distribution of points in the auction where that player will stop bidding. The number of players that will make it to $1 is high, say 99%, the number of players that make it past $20 is probably somewhat low, and the number of players that make it to $1 million is probably nearly zero. As a result, you can evaluate at each point in the game what is the expected number of additional bids that you will have to make to gain the $20 bill. If this is greater than 10 bids (an increase of $20 in your bidding), then you should not bid. If it is less, then you should bid. Depending on the distribution, you may choose not to even play the game or you may likely choose to play up to a few dollars or so because some reasonable fraction of other players give up early. If you played enough such games, you would expect to come out non-negative however on some individual games you would expect to lose money. My guess is that the real distribution for a $20 auction in the world of opponents would be such that not bidding at all makes sense, but if you make it a $1 million auction, then I would guess that due to finite budgets of your average opponent, it would make sense to bid up to some reasonable number and give up.

Interestingly, there are some distributions that would suggest you should never stop bidding. For example if the probability that your opponent continues to bid decreases by 50% every time, then at any point in the game, you would expect to have to bid only 2 more times, meaning you should always keep bidding. However, this is fine as long as your estimate of the distribution is correct because the odds of your opponent bidding 10 more times are tiny (1 in 1024). So you are essentially betting on a fair coin toss repeatedly with a 1:10 payoff.

I have no idea what the distributions are for swoopo auctions. If someone has the dataset, it would be interesting to analyze it and figure out the optimal strategy and see if it ever makes sense to bid (maybe it does in some cases).

rel=canonical part two

2009-12-15T20:09:00.000-08:00

Google today officially announced support for cross-domain rel=canonicals. This is a very important step. I talked about rel=canonical when we announced it 10 months ago. This tweak will be very useful in a variety of cases. A few interesting ones:

Any place where you want to move your site to a new domain but can't control server headers. This can be really useful to help avoid lock in on free hosts for example.
Companies that register several domain names as landing pages for offline advertising, a practice often used for tracking performance of the campaign, can now keep users on the original domain but keep spiders going to the right place so Google doesn't end up with 10 copies of the site.
Syndicating content - if your syndication partner agrees - you can now make sure that your "link juice" flows back to the original article in Google.

I think we'll see lots of creative uses of this tweak in the future as we've seen with the original rel=canonical. Let's just hope those creative talents are used for the power of good.

Favorite Google Doodle: A fun chrome extension

2009-12-08T15:00:00.000-08:00

Google announced extensions for Chrome Web Browser today, a much awaited feature for many. Go try out a few. If you are looking for recommendations, one quirky fun little extension that I would recommend is the Favorite Google Doodle Extension. It lets you select your favorite Google Doodle logo from Google's doodling history and set it as the logo for your Google search page whenever Google isn't running a special logo for that day:

Unfortunately, today isn't the best day since Google is actually running a special logo today celebrating the cartoonist E.C. Sagar, the man behind Popeye the Sailor. However, if you install and choose a logo today, you'll see it on the search results page, and likely tomorrow (depending on whether or not there is a special logo running then too). I've set might to be this fancy scientific logo for now. I found some other of my favorites too:

I will admit that I'm undeniably biased as a friend of mine actually wrote this extension. If you like it, don't forget to rate it too.

Simple Simhashing

2009-10-06T17:26:00.000-07:00

A friend and coworker of mine, Ryan Moulton, just wrote a very nifty knol called "Simple Simhashing". If you like software algorithms, it is most certainly worth a read. Simple simhashing is an algorithm that allows you to take any "thing" and come up with a simhash that should be the same for two "things" in probability relative to how similar those to "things" are. The really nifty part is that the algorithm is completely local and the properties are provable in an easy to understand way. This algorithm is actually used at Google in a few places, although I'll leave it to you to guess how.

Blacksmithing in the San Francisco Bay Area

2009-08-26T19:01:00.000-07:00

For the last 5 weeks, ending yesterday, Cristin and I have been making a once-a-week journey to Oakland to take a introductory blacksmithing class at the Crucible. The Crucible (thecrucible.org) is a non-profit in Oakland which serves as a avenue for the industrial arts in the community. They tend to focus on alot of arts related to fire, hence the name.

The class teaches you several basic blacksmithing techniques as you create 4 projects over 5 weeks. You create a hook, which is looks like an enormous fishing hook, but designed to hang on a wall. You create a knife, or a shiv out of rebar. You create a spoon - the two week project. Finally you create a 2-pronged fork. And most people create two of each - one piece heating up in the forge while the other piece is being worked on. Here is a photo of some of the steps in the various projects:

There are a few other photos up on picasaweb of Cristin and I here: Blacksmithing Photos, but the most important photo is the one that resolves the "pics or it didn't happen" issue:

Cristin and I had a blast, so a small plug for these classes. Turns out that in a few weeks, they are running a special weekend "sampler": . They have morning and afternoon single session classes on both Saturday, Sep 19 and Sunday, Sep 20. It isn't cheap, but it isn't horrible either, $85/class. If you take two on the same day, you get $15 off the set. Cristin and I going to head up there on Saturday, Sep 19 for the whole day and if anyone else is interested in going, let me know.

The classes offered as samplers include:

Blacksmithing: undamental skills needed to forge steel and understand blacksmithing tools. Each students will walk away with a small finished project.
Glass Fusing: glass/heat interaction, glass cutting, fusing (combining colors), slumping (shaping glass into or over molds to create flat and dimensional forms), as well as a brief description of firing and annealing procedures.
Glass Flameworking: fundamentals of flameworking, and will cover a small variety of techniques like color pulling and applying, frit application, and marble making.
MIG Welding: Metal inert gas (MIG) welding, also known as wire-feed welding, falls between arc welding and TIG in the welding spectrum. Quieter and cleaner (and some say easier) than arc welding, MIG welding is typically used in production fabrication such as furniture making, light construction, and auto restoration. In this workshop you’ll learn the basics of MIG welding and plasma cutting.
Ceramics: no description online, but I think this is sculpting.
Jewelry: sawing, filing, sanding, soldering, texturing, annealing, stamping, polishing, safety, and design. Each student will walk away with one finished piece.
Sand Casting: Working with resin-bonded sand, you’ll learn the foundry basics as you build your own molds. As part of a casting team, you’ll have the exhilarating experience of pouring molten aluminum.
Resin Casting: Learn the secrets of plastic casting using flexible molds, just like the ones that professionals use for product development. You'll learn to make molds of found objects and use them to create plastic reproductions. Each student will walk away with one piece.

Duplicate Content & Multiple Site Issues

2009-08-12T14:02:00.000-07:00

Earlier this morning I gave a talk at SES San Jose on Duplicate Content and Multiple Site Issues. For those who couldn't make it to the conference or just want a copy of the slides I'm embedding them here:

Keep an eye on the Google Webmaster Youtube Channel. I going to try to see if I can record the presentation and have it up there as well for anyone who missed it.

Update (09/17/2009):

The re-recorded video is now up:
http://googlewebmastercentral.blogspot.com/2009/09/duplicate-content-and-multiple-site.html

As well as a interview with Mike McDonald of WebProNews:
http://videos.webpronews.com/2009/09/17/canonical-tag-vs-301-redirect/

Join me at SES 2009 San Jose next Wednesday

2009-08-07T10:12:00.000-07:00

Next Wednesday morning, August 12, I'll be speaking at SES San Jose on Duplicate Content & Multiple Site Issues. Learn what happens when you have the same content appear on multiple pages or even sites. Find out if you are at risk of getting "penalized" or "banned" for having similar pages (hint: not likely). Learn a bit more about the new rel canonical tag, tried and true 301 redirects, and other great ways of helping search engines figure out what your site is all about.

Or just find me wandering around after the session, say hi, and hit me with any of your other burning search engine questions. I'm not as famous as Matt Cutts, so I might have a bit more time for a detailed discussion about all things search engine.

See you in San Jose.

Mileage Tax

2009-07-11T20:45:00.000-07:00

Last year around this time, I wrote a blog post suggesting we crank up the gas tax to $10/gallon while using the entire proceeds to give tax rebates to low income families. The reaction was generally plenty negative, although at least one person told me in person that he thought it might have some merit. No worries.

It turns out that there is a new idea for taxing vehicles on the horizon, one that is only recently possible due to the proliferation of inexpensive GPS technology. A tax based directly on mileage driven. I've found that very few people in the US have heard about this possibility, and most reject it at first glance without deeper consideration.

There was an interesting article last weekend in Reuters about Traffic jams in Manhattan. In it, the author posited that the average car driven into Manhattan on a weekday causes 3.26 hours of delays to everybody else. Those delays are felt by individual drivers in terms of a few seconds, but across all of the drivers combined, the delays are very high. The article goes a bit further to claim that the average car has 1.97 people in it and the average per-car value of that lost time is $48.89/hour, so the societal cost of that average car being driven into Manhattan is as high as $160.

In major cities, many road networks have similar properties to a lesser scale than Manhattan. Today I drove through San Francisco with no intent to stop in the
city, but traffic slowed me down to a much lower speed than I was happy with. The traffic alone added at least 30 minutes to my journey. Ick.

A mileage tax at it's simplest adds a device to your car which tracks how many miles you've driven via GPS and charges you a tax based on this. There are obvious reasons why this initial idea would have people roll their eyes. The first is usually that a gas-tax has much the same affect and is cheaper to administer. The second is that it seems to penalize folks who drive more fuel efficient vehicles (or at least not penalize gas-guzzlers).

The important part is that a mileage tax can be far more complex than this. Instead of a flat tax per mile driven, the tax can vary based on where you are driving, what time, and the type of vehicle you drive (ie: a discount applied to lower-emissions vehicles).

Also, a small premium can be charged for driving in San Francisco during rush hour. By charging say $1 to drive through San Francisco at lunch time, those willing to pay will have a shorter, faster trip because others are incented to use alternatives and get off the road. There would be no need for your unit to communicate where it was and when (violating your privacy), it would simply need to have rules for different areas and calculate the tax internally.

A variable rate mileage tax could also nicely solve the problem of toll roads. Toll roads in the US today are often built with the argument that state X shouldn't have to pay for the road that people in state Y are going to make use of for free. Simply collecting the tolls has a cost though. WSDOT did a study in 2007 (See Toll Roads Mean Billions in Extra Costs for Motorists) that indicated for every $100 in revenue raised by toll roads, there was an additional $22 overhead in collecting the tolls themselves. This also doesn't include the drivers' lost time.

A mileage tax isn't a theoretical idea that can't be easily implemented in practice either, it is already in use in some places. Last year Holland passed a law that will phase in a mileage tax over several years with variable rate pricing (See: Road Pricing). In 2006, Oregon recruited 299 drivers for a year-long trial of a mileage tax (See: Oregon's Successful Mileage Tax Experiment).

Obama has however said he is not interested. However, this may become a looming issue over time. As fuel efficiency increases and drivers react to higher gas prices, transportation budgets have been feeling serious shortfalls. It is politically difficult to pass or raise new taxes, which makes regularly updating the gas tax to match the transportation budgets sporadic at best. The time hasn't come, but perhaps it someday will.

Insultingly Stupid Movie Physics

2009-07-06T20:26:00.000-07:00

I recall a high school physics class where my teacher had us watch a clip from Speed 2. The basic plot of this movie, if you've never seen it, is that a cruise liner is taken over by some madman who has rigged the controls such that the cruise liner is on an irreversible collision course, initially with an oil tanker. Later, the cruise liner is redirected and actually runs ashore at full speed into a coastal town. As the ship runs ashore, it's speed drops and you see cut scenes of the speedometer slowly decreasing as the ship plows through a pier, a road, and some buildings. All the while you get long clips of destruction below the ship and the passengers getting thrown around the deck violently.

In physics class however, we pulled out stopwatches. We had to time the boat from the moment it hit shore to the point it stopped moving. Every speedometer cut scene, we'd note both the time and the current speed of the boat. Then we plotted acceleration. The acceleration was obviously uneven. Worse yet, it was tiny. Cruise liners don't move all that fast to begin with, and this scene was horribly drawn out for cinematic effect. We determined that rather than the passengers getting flung across the deck of the ship, they should have instead been sitting on the deck sipping martinis with no worry over spilling a drop.

It was a fun exercise. At the time, it was also a great excuse to watch a movie in class. That physics teacher was Tom Rogers, one of the standout teachers in my personal education. He runs a website, Insultingly Stupid Movie Physics, where he reviews movies from a solely physics standpoint - sound in space, exploding cars, exploding bullets. It's all there. Including a rant about the new Star Trek movie, which I personally did not enjoy much. He's been interviewed on NPR and has a book out all about movie physics. Definitely a fun way to waste a bit of time.

Maker Faire

2009-05-31T21:36:00.000-07:00

Went to the maker faire in San Mateo yesterday. Quite a crowd compared to my recollection of previous years - way too packed. I think next year I might go on Sunday to see if that reduces the crowd any. A few choice photos are below.

First, the cardboard surfboard.

The steampunk victorian house:

The shark car:

And the snail car:

A flaming flower:

A child with wings:

And an egg painting robot:

Tennessee Cove Trail

2009-05-31T21:26:00.001-07:00

View Tennessee Cove Trail in a larger map

Today Cristin and I took a short hike on a little known trail up in the Marin Headlands called Tennessee Cove. I had never been to this area of the park, and it was nice to explore somewhere new so close to home. This hike is 1.9 miles one-way, almost perfectly flat with only about 200ft of elevation change and is paved half-way. A little after the trail branches off the paved trail, it splits into two trails for about a mile. One completely flat walking trail through the marsh and the other a dirt road with small elevation changes that is appropriate for an easy bike ride. Both trails end up at a small cove on the ocean just north of the golden gate bridge. You can't see the bridge from the beach, but it's still a nice view. Here is a photo looking down the beach:

There are also several side-trails with lots of elevation climbing over the headlands for the more adventurous.

Backpacking the Lost Coast

2009-05-27T10:39:00.000-07:00

This monday was memorial day in the US meaning a 3-day weekend. My friend, Jeremy Shapiro, organized a backpacking trip on California's Lost Coast trail. He ~~tricked~~ convinced me to come along for the trip.

Here is an embedded map of our 3 days showing each day's trail with a different color:

View Lost Coast Trail in a larger map

It's a great trail if you are ever looking for a backpacking trip. It's ~25 miles and completely flat but still rough terrain. Much of the time you are walking directly on the beach sand or hopping across boulders. You have to carry a tide table because parts of the trail are only passable at low tide. And lastly, since there are bears, you are legally required to haul around a heavy/bulky bear canister which prevents bears from eating your food even if they do get it. However, for your efforts, you get to hike along almost completely undisturbed coast for 3 days. The trail has almost no structures, definitely no roads, and is pretty isolated. At many times I felt like our group was the only one around, even though it was a pretty popular time to hike the trail. The best campsite areas were a bit full, but since it was BLM land you could camp anywhere you pleased and even rarer for CA - you could have campfires. As for wildlife, I saw several deer (one with fawn), seals (up close), sea lions, octopus (washed up), tidepool life, pelicans (hundreds), and other fun stuff. No bears sadly.

Here is a photo of our small crew. Click the image to be taken to a facebook photo gallery of the trip.

Strip unused form fields from form submissions

2009-05-09T23:11:00.000-07:00

In my recent post "Why do we even need url shorteners?", I laid out a case for why URLs on the web are actually useful User Interface elements as well as wrote a little bit about how certain bits of history colluded to create needlessly long URLs as the norm. One specific case raised was that of HTML forms. Today I want to show you one way in which to simplify form submissions.

Many forms will contain many different input elements with very few of them actually being used at the same time. Most server-side software is equipped to accept the form even if some of the arguments are missing. It will just assume that the arguments take on the default values. For example, the Google advanced search interface has dozens of input elements. Just entering the query [gregable] and submitting the form will generate the following URL:

http://www.google.com/search?hl=en&as_q=gregable&as_epq=&as_oq=

&as_eq=&num=10&lr=&as_filetype=&ft=i&as_sitesearch=&as_qdr=all

&as_rights=&as_occt=any&cr=&as_nlo=&as_nhi=&safe=images

However, take away all of the default values for the field, and we get this completely equivalent URL:

http://www.google.com/search?as_q=gregable

However, browsers offer no easy way for the creator of an HTML form to craft these more usable URLs.  Through javascript however, we certainly can.  First, the demo.  This demo is a stripped down version of the advanced search interface.  I also added a checkbox and two radio buttons to illustrate some additional input types.  If you fill out any field and hit submit, that set of fields and only that set of fields get submitted.  If you are not running javascript this trick doesn't work, all the fields get submitted, but the form still operates.  Give it a whirl:

Demo advanced search interface

How does it work? You can of course look at the source code. I define one short javascript function: stripFormDefaults. Then where I declare the form, I add onsubmit="return stripFormDefaults(this)". Nothing else fancy is going on.

The W3C tells us that when submitting a form, you only submit the values from the successful form controls. One of the rules for being successful is that the control must not be disabled. So our stripFormDefaults code just disables all of the form controls that we don't want to submit immediately before submitting. It then re-enables them in case the user hits the back button.

In order to know which form controls to submit, we take advantage of a little known set of form element properties that store the default value or state of each form control. Depending on the type of control element, this property is called defaultValue, defaultSelected, or defaultChecked. We simply compare the actual value of each form control with the default value and if they are the same, we disable that control before submitting.

Why do we even need URL Shorteners?

2009-05-06T10:35:00.000-07:00

My first thought was to title this post: "Why are URLs long?" but I realize that the reason I'm writing this was because of the recent issues being raised around URL Shorteners (aka: TinyURL). While this post is over a month late to the party, the context seems relevant.

Problem:
So, why do we even need URL Shorteners? The answer is simple: because URLs are too long. This may be an issue made more obvious with twitter, cell phones, or any kind of manual text-entry, but it isn't only related to this. Essentially, most interesting content on the web has a URL that is too long to remember/type in/share. This can be a problem if you are:

Sending an email to someone who uses a crappy email client that wraps (breaks) lines over some character limit.
Hanging posters in your dorm with a URL to get more information.
Giving a talk at a conference and want the audience to write down/remember some URL later.
Having a verbal conversation with a friend: "I'll send you a link later" is a symptom of this issue.

Worse than just long, most URLs are a crappy User Interface.

Root Causes:
When "moving pictures" (video) first became possible to a large audience, we largely just recorded plays - what we were used to pre-video. Only with time did we learn that the new medium afforded interesting new possibilities: camera angles, shifting scenes, overlaid audio, special effects, etc.

The web evolved similarly. In the original web, most web servers were designed to be a way to access a collection of files on a server some where. We were familiar with file systems and the pre-web internet was a lot of FTP and BBS servers. Our URLs naturally then mirrored file systems. There was certainly nothing that I know of in the HTTP spec that said they had to be. This got us into some trouble:

Extensions
With the file system as a metaphor, URLs got extensions (.html, .php, .asp). Even though the HTTP spec defined a way to communicate the content type outside the URL structure, we were familiar with the extension UI element. However the vast majority of the URLs we interacted with were all one content-type: HTML. Sure, HTML embedded .gif and .js, but users didn't directly interact with those URLs often, they were hidden. What type of software generated the page (.php, .asp, .jsp) wasn't remotely interesting. For the vast majority of URLs we were viewing, the information presented in extension was redundantly obvious or plain irrelevant. Even this post will have a URL that ends with .html, 5 characters of needless redundancy!

Directories/Folders
With the file system as a metaphor, URLs became organized hierarchically into directories. We grouped them by topic, date or whatever with well-defined levels of hierarchy. Each file in one folder. Most early http servers would even automatically generate and serve an "index" page which listed all the files in a particular directory. What was a weak metaphor for a hard drive file system became worse on the internet. Hyperlinks made certain of that. Instead of there being only one path to navigate through a series of directories to a document on the internet, links made sure there were plenty of paths to navigate. Our URLs looked like a tree, but on closer inspection, we had really built a web.

Take this post for example. It's path looks something like:

/2009/05/why-do-we-even-need-url-shorteners.html

However, I sincerely doubt that you navigated to this post by first looking for documents that I created in 2009, followed by those I created in May (month 05). You came through either a hyperlink or a feed reader. The directory structure here is showing information that isn't usually that interesting to a user actually interacting with a URL. How often are book titles based on Dewey Decimal categories?

Search Engines
The file system metaphor can't explain all our woes. After all who in their right mind would ever name a file something so long as why-do-we-even-need-url-shorteners.html? And originally, the web wasn't named this way. Had I chosen it, this page might have a name of url-shorteners.html or long-URLs-rant.html. But then search engines came along. And before long it became known that one of their ranking signals was words contained in the URLs. Users didn't type in URLs anyway, right? They just clicked on them, so it quickly became more important to create URLs for Search Engine Marketing than for Usability: more keywords are always better.

But you can't blame Search Engines. People frequently named their pages with descriptive URLs. Using this as a signal made lots of sense. And once webmasters noticed it and reacted to it, this custom was only further reinforced. As a result we have, why-do-we-even-need-url-shorteners.html(39 characters) instead of url-shorteners.html (19 characters).

Forms
The HTML spec isn't completely blameless either. Since our metaphor was a file system, we never really expected significant amounts of dynamic content. When HTML forms were designed, we imagined things like a way to leave a comment for a webmaster, or a way to upload a file. After all, what other interactions had we really done in the days of FTP or BBS systems?

As content on the internet became more dynamic, forms started to be used more frequently for navigation: search boxes, preference settings pages, javascript drop down elements. All of these things created URLs that were strictly defined by how the HTML spec required GET method forms to interact. For example, when submitting a form, even if only one of the fields is filled in, all of the fields become part of the URL: ?q=foo+bar&page=&sort=&width= Repeated values create repeated keys as well: ?opt=red&opt=blue&opt=green What a waste.

WWW

Historically each hostname (subdomain) generally referred to a different machine. Most machines exposed to the internet were not running HTTP servers. As a result, most uses of hostnames were for things other than a web browser. Since the default was not HTTP, we needed a way to refer to the machine running the HTTP server. A custom arose - the HTTP server would run on the machine named www. It was short, easy to type, memorable, and unique. These days with hardware load balancers, HTTP hostnames rarely refer to individual machines directly. Instead a single hostname can refer to hundreds of separate machines. However www has stuck around because people have come to expect it. The mere presence of a www prefix calls up the concept of a web page in most minds. As you'll notice, gregable.com doesn't have a www and neither do url shorteners - 4 unneeded characters that will be with most URLs for a long time.

Change you can believe in:
Fortunately, this is not a chicken and egg problem. If you run a website or a CMS system, you could write better URLs today without waiting for your customers to do something first. Not all chickens have that much control, but many do. And many websites are already paying attention. Take a close look at how Twitter carefully crafts their URLs to be user interface elements in themselves.

A few of my suggested rules of thumb, but first an important disclaimer. I do work for a search engine company, but the opinions expressed on my blog are my own and not necessarily those of my employer. These recommendations may not be valid in the context of search engine optimization. They are simply my opinions about how URLS could be effectively used as a User Interface Element. With that out of the way, here we go:

Drop the www. But if your users type it, make sure you still get them to the right place.
Drop the extensions (.html, .php) for HTML pages - they are the default. Keep them for non-HTML documents (PDF, images, text) because they are useful hints to a user about what to expect.
Don't let HTML forms dictate your URL structure. They are a necessary evil for actual user-input, but they create awful URL UI experiences.
Use directory structures for things users care about, not uninteresting categorization. Each level you add makes the URL longer and potentially harder to remember/reuse.
Urls should be descriptive. Long numbers are often really bad, a few words are really good.

Finally, think about what is the shortest URL for a given page that would be specific and convey alot of information about what you might expect to find there.

For example, this URL could easily have been as long as:

http://www.gregable.blogspot.com/2009/05/why-do-we-even-need-url-shorteners.html  (80 chars)

Or it could potentially have been as short and descriptive as:
http://gregable.com/long-urls.html (34 chars)
 34 chars isn't bad.  Even a tinyurl would look like http://tinyurl.com/ddvhhc (25 chars).  And consider how much more information is conveyed in the short and descriptive URL for a cost of 9 measly characters.

City of Light; Dystopia Looming

2009-04-29T14:49:00.000-07:00

My sister just posted some new photos of another piece called City of Light; Dystopia Looming. You can see the photos and more information on Flickr by clicking on the image above. It also looks like you can follow her on Twitter now: http://twitter.com/GGrothaus Say hello to her for me.

Big Basin Backpacking

2009-04-27T20:24:00.000-07:00

This last weekend, I went backpacking with Jeremy Shapiro and David Signoff in Big Basin Park (see trail map). There is a trail named "Skyline to the Sea" which starts up in the Santa Cruz Mountains and ends up on the Beach at Highway 1. We didn't follow this trail most of the way, although we pretty much started and ended at the same places. Instead, we took a little longer route that involved alot more elevation change. Kicked the crap out of me, but was alot of fun.

Day 1, our hike started at the Big Basin Park headquarters up in the Santa Cruz Mountains, following Sunset trail for the first half of the day and passing by 3 waterfalls. We sat down and had lunch at the first waterfall, spotting some newts in the water at the base. The trail then hits Waddell Creek and meets up with Skyline to the Sea which we followed from there. Shortly thereafter, it turns into a fire road following the creek all the way to the ocean. We camped at Twin Redwoods camp, about 2 miles before you get to the ocean. After setting up camp, we took some water and finished the trail down to the ocean where there were a few dozen windsurfers having a blast. After getting tired of being blown in the face with sand, we hiked back up to camp for the night, made dinner, popcorn, and a little bourbon. About 11 miles to camp, plus 4 round trip to the ocean for 15 miles today.

Day 2, We woke up a little late, had breakfast and made camp. Retracing our steps the previous day, we hiked north along skyline to the sea until reaching McCrary Ridge trail. McCrary Ridge has a nice sign reading "Trail Recommended for horse use only, very steep climbs.". It wasn't kidding, around a 20% average grade (~1,700 ft gain in 2 miles) and there was a decent bit of flat, so lots of higher grade mixed in for fun. Still, the views from the ridge were very nice. Being probably the only hikers there that day, we spotted some illegal burning over on a nearby ridge and Jeremy called it into the park rangers. At the tail end of McCrary ridge, we took a connector back to Sunset and rode sunset back to the park HQ. A little shorter distance, but a slower uphill pace, we did around 10 miles on Day 2.

All in all, around 25 miles in 2 days. 3 Waterfalls, mountain vistas, and an ocean beach. Not a bad way to spend a weekend. All the photos are up on picasa.

Future Car

2009-04-23T20:32:00.001-07:00

Google was running a number of Earth-Day related events this entire week. Today, they had a talk and demonstration by Aptera, a startup car company hoping to break into the car business in about a year. I brought along my cell camera and was even lucky enough to get a short ride in the car.

Aptera is planning on building and starting to sell an all-electric car late this year, with hybrid and all-gas versions coming out later. The main innovation these guys are bringing to the car space is a car body that is amazingly efficient. They are using composite materials, the same kinds of things Burt Rutan used to build Space Ship One, to build a very strong but very lightweight body. The body is then shaped to absolutely minimize the drag as the car moves through air. It looks something like an advanced aircraft or maybe a spaceship. A common question they get is "Does it fly?".

They managed to get the weight down to 1,500 lbs, which is less than the super-tiny Honda Insight. The drag decreases were even more amazing. Besides just making it aerodynamic, the design has only 3 wheels. Between the low weight and drag, projects are that the car will get 130 miles per gallon, and one of their initial design's they claimed could get as high as 330 miles per gallon - this was not even a hybrid system, just gasoline. All of these numbers I pulled from wikipedia to make sure I had them right.

What now sounds like a flimsy car is far from it. The composite materials are extremely strong. They said they held a party and had an unpainted car body in the shop. They passed around a sledgehammer and put up a $100 prize to anyone who could dent/scratch or otherwise damage the body and they have yet to pay out. They did all sorts of other tests for safety as well, including crush tests, stopping distance, and standard safety gear like airbags. When riding in it, it felt very responsive, cornered well and accelerated/decelerated quickly without feeling unstable in the least. My guess is that the simple design and super-strong materials meant it was unlikely to need much repair beyond oil/tires unless it got into a serious wreck.

The car seats two pretty comfortably, and has enough cargo room "to fit several surf boards". It has built in GPS, bluetooth interface with cell phones that can give detailed data about the car's performance to the phone, rear camera, and airbags. The above photo shows the accelerator and brake pedals, with cute '+' and '-' symbols on them. The car is clearly meant to be fun.

They say the initial electric-only version will retail between $25,000-$40,000. It has a range of 120 miles, and can recharge in 4 hours from a high-voltage connection (like your washer/dryer) or "overnight" from a standard outlet. I asked the founder if the later gas version would be more or less and he thought it probably would be a bit cheaper even. At 100+ MPG, issues of range wouldn't be much to worry about at all.

All in all, pretty neat. Check it out at Aptera Motors.

Systemic

2009-04-23T10:04:00.000-07:00

Some of you may remember my earlier post "If you enjoy art" where my cat was showing off a piece of artwork from my talented sister, Grace Grothaus.

She has made a few more pieces lately and is selling them as she calls it: "Recession-Friendly Art Collecting". Direct from the artist, no gallery markups (which are usually 100%). I'll post a few of these in the next couple days. If you are interested in Grace or her art, check out her website at GraceGrothaus.com.

Like the piece I have at home, this one also makes an interesting use of materials in the art. If you look closely, you can see the use of actual leaves within the art. The piece I have is appropriately called "Urban Organic II" Here is a closeup of the above piece:

The piece is 2 1/2' x 1 1/2' and she is selling it for $550 framed + shipping.

Craigslist Scam

2009-04-20T09:49:00.000-07:00

For those who don't pay attention, rental prices in the san francisco bay area have dropped like a rock. Most complexes aren't lowering rates for renewals with the hope that residents will opt to stay to avoid the hassle. As a result, many renters are considering moving to take advantage of better rates. I'm one such renter.

This weekend, I spent some time on craiglist checking out the opportunities. I saw an ad for a great looking rental house way under market rate - at least 50% below. The ad had photos and an address of a real house (I could verify they matched via street view), but no phone number, so I got a little suspicious of a scam. Mostly because I was curious how the scam worked, I dropped an email to the address in question which was @yahoo. This morning I got a reply:

I did get your response concerning the AD I posted on craigslist. The house is still available but presently I'm not around.. I did bid for a portion of petroleum land sometimes ago in West Africa and fortunately I won the bidding so I have to move quickly down to Africa to have my company set up because I will still have to rebid for it in the next 10 years. I came over here with my wife, we both built the house when we got married.. As soon as we settle down here I had a thought of selling the house so I have to look for an agent, after getting one, we got a deal but later my wife advised against that. She said we may not be able to win the bidding next time, in other to keep our head when we return that we have to keep the house. I reasoned with her and accepted her advise. So I contacted the agent back and requested for my keys and documents. Later we decided to have the house rent out, we would have give the same agent this job also but the truth of the matter is that the agent would want to handle it professionally and the occupant may not be able to reason along with him later. If you notice, you will discovered that the price we are offering is far below standard price, this is enough for you to know that we are not after the rental fee but the absolute care for the property. I know there is no way I can be sure that you are the right person to live in the house because we won't be able to see physical before sending you the keys and the documents to occupy the space. But I just had a feeling that anyone who knows what it takes to put the kind of structure down should know that maintaining a building is mandatory, so if you belief you can take good care of the house and handle it like yours then I will be more than happy to let you rent the house.

Please if you are ready now to occupy the house kindly provide the information below for record purpose

PLEASE TELL US ABOUT YOURSELF
Full Name__________________________________________________ Home Phone ( ) ________________________
Date of Birth_________________________________
Other Phone ( ) ___________________
Current Address_______________________________Apt#________ City__________________ State______ Zip________
Reasons for Leaving____________________________Rent $__________Phone ( ) ____________________________
Are you married____________________________
How many people will be living in the house____________________________
How many people will be living in the house____________________________
Do you have a pet____________________________
Do you have a car____________________________
Occupation____________________________
Move In Date____________________________

P.S: YOU WILL NOT BE ABLE TO VIEW THE INTERIOR PART OF THE HOUSE BECAUSE THE KEYS ARE HERE WITH ME AND THERE IS NOBODY WHO CAN SHOW YOU THE HOUSE

He also provided an international phone number - with country code (you guessed it) Nigeria, My guess is that with this information, my friend here can open a line of credit in my name. If not, I'm sure the next email will require my social security number for a credit check. It's so sad.

Australia Trip

2009-04-15T22:21:00.000-07:00

Cristin and I returned last weekend from our trip to Australia. I went for SMX Sydney initially, but Cristin and I took a side trip to the beautiful Whitsunday Islands after the conference. I'll bore most people with too many details, but I will say a few words about the trip.

The conference itself was excellent, thanks to the organizer, Barry Smyth. More relaxed and informal than most. A bit smaller group than the US counterpart, but anecdotally I felt like the average level of expertise amongst the participants was a good bit higher than many of the other conferences. It may have just been the folks I talked to or the sessions I hung out in, who knows. The after conference was also a blast, especially the "barbecue under the bridge" which turned into a dinner cruise in Sydney Harbor. My favorite talk was actually on Day 2, "Twitter Tips & Etiquette" by Darren Rowse of Problogger.net. There were tons of other great talks and speakers, but twitter and social networking was probably the topic I personally was least familiar with.

Speaking of twitter, I got my first drinking from the firehose experience at SMX West as well. Half the attendees were live-twitting (is that the right word) the event or watching the stream. I got a real kick out of seeing what folks were writing about the speaker in real time. By the way, if for some reason you want to follow me on Twitter in addition to gregable.com, my name over there is also gregable. Gregable was taken, and Gregacious sounds like loquacious which seems appropriate for twitter.

After the trip, Cristin and I headed to Hamilton Island which is part of the Whitsunday islands. Using the island as a starting point, we took a few side trips:

A dive boat to the Great Barrier reef for a few reef dives - spectacular coral, I wish I had some photos.
A trip to the sparkling shores of Whitehaven beach on Whitsunday Island itself where Cristin polished her ring using the nearly pure silica sand.
A day on the mainland boating up the proserpine river, on the lookout for Crocodiles lounging on the banks and hunting mud crabs.

In addition, we did alot of lounging/snorkeling in the pools and beaches of Hamilton Island proper, a resort island with alot of amenities. Cristin and I got to pet some Koalas, Kangaroos, Wallabees, Dingoes, and a Wombat, as well as hold a few rainbow lorikeets. A photo of Cristin posing holding a 15-month-old Koala is above (Isn't that photo awesome?), a few more photos are available on picasaweb. All and all, a great time by both of us.

Earth Hour at the Sydney Opera House

2009-03-30T15:59:00.000-07:00

I'm heading to Sydney today for SMX (see my last post for context). This past Saturday night ago was Earth Hour, an event which I understand started in Sydney where people around the world are voluntarily encouraged to turn off lights for one hour in recognition of climate change. It is of course only symbolic, not a real fix, which bothers some folks but it is OK with me.

Anyway, the Boston Big Picture, one of my favorite blogs which I've talked about before, had a post showing before and after photos of various landmarks during the earth-hour blackout. It is quite stunning - click on each photo to see it fade in and out of darkness. Highly recommended for taking a look at. Above are a few of their photos from the Sydney Opera house which I will be seeing for the first time in person very soon.

Heading to Australia for SMX Sydney Events

2009-03-26T20:30:00.000-07:00

This time next week, I'll be in Sydney Australia to attend SMX Sydney 2009 (Search Marketing Expo). I've never been to Australia before, but it looks like a great conference and a fun city, so I'm really looking forward to the trip.

For the agenda: On Thursday, I'll be talking about the mysteries of URLs, domains, and dreaded "duplicate content" issues in the Links and URLs track. I'll be discussing how to deal with multiple domains, subdomains, URLs and sitemaps. I'll give a quick review of how the new rel=canonical tag works and how we are seeing it get used.

Friday, I'll be doing a lunch Q&A panel, Meet the Search Engines, as well as my personal favorite, a site clinic. The sites we will be discussing are a secret even to me, so think of it as a improv act.

If you are going to be in Sydney for the conference, make sure you find me and introduce yourself. My ugly mug shot is up on the website. Tell me you heard about it on Gregable first! I'll be at all of the social events in the evening as well, so bring all your burning Google questions. I expect that this will be a fun event, and I look forward to meeting some of you.

rel=canonical

2009-02-13T13:45:00.001-08:00

I'm glad to see that yesterday all three major US search engines jointly announced support for the rel=canonical tag:

http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
http://ysearchblog.com/2009/02/12/fighting-duplication-adding-more-arrows-to-your-quiver/
http://blogs.msdn.com/webmaster/archive/2009/02/12/partnering-to-help-solve-duplicate-content-issues.aspx

I'm more excited than most people, because I'm largely responsible for making this idea a reality at Google. To be fair, many people have had similar ideas in the past, and the effort was certainly a joint one between Google, Yahoo, and Microsoft. I just did my small part.

I was speaking at SES San Jose in 2007 and listening to the frustrations of webmasters when dealing with canonicalization issues. In the past, our suggestion was to always do a 301 redirect to the canonical of a page if Google wasn't getting things straight. Usually we do "the right thing" automatically, but we aren't perfect. 301 was the answer when you want to be sure (and it is still a valid answer).

However, there were some scenarios that people pointed out to me for which 301 was not ideal:

301 simply won't work for print-friendly versions of a page.
The same story is true for any page where the content is constant, but the UI elements on the page change relative to the URL, for example having a sort field. Sure you could use cookies or other mechanisms to track sort fields, but that breaks bookmarking or sharing the URLs with other people, and it breaks the entire experience if the user has disabled cookies.
Many webmasters don't have control over the server headers, for example freehosts like blogspot, or webmasters that aren't terribly tech-savvy.
Sometimes systems or a client requires the use of session ids in URLs to track users throughout a site.
Needing to 301 redirect a user doubles the load time of a web page because now their browser makes two round trips to your server to get the content. Faster is my favorite feature.

Here is a specific example (URLs are fake, no need to click on them):

http://stuff.com/breadcrumbs/tents/bags/red/tent_bag.html
http://stuff.com/breadcrumbs/bags/tents/red/tent_bag.html
http://stuff.com/breadcrumbs/bags/tents/red/tent_bag.html?view=print

All of these pages have similar (although not exact duplicate) content as they are generated by a mod_rewrite and a database script. Search engines might automatically group them in some cases, and might mess up in others. Any of them (except for the printer version) can be the canonical and the site owner probably doesn't care - just that only 1 of them is. The owner wants all URLS to exist (aka, doesn't want to use 301s) because they reflect the user's browsing path to get to the red tent bag product and hence the URLs themselves are good user experiences. All pagerank/links to any of these pages should flow to whatever canonical chosen. There was no good solution to a problem like this. You have to sacrifice something: usability or search engine optimization.

The new rel=canonical suggestion is simply to add to all 3 of these pages one single tag:

<link rel="canonical" href="http://stuff.com/breadcrumbs/bags/tents/red/tent_bag.html">

In this case, I arbitrarily sorted the breadcrumbs. This could easily be done server side without having to manually pick which path I want to be the canonical. I could sort by something other than alphabetical too if there was a reason I wanted one URL in the index instead of another.

I wouldn't expect most webmasters manually working on their site to use this often, but for software (CMS) systems, it would give an easy way to avoid using 301s or robots for canonical issues. In many cases where webmasters use a package CMS software, the webmasters need not even be aware of the canonical tag.

I've seen some discussion on the web indicating that this is just a hacky way to solve problems that are simply a reflection of bad website design, or people who are concerned that they need to implement yet one more thing for all of their web sites. I think people are missing the cases where the previous options really weren't ideal. Also, this is not required maintenance - all search engines rightly assume that the vast majority of the web won't mess with this stuff and as a result they all do their best to crawl and index the "right way" regardless.

There are definitely other great tools out there for solving these problems: cookie-based sessions, 301s, robots.txt. Yahoo even launched some time ago a feature in Site Explorer called dynamic url rewriting. I think all of these tools are great. One of the best parts about rel=canonical though is that any search engine out there can support it easily without webmasters having to change a thing. Set up rel=canonical tags for yahoo, and google/microsoft/etc can start indexing your content more effectively without any additional work on your part as a webmaster.