Tim Trueman

This one goes to 11

2011-09-05T00:00:00-07:00

Modena, Italia

I always knew Pagani cars were very exclusive but wasn't until today that I realized how truly rare they are…

Our tour guide Francesco turned our van down a street with a black sign labeled "Pagani Automobili S.p.A." in the Italian countryside just southeast of Modena in the Emilia-Romagna region. I was looking for a smaller factory than Ferrari's massive Maranello facility but I saw nothing, even when we stopped at our destination. I then noticed the Pagani logo next to an open window on a humble building.

Francesco walked to a locked gate as a woman exited and pressed a button next to the hinges. He waited for the buzzing sound by bracing himself against the black metal bars of the gate in the fence; bushes obscured the view of the building. Moments later we entered as he whispered to us that that woman who passed us was the wife of Mr. Pagani. As we approached the door of the building we could see three cars so expensive you'd have to sell not one but several homes just to buy one of them—that is, if you could somehow manage to get yourself invited to buy one of the 18 super-cars they pour their souls into each year.

We're sardined into a room barely big enough for the cars and our group, but we don't mind. There's a Zonda F in the center, flanked on the right by a Zonda Tricolore with narrow racing stripes in the colors of the Italian flag, draped over carbon fiber with a thin coat of lacquer that's got a blue tint to it to match the anodized blue aluminum bolts, hoses and exhausts tucked behind the gills of the front and back of the body. In case you're wondering how much one is, the Zonda Tricolore is a production run of three and will sell for €1.3M before taxes (and it's leaving soon to be delivered to it's owner). Zondas fall between unobtanium and unicorns on the rarity scale.

On the left is a Zonda R, the current record holder of the Nürburgring Nordschleife, an old, famous and brutal racetrack with dozens of corners over its 20.8 km (12.9 mi) length. It beat Ferrari's best lap, the previous record, by 11 seconds—an eternity in racing—with an unfathomable 6:47 run.

Francesco pulls us away from feasting our eyes upon the details of these masterpieces; the clearly visible, naked weave of the carbon fiber makes stripes that line up like an expensive Armani suit. We put away our cameras because we're about to get a tour of the factory itself. It feels like being let in on an exclusive family secret recipe, in part because that's how it is. You see, Francesco used to work here, in fact he's worked at Ferrari, Lamborghini and Ducati as well. He seems to know everyone by name and smiles gently as he greets everyone and jests with them in Italian. There's a yellow line we have to stand behind but Francesco walks right up to everything and pulls items off shelves, out of drawers and from underneath covers to show us how Pagani builds only three cars at a time at its three assembly stations. It takes 7-9 months for each one to be completed. This is exclusivity like you've never seen before and it's immediately clear how special this experience is.

The three cars currently under construction are a Zonda R, the very last Zonda F ever to be produced, and the first of the brand new Huayra. Francesco's insight into the history of Pagani engineering is able to paint a clear progression from Zonda F to the Zonda R and into the undelivered Huayra. The finest details of the chassis and suspension illustrate the subtle evolution of from the final Zonda F to one of the world's most expensive laboratories, the Zonda R, into the Huayra.

Everything is handmade; when Francesco opens the engine cover the signature of the one guy who built the engine's name is inscribed on a metal plaque. It's difficult to convey how impressive the details are, especially the unseen ones such as the active aerodynamics on the Huayra: software and computers send electrical impulses to actuators that constantly move flaps on the body to slow the car during braking, or to generate more downforce when you fling the car through a bend in the road going faster than the Polizia would prefer.

Francesco raps his knuckles hard on the monocoque of the carbon fiber chassis he used to help build. Strewn about the shop are hundreds of parts all painstakingly made from carbon fiber with perfectly aligned weaves. A woman beside us is sealing the vacuum bag around a single carbon fiber part. Two giant autoclaves dominate the side of the another room; they'll cook the parts at 130 C at 80-85 psi for several hours.

There's molds with finished parts being coaxed out gently with rubber mallets. There's parts of chassises lying around collecting dust. This isn't a spotless white factory; it's a gritty and ordinary mom and pop shop that's turning out works of art that also happen to be engineering triumphs in their own right. The dozen or so workers visible are moving lethargically but with purpose and deliberate skill in the stifling heat coming from an open garage door to the alley. Every last part has a precision and level of detail to it that would give Michelangelo's David a run for his money. The welds look like they don't even exist, a carbon nose to a future car shows graceful curves that suck the car onto the road for maximum traction, and a worker stares unblinkingly at the edge of a piece of bodywork as he sands it to the millimeter of perfection.

We're whisked out as Mr. Pagani escorts a customer about to take delivery around the factory floor—not too hard to do with only 18 customers a year. Before we leave we're given a couple gifts and invited to sign and leave Mr. Pagani a note in his guestbook. This visit was very special to me and I felt honored to be granted such incredible access to this one-of-kind birthplace of power and beauty.

"Today" refers to September 5th 2011. I didn't want to forget the details so I wrote this up on my iPhone that day since I didn't have a computer while I was in Italy, so cut me some slack. All photos were taken with a Nikon D7000 DSLR camera with an AF-S NIKKOR 50mm ƒ/1.4G lens. Thanks to Chelsea Henry for proof-reading this post.

Memory #458

2011-09-05T00:00:00-07:00

Maranello, Italia

As we approached the Ferrari's one and only factory, we were greeted with an opening salvo of beauty and passion—a black Ferrari 458 Italia dashed in front of us before bolting out of sight. Giant yellow letters joined together to spell "Ferrari" in its well-recognized font, Ferro Rosso.

Our van skirted along the outside perimeter, passing by the security gate that was letting a factory worker who must have overslept through. The iconic wind tunnel loomed large to our right as we continued by; even the factories and gritty industrial structures have little details and architectural elements that make them more elegant and classy—in this case it was the wind tunnel that was designed by one of Italy's most famous architects, Renzo Piano, in 1997.

As we finished our drive-by tour of Ferrari (nobody is allowed tours inside except owners, sponsors of Scuderia Ferrari, their Formula 1 racing team and esteemed journalists), we pulled up to a shop with a half dozen Ferraris and a Lamborghini parked outside. These were the cars we were about to play a symphony for the Italian countryside in as we tore through Maranello with the V8 of a 458 Italia, the very latest Ferrari.

With an overcast sky I slung myself down almost to the ground as I slid into the hand-stitched black leather and Alcantara seat. I leaned back onto the prancing horse sewn into the headrest as I inserted the key. I placed my hands around the steering wheel and found they rested naturally in a spot that fit like a glove. It was the calm before the storm.

With fear and excitement I placed a finger on the red "start" button on the bottom left of the steering wheel and applied a bit of pressure. A guttural bellow roared through my spine and into my ears. Nothing could stop the grin that instantly told bystanders watching what I thought so far. A quick inspection of the paddles and a flick of the reverse button in the center console marked the start of an extremely cautious start of the journey in a super-car that costs a quarter of a million.

I pulled the right paddle behind the wheel back with a squeeze. Everything in this work of art gives the most pleasing and fine-tuned tactile feedback. The LED gear indicator in the middle of the largest instrument, a centered red tachometer with gold numbers and lines showed "1" in an angular inset screen on the bottom right of the dial that had room for nothing else. Carefully, expecting a lion to jump out and snap my head off, I pulled away.

I had been preparing for a brutal and unforgiving clutch that would bite and jerk the car around but that never came. Gracefully and with the finesse of a ballet dancer the Italia edged forward. Together we pulled out out the street. I pressed the button with a left arrow in the center of the steering wheel to signal my turn away from the direction of the birthplace of the sleek composite machine containing a heart beating with eight cylinders.

The easy, gentle ride begged to go faster so I obliged my eager stallion with some pressure from my right foot. I was instantly pushed back more securely into the comfortable seat and the symphony that erupted floated through the open windows as the wind blowing in picked up. My heart somehow managed to thump faster.

Dark splotches on the road told the cautionary tale of rain from earlier in the morning and I let off the aluminum petal as I approached the back of a car before turning onto the highway. The car rolled to a slow stop as we waited for a break in the traffic to pull onto the highway. Pausing only to switch the dual-clutch gearbox to manual, the 458 kicked me back into the support of the seat as we set off onto the highway; I was pinned to it with the force of hundreds of horses.

The exhaust trumpeted the carefully tuned notes of our getaway song. The song started out with the smooth orchestration of many parts working as one and crescendoed until I pulled the paddle on the right and the LED flicked into a "2" and then a "3" and on up to "5". I still had two gears left. The car that had just flown by us had the favor returned only seconds later. And the grin was now permanently etched onto my face. I wanted this bonding of steed and rider to go on forever. I felt like I could drive for days without stopping. But minutes later I was pointing the nose of the beast back into its Maranello stable.

It's hard to describe the emotions a ride like this can surface but it was pure, unadulterated joy so raw and powerful that you can't help but fight back the welling up of happy tears.

Later (and even now truthfully) I realized I lacked the vocabulary to describe the experience while attempting to discuss it over cappuccinos with the others who had shared the experience with me. We all knew how special today had been even if we were fumbling over hyperbole attempting to try and describe it.

I highly recommend watching all of this video if you want to appreciate how awesome the Ferrari 458 Italia is:

Caching counts for Django's built-in pagination

2010-12-28T00:00:00-08:00

If you've ever used the Django debug toolbar on a Django app with a large database, chances are you've seen that counts can be quite slow and are generally something to avoid. You'll probably either want to cache or guesstimate them. You may have even written some code yourself. If you've ever gotten stuck on Django's object_list still triggering .count(), then I have a solution for you. The trick is you can monkey-patch your queryset's .count() method but it will still trigger a database call. Why? object_list clones your queryset and that resets its .count() method. You'll need to monkey-patch the ._clone() method to monkey-patch .count() after it clones the queryset. Then things will magically work and you want get the FBI or CIA knocking on your door asking why you were tampering with querysets unsuccessfully.

Here's the code:

class CachedCountCloneProxy(object):
    ''' This allows us to monkey-patch count() on QuerySets so we can cache it and speed things up.
        ._clone is called so we have to monkey-patch that first...
    '''

    def __init__(self, queryset):
        self._queryset = queryset
        self._queryset._clone_original = self._queryset._clone

    def __call__(self):
        queryset = self._queryset._clone_original()
        queryset.count = CachedCountProxy(queryset)
        return queryset

class CachedCountProxy(object):
    ''' This allows us to monkey-patch count() on QuerySets so we can cache it and speed things up.
    '''

    def __init__(self, queryset):
        self._queryset = queryset
        self._queryset._original_count = self._queryset.count
        self._sql = self._queryset.query.get_compiler(self._queryset.db).as_sql()
        self._sql = self._sql[0] % self._sql[1]

    def __call__(self):
        ''' 1. Check cache
            2. Return cache if it's set
            3. If it's not set, call super and get the count
            4. Cache that for X minutes
        '''
        key = "paginator_count_%s" % hashlib.sha224(self._sql).hexdigest()
        count = cache.get(key)
        if count is None:
            count = self._queryset._original_count()
            cache.set(key, count, 300)
        return count

# To use:
# queryset._clone = CachedCountCloneProxy(queryset) # monkeypatch solution to cache the count for performance

View as Gist on Github

One thing to note is this is a generic solution that will cache based on the SQL that is generated. There's probably a better way but anything is faster than .count() on a large database table.

Yes, I'm aware this is a hack. The best way to fix this would probably be patching Django or something. I didn't bother, so sue me.

The tale of a family of inertial navigators

2010-07-20T00:00:00-07:00

"T-minus 31 seconds and we're go for auto-sequence start…"

These words have little meaning to those of us not preoccupied with sending a 2000 ton spacecraft 185 miles up into the expanse of blue that only 49 years ago was science fiction—but for the curious, those words mean everything. With 31 seconds remaining, the seven humans surrender control to the machines. They place their lives into the hands of a family of five computers. The four identical quintuplets and one odd sibling seize control of the critical functions of a spacecraft strapped to millions of pounds of fuel. They choreograph what I would argue is one of the most beautiful concerts of coordination; 250 times a second they cast their votes on what actions to take. If one of the machines were to malfunction its watchful siblings would outvote it and continue the mission without hesitation.

At T-minus 6.6 seconds the computers throttle up each of the engines one-by-one, 120 milliseconds apart. All five computers nod in agreement when it’s confirmed: engines and systems look good. Moments later launch arms are retracted as the juggernaut accelerates away from the pad.

Upon clearing the tower the computers command a careful, calculated and complex maneuver rotating the vehicle in all three axis: pitch, roll and yaw. It rolls gently onto its back to gracefully and delicately settle into a steady ascent headed downrange. It races towards an orbital path that will cover nearly five miles per second.

Throughout the violent acceleration towards the stars the machines maintain a perfect, constant and anchored orientation like a monk poised in a deep trance. One minute into the flight the vehicle reaches max-Q. It's when the throttle has to be eased back to 65% in order to survive the aerodynamic forces that mercilessly assault the structure protecting the fragile bodies of its passengers. The thunderous, deafening roar of the wind rushes over the aircraft at nearly 1000 MPH. Those computers are what keep the passengers safe as they compute the orientation of the spacecraft and make hundreds of tiny adjustments every second that to the casual observer on the ground over seven miles below them seems rock solid and sure.

Even today this feat is magic to all but a handful. And yet this technology is on the verge of celebrating its 30th birthday. Perhaps the time to pry open the mystery box.

↓

When a human is gripping the joystick of an aircraft with expert hands they are relying on external references to maintain straight and level flight. They gaze to the horizon subconsciously. The AP-101S computers in the Space Shuttle on the other hand know down to a degree of accuracy exactly how the spacecraft is oriented. The magic behind this knowledge is the inertial navigation system.

Inertial navigation is like putting on a blindfold and then navigating simply by feeling which direction you've turn and guessing how far you've walked. Instead of feeling and guessing inertial navigation systems (INS) use a set of sensors: gyros and accelerometers. Gyros sense how fast you're turning in a direction and accelerometers let you feel how fast you're beginning to move forward.

How exactly do gyros sense how fast you're turning? These critical sensors—which sound like something straight out of Tony Stark's lab—are called ring laser gyroscopes. They use the Sagnac effect to detect with incredible accuracy the rate at which the gyroscope is rotating.

OK, so how does the degrees per second at which you're turning help give a computer exactly what angle the vehicle is pointed? A wee bit of calculus, some special software and an exhausted yet excited engineer combine forces to produce the result. What it boils down to is integration. All you have to do is add up the degrees per second. If you were to roll to the right at a constant 15 degrees per second, you would know how many degrees of roll you had accumulated at any given time. At half a second you'd be at +7.5 degrees and at 3 seconds you'd be at +45 degrees. That's essentially what inertial navigation is doing, except much faster: as much as several hundred times every second. Instead of checking every second for the turning rate the sensor is checked hundreds of times per second. The accuracy of the sensor combined with the high update rate allow for a precision that a human could never match.

Of course I'm grossly oversimplifying inertial navigation. There's lots of issues to take into consideration: centripetal force, temperature, even the Coriolis effect. The accelerometers and lots of algorithms to counter-act these issues are critical to inertial navigation.

Inertial navigation is badass, right? Don't you just want to toy around with it yourself? Oh right. Ring laser gyros: the only thing less likely to fall into your hands other than Scarlett Johansson. Fortunately there's a technology rapidly making projects only NASA could take on possible for you and me. It's called MEMS—the much-easier-to-remember-and-pronounce shorthand for microelectromechanical systems. They are tiny. You may never have heard of MEMS but they are everywhere now. Every iPhone uses MEMS sensors and the shiny new iPhone 4 even includes a 3-axis MEMS gyro. It's got enough sensors and processing power to do inertial navigation. Which means…you can afford the gyros and other equipment to build your own INS. Let's do this!

↓

Over many months Ryan Beall (he is the shit) and I worked on a tiny piece of software. It's an open-source (under the MIT license) piece of software written in Python that does inertial navigation based on a BYU paper. I call it Shumai—a reference to a tribe of people who are expert navigators by using their advanced knowledge of the stars in an obscure post-apocalyptic science fiction book series I once read (I'm not nerdy, I'm not nerdy…). Also, even though gyros are cheap, you won't even need them to get started. Shumai works out of the box with the suite of virtual sensors in the flight simulator X-Plane.

At first glance Shumai may seem complicated but its purpose isn't so hard to understand. It collects readings from the gyros, accelerometers, static and differential pressure sensors (altitude and airspeed respectively), GPS and magnetometers (not directly included but emulated by an extra algorithm included in Shumai). After gathering these values it simply produces estimates for pitch, roll, yaw and the position (as well as wind direction and velocity).

To get started all you need to do is head over to Github to get the source code: http://github.com/dronedynamics/shumai. Be sure to follow the step-by-step instructions here: http://dronedynamics.com/shumai/. My ambitions for Shumai is to build the open-source inertial navigation software project that is the most comprehensive and most highly accurate. I would love to help anyone get involved if they'd like to start contributing to the project. Feel free to email if that is the case: tim@dronedynamics.com.

Understanding solid-state drives

2008-12-07T00:00:00-08:00

It seems I've forgotten about writing for a few months. I miss writing to be honest and I think it's time to end the hiatus by writing a little something about a package I received a few weeks ago.

It seems the Intel X25-M solid-state drive has been turning a lot of heads. I'm not sure very many people really understand the implications of both Intel's advances and solid-state technology in general. They look at the benchmarks and the two orders of magnitude lower seek time. They smile and nod and that's it…end of story. Not so fast.

SLC versus MLC Solid-state drives are usually classified into two categories: Those made from single-level cells (SLC) and those made from multi-level cells (MLC). The difference is SLC stores 1-bit per transistor and MLC stores 2-bits per transistor. The implications are realized in performance, reliability and cost.

Performance-wise MLC typically takes twice as long to read, the same amount of time to erase, and usually more than three times longer to write than SLC. Keep in mind that athough MLC is significantly slower to write to than SLC it's still faster than a rotational disk.

SLC's biggest advantage is lifespan not performance: SLC cells can be erased and reprogrammed 100,000 times versus 10,000 for MLC. The reason this difference is important is write amplification. Write amplification is the actual size of the write versus the requested write size; This occurs when saving a file that is smaller than the page/block size on an SSD. SSDs have traditionally had a write amplification of 20-40x.

Price-wise it's much more expensive to build SLC SSDs over MLC because you need twice the number of transistors to store the same amount of data. If you thought MLC was out of your budget SLC is way out of your league.

AnandTech does a great job of explaining the technical details of how NAND flash memory works if you're even more curious.

What makes the Intel X-25M better than other SSDs?

1. Reduce write amplication The Intel drive has reduced the write amplification to 1.1 or less which is hugely impressive. This contributes to massively improving the lifespan of the device.

2. Build a better controller Intel's excellent implementation of a controller makes a pretty big difference. Other SSDs have done a really weak job in this aspect. Intel nails it with their controller. It improves caching, queuing and running concurrent operations, improves the lifespan and provides additional information specific to SSDs.

3. Better wear-leveling The Intel controller takes advantage of the fact that read performance is the same anywhere on the disk, so deleting a file, and redownloading it will write it to another part of the SSD; This is called wear leveling. Intel's wear leveling has a difference of 1.1 or less between maximum wear and average wear versus the traditional difference of ~3x. This also helps improve drastically the lifespan of the device.

4. Revolutionary improvements to reliability Intel promises you can write 100GB per day for 5 years before the drive fails but it should last much longer than that. Intel does that by keeping 7.5-8% (6-6.4GB on the 80GB drive) in reserve so that as blocks fail it can switch the reserve blocks. The drive supports two additional SMART attributes, one that says how close to the rated cycling limit you are, and one that says when you've run out of (or are about to run out) reserve blocks. This means you'll know exactly when your drive is about to fail. That's frickin' awesome!

Results These are my Xbench disk results before and after on my unibody MacBook 2GHz with 4GB of RAM. Real-world increase of 5.1x in disk performance. Everything just feels…instant. Pinwheels are an extremely rare occurrence.

I can't even imagine the scary performance Intel can wring out of their upcoming SLC drive. It's going to be a monster, and a game changer quite possibly (in server environments).

What Intel has done is raise the bar for everyone by showing them what's really possible with solid-state drives. Honestly after weeks of use and comparing this to my daily use of a faster MacBook Pro without an SSD at work, I can say it's the single best upgrade you can do to any computer to make your computing life better…if you can afford one. It's a night and day difference.

The best part is the drive's already dropped over $100 since I bought it; It's $539 $399 on Amazon.

AT&T Uverse: buyer's guide

2008-12-07T00:00:00-08:00

So you're thinking about getting AT&T Uverse. You thought you'd look up what people thought about it and you ended up here. I'll break it down as simply as I can.

Pros

Internet occasionally mostly works for a few hours albeit at DSL speeds (~5Mbps)

Cons

More expensive than competitors
Slower than competitors (although on the plus side, it's slightly faster than DSL that costs less than 20% of what Uverse costs)
More than 50% of the time something isn't working (DNS issues or just down totally)
DNS issues are frequent--usually at least 30% DNS lookups time out (Akamai domains frequently failed to resolve so major websites would have no CSS, JavaScript or images)
Terrible device that gives you no control and doesn't offer the latest tech (802.11n)
Awful tech support (Uverse technicians are worse than regular AT&T technicians--intelligible as well as totally incompetent, e.g. they seemed to think I change settings on my Mac by clicking "start" and then "control panel")
They charged me roughly a $1 for a few months after I cancelled the service for who knows what reason
Charged me $487.14 (I didn't chose to pay this, they charged me without asking--after I cancelled autopay at least a couple months before) for not returning the equipment which apparently I can't get an address to ship it to and they refuse to send me the boxes they promise even after "verifying" my address twice
Their website is totally unusable (maybe it works in IE 6, but I wasn't able to try it since I was already on IE 7) (I should have read the sign and never used a service with a shitty website but then again what option did I have)
There apparently aren't any managers on the Uverse program that know how to return phone calls (and all the managers are always busy so you can't talk to them directly)
Uverse's ticketing system seems to automatically close any ticket I request they open without any action whatsoever being done
If you ever need help, you'll dial the "Uverse" number that magically connects you to the regular AT&T technicians in Texas (who don't know anything about Uverse and transfer you back to the same line--this process happens over and over until you've wasted about 30 minutes on hold and going through the same menus over and over and somehow it magically breaks out of the loop)

Seriously AT&T, $487.14? What the FUCK?! I've got your expensive hardware sitting by my door in hopes you'll refund me for your criminally awful "service" (I'd normally expect interest as well as a refund for the percentage of time your service wasn't working but you've buried the bar so low it doesn't matter at this point).

Basically if you're considering AT&T Uverse...don't do it. If it's your only option...I'm not kidding even a bit...MOVE somewhere else. Am I bitter? Yes. I admit it. But I'm trying to save others the agony.

I'm now with Comcast. I pay 30% less now and I get ~30Mbps downstream and 3-4Mbps up. It's been down twice in the last six months both times for about 30 minutes, which isn't too unreasonable. And any future place I consider living in I'm definitely checking to make sure something other than Uverse is available.

Sorry for the negative post, I just wanted to put this out there so others don't make the same mistake as me.

Character encoding confusion

2008-09-15T00:00:00-07:00

As you can see above my photography skills have gone up, although this has been at a negative correlation to my programming skills. And that's what this is about. My inexperience with the clusterfuck that is character encoding (we haven't come a long way sadly since ASCII was invented in 1963).

Alright so there I was at Open Hack 2008. I work for Yahoo! so I wasn't allowed to compete, but I just figured I'd keep everyone company and work on a personal project. It's a simple service I will be releasing shortly (I hope). I copied the code I had already over to my laptop using my Linux desktop's http server.

I started coding and after a few minutes I went to run my project. I was met with a friendly unexpected T_CONSTANT_ENCAPSED_STRING. In the dozen lines of code I had, I couldn't see the issue. I figured I was just missing something stupid so I posted to Twitter asking for help. I got a quick reply and headed downstairs to take a look at the issue. We tried to isolate the issue on the line throwing the error:

$ php -r "$subject = $argv[1];" Parse error: syntax error, unexpected ‘=’ on line 1

What the hell? I'll give you a million dollars* if you can spot the issue (ignore $argv[1] being undefined, that's not it). About an hour and a dozen engineers later (including Rasmus) we solved the issue. Here's what the code really looked like after I opened it in TextEdit.

$ php -r "$subject¬=¬$argv[1];" Parse error: syntax error, unexpected ‘=’ on line 1

Note the negation symbol before and after the equals sign. What amazes me is that none of the editors I had used before (TextMate, vim, nano) had caught the invisible gremlins.

Here's what I think happened (I'm still not 100% sure). I wrote the original code in gedit which I verified saved in the default of UTF-8 (as it should). I then transfered the code to my laptop via an http server on my desktop. Since I didn't set a charset Safari chose one for me: Latin-1 or its more catchy name, ISO-8859-1. Above the first 127 character values most charsets differ and this conversion produced gremlins that I could see when I pasted my code into a new document and saved it as UTF-8. Anyone who's had to work with double-byte languages such as Chinese are already aware of this. They've probable have seen more mojibake then they've wanted to.

The core problem

Back in the day I read Joel Spolsky's "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)". I was young and naive; I promptly brushed it off as if it would never apply to me.

My mistake…

The root of the problem is this: most web developers are unaware of the character encoding issues. So they forget to specify one. So it's up to the software you're running to pick a character encoding for them.

For email, most clients support UTF-8 finally and if they don't they should be taken out back and shot. Then stabbed with a spork. If I don't see this in the header I'm sure somewhere in the world a tiny, cute, fuzzy kitten dies:

Content-Type: text/plain; charset="UTF-8"

On the web, you should always specify your charset, like this:

Use UTF-8, if you want to save cute kittens. I'm sure a few years ago most documents that didn't specify a charset were Latin-1 but today I hope that's not the case. Here's how to change the default charset in your browser:

(Mac) Safari Preferences > Appearance > Default Encoding Western (ISO Latin 1) => Unicode (UTF-8)

(Windows/Linux) Firefox Edit > Preferences > Content > Fonts & Colors > Advanced > Default Character Encoding Western (ISO-8859-1) => Unicode (UTF-8)

I still can't believe how much time charsets can cause you to lose. There should be some sort of awareness campaign with charity runs and those plastic yellow bracelets. At least I had a great time at Hack Day and I learned a ton of awesome command line and vim tricks (thanks James).

*Did I say a million dollars? I meant grains of sand.

Cupertino, start your copiers

2008-04-06T00:00:00-07:00

[Apparently this wasn't clear so be warned: This is what I would do if I was armchair CEO of Microsoft. I know they're never going to do this. I'm just trying to imagine how Microsoft could be positioned for the future. I know they're wildly successful and I'm not suggesting they should completely give up strategies that are making them boatloads of cash already.]

As I said at the end of this article back in February, I think Microsoft could try something crazy that just might work; Windows 7 could make Microsoft the first company to actually make use of the Internet in an operating system. Steve Ballmer gets that Microsoft needs a big change I think, but I'm not sure he really knows how to do it. In fact I'm pretty sure he doesn't know what to do. Yahoo! isn't the answer to your prayers Steve, trust me. Here's a freebie for you…

I've come up with the 12 Steps to Windows 7 of Awesomeness™!

1. Ditch the NT kernel; use Linux. Just because Linux wasn't invented at Microsoft doesn't mean it's not a total waste of resources to maintain Windows. I'm not saying I think it's crap; I think very highly of it, but you're not getting any return on investment for these resources. Why spend tons of money here when you don't have to? Just do it. And contribute the awesome things you used in NT back to the Linux kernel.

2. Ditch the Windows name. Props to ExtremeTech for this idea. The name just leaves a bad taste in my mouth, and, even if it's not true, Windows now stands for insecurity, pain, and suffering to everyone. Leave those memories behind. Don't pick something shitty and misty like Vista. You want to signal customers you're doing something new, exciting, and thinking totally different.

3. Build a new type of browser that blurs the distinction between desktop and web apps. There's a lot of reasons web apps cannot compete. Arguably the biggest difference is caused by Fitts' law. Fitts' law is a mathematical formula for the amount of time it will take to hit a target, depending on the size and distance of that target. Some places on the screen are easier to get to than others. As Bruce "Tog" Tognazzini explains, the easiest is directly under the pointer, because you don't have to move anything. The next easiest location is the corner, because it has essentially infinite height and width, making it a really easy target to hit. The edges only have one infinite direction but they are also easy to reach. The thing about web apps is they cannot use the corners or the edges of the screen. Desktop apps can.

Here's what I'm suggesting: build a browser that lets web apps have the same functionality as desktop apps. It will need to let the web app take complete control over the browser's menus and windows. It needs to be able to open and save files like a desktop app. It should be able to provide a systray/menubar item for easy access. A few tricks to getting this to work would be thinking of each web app as a separate instance of the browser, just like desktop applications are separate. Caching web apps could provide pretty decent performance and strong offline APIs (e.g. Google Gears, or a SQLite DB similar to WebKit's HTML 5 implementation) could blur the line even further. Imagine Google Docs in this browser. Or Flickr. Hells yes!

4. Allow OS to be installed completely over the Internet. Look at the beginning of the trend: the MacBook Air doesn't have a DVD drive. Seriously, physical media has got to die. CD/DVDs suck. A lot. The trick is how do we live without them. Trick one: allow a direct install from microsoft.com. Trick two: allow the user to install from an ISO image on another machine similar to Apple's Remote Disk but without the physical disk. Just write some software to server up ISO images. You could use multicast DNS so that there's no configuration, it just auto-discovers available media on the local network.

5. Stop worrying about piracy; you're only pissing people off. Seriously. No more keys, licensing, or Windows Genuine Advantage validation shit. People have about zero tolerance for being punished just because you're paranoid of losing revenue from a few customers. If you have to, think of the lost revenue as marketing your product as not fucking obnoxious.

6. Time Machine style backups to something functionally equivalent to Amazon S3. Copy Amazon's Web Service work. Then you can build amazing products on top of that infrastructure. The goal here is to have all the user's data stored on Microsoft's servers, with the hard drive acting as more of a cache. That way the user can have way more data than their hard drive can hold and they don't have to worry about managing it. All the user's documents will be automatically backed up and placed in a versioning system on the Internet in a fashion similar to Apple's Time Machine. Users don't want to buy hardware and manage it themselves. Highly skilled IT teams do a much better job. I'll explain later how you can afford to do this for all your users.

7. Nail the operating system's user experience. I've already hammered on why using an Apple style menubar is important. I also think it's important to come up with something better than the dock, which is one of the worst UI elements ever invented. Make use of the corners, which Ubuntu does really well. Clockwise from top-left it's the application menu, power menu, trash, and show desktop. Both Windows and Mac OS X fail to use all four corners. What you want to do is really worry about creating a way for your users to be able to string together one small victory after another (e.g. "I downloaded a PDF, and I didn't have anything that could read PDFs, but it downloaded and launched a reader in seconds…it just worked!"). Which leads me to my next point…

8. Autoinstall anything needed like Ubuntu. Users will be grinning ear-to-ear when software simply works. If the user really cared about which Bittorrent app they used they would install it themselves. Which is why users don't really care about what software they use to accomplish a task. Take care of this for the user. The average person wasn't meant to figure this stuff out on their own and it's just a bad experience to force them to manage software by themselves. Users want things to work and the less thinking they have to do about it, the happier they will be. Happiness is what you should be concerned with, as this quote explains how frustrating experiences are today:

Every few days some crappy software I can’t even remember installing pops up noisy bulletins asking me if I want to upgrade something or other. I could not care LESS. I’m doing something. Leave me alone! I’m sure that the team at Sun Microsystems who just released this fabulous new version of the Java virtual machine have been thinking about the incremental release night and day for months and months, but the other 5,000,000,000 of us here on the planet really don’t give a flying monkey. You just cannot imagine how little I want to spend even three seconds of my life thinking about whether or not to install that new JVM. Somebody out there is already firing up Gmail to tell me that the JVM mustn’t just upgrade itself “because that might break something.” Yeah, if the entire collective wisdom of the Java development team doesn’t know if it’s going to break something, how am I supposed to know? Sheeesh. Joel On Software, Elegance

9. Move your apps (e.g. Office) to web for free. No sane human being is going pay $300 or whatever it is you charge as apps become more web-based. You've made your money here. Move on. Besides you can charge users for storage and bandwidth on a per-month basis. Same thing for businesses with collaboration packages on a per-employee, per-month basis. You'll make more in the end. How many companies can I think of still running Office 2003 (or even 2000)? Oh yeah, all of them.

10. Decide on a regular and quick release schedule. Hey, it works pretty well for Ubuntu, although you don't have to be as aggressive as six months. I'd shoot for 12-18 months. 60+ months just isn't going to cut it. Don't ever pull that shit again. Ever.

11. Do sassy advertising; I know you guys have a sense of humor. Be edgy, stick it to the guys who have been tearing you apart. People should be laughing so hard they aren't physically able to send the YouTube URL for your commercial to their friends. I can imagine a nice, sassy marketing campaign sticking it to Apple about their "Think Different" slogan.

12. Hire some young charismatic executives and put them front and center. You're not going to find Steve Jobs, but you can do better than Bill Gates or yourself (although that remix of your little "developers, developers, developers" speech is entertaining, in a sad way). Here's how you should measure success: if there aren't any rumor sites about your new OS, you haven't succeeded.

Bonus Points

Embrace standards and go totally open source. Don't waste time on something just because you didn't build it. Use what's out there. And if you contribute back to the community you'll be buying goodwill for your brand.

What about the bottom line?

OK, I know what Steve is thinking at this point. How the fuck am I going to make any money off this? Right, I've got an idea. Give this futuristic OS away for free. Totally free. It'll be brilliant. Wait that didn't involve money going into Microsoft's coffers. Oh yes, charge for bandwidth and storage your customers actually use. Amazon Web Services style. Backups, system installs, restores, and saving those Office documents. Charge for the storage usage and bandwidth used every month. You can probably charge more than you tried to for Vista. People will love it and not even notice you're making more money than before--nobody likes to pay $200+ at once. For example, XP cost $200. It lasted for six years (2002-2008). This equates to $33 per year for Microsoft. If you just charge for bandwidth and storage at a rate of $5-20 a month, you're making anywhere from $60 to $240 a year. That's more money for you, as well as happier customers with better experiences.

What do I want to do today? Use an OS like what I just described. If anyone can deliver it, it's Microsoft. Apple's the only other player and they just don't really get the Internet. Then again, nobody really does.

Maybe I'm just crazy…thinking that it's possible Windows 7 could actually be the second coming.

So, when is the future going to arrive Steve?

Explaining my addiction

2008-03-23T00:00:00-07:00

I have an addiction; it is simply impossible for me to kick the habit, not that I'm trying. Cocaine? Nah, it's worse than that. Let me explain…

BMW recently updated their M3. It's an amazing car, that's now powered by a monstrously powerful 4 liter V8 engine that revs up to 8,300 RPM and can get to 60 MPH in 4.2 seconds. That's fast…

…for a road car. Imagine a car that has a much smaller 2.4 liter V8 that revs up to the race-regulated 19,000 RPM, summons up gear changes in just milliseconds, and can accelerate to 60 MPH in just 2.4 seconds. And acceleration isn't even the amazing part of a Formula 1 car: its brakes can tear your head off. In just seven seconds an F1 car will start from a dead stop, blow past 125 MPH, and come to rest again with a screeching 5-6 g's of deceleration.

It's not that the BMW M3 is a bad car; It's utterly outclassed on an inconceivable scale. The M3 has a power-to-weight ratio of 230 horsepower-per-ton. The F1 car has more than 1,100. It's crazy. Just watch:

"…190 mph at the bottom of the hill, 5 g in compression, 4 g in lateral forces, you pop out over the top in Les Combes and you're still accelerating, climbing gently uphill too, 210 mph…"

"…bring it down to 90"

"…another 100 mph corner"

Is it just me or is this infuckingsane? 5g compression? At 5 times the force of gravity I'd weigh 875 pounds. Holy shit, how do the drivers cope with that lap after gut-wrenching lap?

How did I get sucked into this? Let me explain...I was never into cars until a certain chain of events. It started with my friends telling me to just visit my school's solar car team meeting. When I showed up they immediately put me in charge of building the telemetry system for the upcoming car. Solar cars are pretty crazy and share lots of engineering such as carbon fiber chassis with F1 cars.

After a few months of solar car I found my first Top Gear video on YouTube. Quickly I caught up with the last few seasons of Top Gear. One of the episodes had a segment where they brought a F1 car out to show just how different they are from even the fastest supercar, the fastest of which was the Ferrari Enzo with a lap record of 1:19. They talked about all the crazy stats behind the car and then put the car in the hands of The Stig. The F1 car destroyed that time with a 0:59 in the wet. This is when I started watching F1—the start of the 2007 season.

Being a nerd, I have an unchangeable need to consume information. I want to grasp all aspects of a problem until I understand it. Then I can move on to the next problem. The only catch with F1 is that the whole system is changing year-to-year and race-to-race. New drivers, new regulations, improved technologies, and unpredictable weather. I'm never going to be satiated and that's OK with me.

I think there's a second reason I'm addicted to F1. When I was on solar car I would constantly think, sketch, imagine, and discuss ways to improve the aerodynamics, decrease the weight, create better systems for race strategy. It's the same thing with F1. I imagine ways to use machine learning to determine race strategy, predict failures, or design a better car. F1's official website has great technical analysis, detailing each of the changes made by the teams with pretty diagrams, and explaining how it will affect the performance of the car, and giving me even more gritty details to waste my precious neurons pondering.

Pitstops in F1 are serious business. Guess how many people are involved on average. If you guessed less than 25 you're wrong. Let's break it down:

3 guys per wheel: 1 to operate the airgun, 1 to take away the old tire, and 1 carrying the new tire
Lollipop man who signals the driver which pit lane to enter, shows the driver where to stop, tells him when to put the car in gear, then, pending traffic, tells the driver to go by lifting the lollipop
2 guys to clear debris caught in the radiators
2 guys, one front and one back to lift the car up so the tires can be changed
3 guys to operate the fuel rig, which dumps more than 12 liters (3+ gallons) of fuel per second
2 guys to adjust front aerodynamics (sometimes another 2 on the back)
2 guys with fire extinguishers just in case
1 driver

Generally pitstops take 6-8 seconds, but if there's damage to anything you can add a few people and seconds. It's "totally fucking busy" as Rands would say.

The engine in an F1 car gets unbelievably hot, reaching 1,800 to 2,000 degrees Fahrenheit. Cooling is provided by 23 cubic feet of air per second during the race—as much air as a human breathes while running a mile. You can imagine why F1 cars can't drive right behind another car for more than a few laps because the cooler air they would normally be breathing in is replaced with the hot exhaust from the leading car.

Sure they barely get over 3 MPG, but I'm guessing they're more efficient at converting fuel into energy than almost anything. When was the last time you took a corner at 140 MPH? Oh, that's right, that's faster than your car's top speed. Trust me if a team could figure how a way to increase their MPG they would do it. Less fuel means less weight which means a few tenths of a second advantage. F1 cars are essentially maximization functions manifested in physical form.

After all the crazy stats, there's even more to love about this sport. And yes, it's a sport. Just try wearing four layers of fire-retardant clothing in temperatures that exceed 120 degree Fahrenheit, while you're subjected to dozens of high g-force accelerations, decelerations, and turns, every lap. These forces punish the driver's body, especially their neck, which must support 3-5 g's on their head in every direction. Over the course of a single race, the extreme heat and exercise causes drivers to lose about 6.6 pounds.

Lewis Hamilton was in his rookie season as I started watching and I quickly become a fan. The 23-year old's enthusiasm and love for F1 really shows in his driving. He nearly took the world championship in his first season, which has never been done, but sadly a couple late-season rookie mistakes (of course) prevented him from sealing victory.

I have yet to leave the U.S. and I plan on it as soon as I can. Let me tell you, it's going to be to see an F1 race. Not sure which one (suggestions?), but I cannot wait to feel the thunderous roar of 22 cars shaking the ground.

Pretty decent self-justification for an obsession, don't you think?

A quick foray into linear algebra and Python: tf-idf

2008-02-10T00:00:00-08:00

My goal with this is to introduce you, the reader, to two things: a beautiful language called Python and an algorithm for finding which words most uniquely describe a document. Hopefully this will be easier than finding an anti-gravity repository, but who knows I hated writing in college and nobody said I was very good at it. Anyways.

Linear Algebra

I'm not sure there's any other when to say this other than to just say it...term frequency--inverse document frequency. That's the name of the game, but I'll call it tf-idf instead of its catchy full name. Tf-idf is straight up statistics for determining how unique a token or word is to a specific document. For instance "battle school" and "toon leader" are probably very unique to Ender's Game. I mean it's pretty unlikely they'll show in something like Harry Potter or Lord of the Rings. What's brilliant about this algorithm is common words such as "the" or "and" aren't unique to any one document, so their frequency in one document compared to all the rest actually approaches zero rather quickly.

Before I scare away anyone with crazy looking equations, let me break it down to you. The td-idf weight of a word is calculated by multiplying together two things, tf and idf, so let's look at tf, or term frequency first.

The term frequency is actually really simple, it's the number of times a given word occurs divided by the number or words in the document. Why do we divide by the number or words in the document? That's just a simple way to normalize the bias toward longer documents. I mean Harry Potter probably has the word "brown" more times than "the quick brown fox jumped over the lazy dog" but it's not as unique to that document. Hence we need a way to level the playing field for documents of all shapes and sizes. Maybe just sizes.

Idf, the other factor in the equation, also known as inverse document frequency, normalizes the term frequency value so that common words used by all documents are given very little importance. Examples include "the" for English documents or "for" for source code, as neither term is unique to any particular document even though they both probably appear very often in all documents.

One example application of this algorithm would be listing the top ten most interesting words for a post on a blog. This would surface words that more or less uniquely identify a post, and could be used for the keywords in the HTML header to maximize the SEO value of a permalink. But of course there's plenty of other uses.

Still confused? Let's just jump right into the code!

Putting tf-idf into action: using Python

As Randall Munroe puts it, Python is executable psuedocode. Take a for loop in PHP for example:

for ($i=0; $i<10; $i++) {
  echo $i;
}

In PHP you use curly braces to indicate control flow. It's great and explicit. But when I'm prototyping something quickly it's kind of annoying to have to use curly braces to realize an idea in my head. Sheesh. If I just wanted to write the pseudocode for it now and write the real, executable code later I would write something like this:

for i in range(10):
  print i

Oh wait, that's Python. And it's executable. Niiiiice. The whitespace is important as it contains the control flow information. Everything that's indented after that for statement will be looped through and the first non-indented statement after the for loop will indicate the end of the for loop.

OK, so let's put this tf-idf algorithm into psuedocode/Python! Let's make it object-oriented so I'll start with the tf part of the equation:

def tf(word, document):
  return (freq(word,document) / float(wordCount(document)))

Hmm, looks like we'll need to write a couple helper functions...

def freq(word, document):
  return document.split(None).count(word)

def wordCount(document):
  return len(document.split(None))

There we go. If you're curious what the split(None) does it breaks a string into chunks when it hits whitespace. Alright, half way there...now for idf:

def idf(word, documentList):
  return math.log(len(documentList) / numDocsContaining(word,documentList))

Again we'll need a helper function that counts the number of documents containing a word:

def numDocsContaining(word,documentList):
  count = 0
  for document in documentList:
    if freq(word,document) > 0:
      count += 1
  return count

Finally we've got both parts of the equations so all we need is one last function!

def tfidf(word, document, documentList):
  return (tf(word,document) * idf(word,documentList))

Tada! That's it...executable pseudocode for finding out how unique a word is to a specific document. If you want to know more check out the Wikipedia entry for tf-idf.

Just copy and paste the following code save it as tfidf.py, head over to your shell of choice and type "python tfidf.py" and you're good to go!

#!/usr/bin/env python

import math
from operator import itemgetter

def freq(word, document):
  return document.split(None).count(word)

def wordCount(document):
  return len(document.split(None))

def numDocsContaining(word,documentList):
  count = 0
  for document in documentList:
    if freq(word,document) > 0:
      count += 1
  return count

def tf(word, document):
  return (freq(word,document) / float(wordCount(document)))

def idf(word, documentList):
  return math.log(len(documentList) / numDocsContaining(word,documentList))

def tfidf(word, document, documentList):
  return (tf(word,document) * idf(word,documentList))

if __name__ == '__main__':
  documentList = []
  documentList.append("""DOCUMENT #1 TEXT""")
  documentList.append("""DOCUMENT #2 TEXT""")
  documentList.append("""DOCUMENT #3 TEXT""")
  words = {}
  documentNumber = 0
  for word in documentList[documentNumber].split(None):
    words[word] = tfidf(word,documentList[documentNumber],documentList)
  for item in sorted(words.items(), key=itemgetter(1), reverse=True):
    print "%f <= %s" % (item[1], item[0])

Download or fork the source code on Github: http://github.com/timtrueman/tf-idf