Photo by En Tsai

This is my thousandth post here, and for me that's an important milestone. Years ago I read a study about abandoned blogs. They mentioned that most died after one or two posts, but one had reached a thousand before it went quiet. In the rough early days, that gave me a goal. If I was going to abandon my blog, I would at least beat that post record goddamnit!

Now I'm here, what did I learn during the last six years? 

Blogging is cathartic

I started blogging out of frustration. I felt trapped in my job, with nobody to talk to about the amazing possibilities I could see in the technology world. I was surprised to find that just typing out my thoughts was a big help, even when nobody responded. The process of organizing my thoughts made my internal struggles make more sense, left me feeling more at peace because it gave me a clearer view of the problems I was dealing with.

I still turn to my blog when I'm frustrated, and the funny thing is the posts that come out of that are often the most popular! Failure sucks, but instructs.

Having an audience is addictive

When I first started I would sit in Typepad's dashboard refreshing constantly in the hope of seeing a visitor. There were some days where nobody at all came to the blog. Now, on an average day between five hundred and a thousand people make it here. A fair number of those are for old posts (there are evidently still damned souls who have to write BHOs for Internet Explorer) but the long slog to build an audience makes me happy to see every single one of them. Knowing that people are actually paying attention to what I write is a heady drug, even at my modest scale.

I get to indulge my innate urge to pontificate without having to inflict it on my loved ones, and even get validation from comments and responses from people I admire. Behind every writer's cool exterior there's something pathetically vulnerable that craves attention and drives them on. Having experienced the thrill of seeing tens of thousands of people reading and discussing something I wrote, I can't imagine how anyone could avoid getting hooked.

Writing fast is a bloody useful skill

When I first started blogging, I set aside thirty minutes each weekday morning to write a post. I forced myself to publish whatever I had at the end of the half hour. This initially led to some awful blog posts, but luckily at that point I had no readers (see above). Over time I found it became easier to create something worthwhile with a tight deadline, and writing at speed was the most important skill blogging taught me. I could produce coherent writing before I blogged but it would take me three or four times as long.

Being able to organize my thoughts and type out an argument or explanation within a few minutes has allowed me to do things I'd never have time for otherwise. Creating documentation, replying to user emails, convincing colleagues, or pitching investors, it's amazing how much of my day is spent writing, and it all goes a lot more quickly thanks to my blogging practice.

Blogging is irrational

When people tell me they're thinking of writing a blog, I try to discourage them. By any sane measure the hours I've spent on this haven't had a great return on investment. The thing is, I can't help it! I need this outlet, and you should only be writing a blog if you've got the same screws loose as me, if you feel compelled.

I've enjoyed the last six years of blogging more than I can say, and as a final note I'd like to thank all of you for joining me in this long conversation. I've learned so much from everyone, and made some lifelong friends. I'm so grateful I had the chance to make so many wonderful connections, and I hope you'll join me for another thousand blog posts!

Photo by Tanaka Juuyoh

strcpycat - How hard can it be to write a function to copy one string to another? This exploration shows how tough it is to create an algorithm that's truly generic. Seemingly-harmless design choices like returning the length of the source string will kill you when you're copying small chunks from a massive string.

Is there life on Venus? - We believe our own eyes, even when we shouldn't. This is a cautionary tale of a respected Russian astronomer who started to see life forms in the image-processing artifacts of old space missions. I used to generate 8x8 sprites for my 80's game programming by cycling through random blocks of pixels until something caught my eye, so I'm aware of how powerful pareidolia can be.

How to digitally sign a PDF - I can't believe I never knew you could do this, I've wasted so much time printing out and rescanning documents over the last few years of startups! In Lion it's so easy, you can just write your signature on a piece of white paper and hold it up to the camera, and after that just position it in any PDF you've loaded in Preview.

IMDB data set - Emphatically not free and open, but at least available, I'm intrigued by the Kevin Bacon possibilities here.

Computer Scientists and Google+: Something Interesting is Happening - As you may have noticed, I'm optimistic about Google+'s prospects. It comes down to my personal experiences, I'm discovering a lot of content that I just don't see on Facebook or Twitter, and it looks like I'm not the only one. 

Photo by Miuenski

Fundamental Oracle flaw revealed - This is a fascinating piece of detective work on a bug, but also a cautionary tale of how even the most conservative assumptions can be proved wrong as data processing speeds and volumes grow.

Extracting structured data from Common Crawl - Shows exactly why I'm so excited by the potential of Common Crawl. Even just a list of all the hcard records from five billion web pages is going to be an amazing research resource, I have plans for doing fun things with the street addresses already.

Travel itineraries with long-exposure photos - I love the way the students used analog techniques to produce a high-tech looking visualizations.

Social Graph and Needlebase are dead - Google's API for publishing unified public profile information to developers never really caught on, but it's a shame to see it vanish. Needlebase's shutdown is less surprising, it always seemed likely to be useful more internally to Google, but I'm still sorry to see it lost to the outside world, it was a great tool.

The Apple logo in unicode - It's great to see how convoluted and political something as seemingly-simple as defining an international character set can be.

Photo by Eva Ho

Google+ has become very popular with photographers and hosts some amazing pictures, so I've been keen to help people discover the awesome travel ones through Jetpac. It took some head-scratching (the API is still in its very early stages) but you can now sign up using your Google account! Log in, and we'll give you inspiring photos from people you follow for wherever you're dreaming of traveling. It's been awesome discovering the wonderful content friends like Eva are putting out there, pictures I'd never have known about otherwise. I bet you'll find some treasures too!

Photo by Mark Kelley

If you're in the Bay Area on February 8th, I highly recommend joining me at the Silicon Valley Big Data group's war stories event. It's being put on by some good friends from places like Kosmix (now Walmart Labs) and other folks who've been fighting in the Big Data trenches. The goal is to demystify the field, and show how any engineer can learn the techniques you need to create value from massive data sets, it's not just for Stanford PhD's any more! I hope that after hearing the stories and talking with the panelists, you'll feel confident you can dive in and start hacking.

Photo by Daryl Mitchell

DynamoDB - I can't tell you how excited I am to see Amazon's new hosted NoSQL service. I so wanted SimpleDB to be usable, but it was too hard to work with thanks to the way it required clients to deal with implementation details like servers and sharding. I ended up using S3 for some projects, but if I was starting Jetpac today I'd seriously consider going with Dynamo rather than self-hosted Cassandra. The main drawback for our requirements is that we couldn't run Pig scripts across the data, but I'd imagine analytics will come.

Unsupervised Decomposition of a Document into Authorial Components - Uses machine-learning techniques to figure out the authorship of books from the bible, in a way that matches the results of traditional biblical scholars. I love applications of computer science to the humanities, I think there's a lot of ground for cross-fertilization.

The state of NoSQL in 2012 - An insightful look at the past and future of the new wave of database systems, from someone who's been in the trenches working with them for years.

Auto scaling in the Amazon cloud - Describes how Netflix keeps up with demand by automatically creating and destroying instances based on usage. The CPU utilization graphs show effective they are at managing the process, but they also hint at the work that's required to build AMIs for all services that can be spun up automatically.

Google Plus Scraper - There's no API yet to most of the interesting bits of Google+'s content, but a lot of it is available publicly to web crawlers, and the information is delivered as a giant JSON array embedded within the page, so it's surprisingly easy to decode. The biggest pain is the space-saving variant of JSON that Google use, which leaves out explicit null values in arrays, so you get ['a',,'b',,,'c'] instead of ['a',null,'b',null,null,'c']. 

One of the toughest challenges with Jetpac is persuading people to sign up with Facebook. It has a lot to offer if you make it over that hurdle, but some of the most thoughtful people I know never will, for reasons I respect and understand.

The nice thing about using unstructured text as our location information is that we can go anywhere there's good photo captions. We started on Facebook just because there's so much data available (the average user has had over 200,000 photos shared with them by their friends) but now I'm finally getting a chance to address other sources. First up is Tumblr, so if you have your own or are a fan of one that mentions locations in any of the posts, you should be able to get a quick visualization of it, no login required, by going to:

https://www.jetpac.com/globe/create

Here's an example of what you'll get:

http://blog.jetpac.com/post/16028230197/everything-everywhere-travel-photography

We take all your photos, and build an HTML5/WebGL globe to help you explore them by place. Type in the URL, get a globe, it's as simple as that!

Just for fun I pointed the service at The Economist's Tumblr, to see how it coped with posts that definitely weren't travel photos. It didn't turn out half bad!

https://www.jetpac.com/globe/theeconomist.tumblr.com

Photo by Phillip Chapman-Bell

Zipscribble map of Italy - Patterns leap out of post code data when you connect adjacent codes with lines, and color them according to the most significant digits. Shows how useful pictures can be when we need to make sense of complex data. 

The Gentleman Hacker's 1903 lulz - Innovators never enjoy pesky outsiders pointing out flaws with their technology. What's interesting is that I couldn't discover any practical exploits against early radio signals, despite how obvious the flaws were in the wake of these demonstrations. I did discover there's a world of Morse code software I never imagined existed though!

ARM instruction guide - Back in 1990, I learnt ARM assembler as my second language, after Basic, and though I haven't used it since then, I enjoyed this companion to a workshop on hacking the processor's security model because it's actually a concise, useful guide to its important features.

A revolution in mathematics? - An approachable look at the underappreciated changes in maths at the end of the 19th century. The short version is that the process of creating proofs became highly formalized, which sounds dry as dust but actually opened up a world of new possibilities. It's worth quoting at length:

"Well-optimized modern definitions have unexpected advantages. They give access to material that is not (as far as we know) reflected in the physical world. A really “good” definition often has logical consequences that are unanticipated or counterintuitive. A great deal of modern mathematics is built on these unexpected bonuses, but they would have been rejected in the old, more scientific approach. Finally, modern definitions are more accessible to new users. Intuitions can be developed by working directly with definitions, and this is faster and more reliable than trying to contrive a link to physical experience...rank and-file mathematicians can use the new methods confidently and effectively, while success with older methods was mostly limited to the elite"

Stepping away from "common sense" and relying on the logical outcomes of an abstract system that doesn't provide intuitive reasons sounds a lot like where we're headed in the data world. The question "Why does that work?" will often come up when you're making choices based on AB testing, and often the honest answer is "I don't know, but it does!".

Spark - A framework implementing a higher-level approach to writing distributed algorithms, with a more readable statement of the problem than standard MapReduce produces.

Photo by RHiNO NEAL

Your ideal performance/consistency tradeoff - It's unclear what the right number of nodes and level of redundancy for a Cassandra cluster are for any particular performance requirements, so most of us experiment until we have something that vaguely seems to work. Thanks to the folks at Berkeley, there's now a better way to figure it out via an interactive tool. Interestingly, they ended up using a Monte Carlo simulation rather than a formula, which shows how complex the problem is.

Why is finance so complex? - One of the most interesting articles I've read in a long time. It posits that finance is effectively a benign con trick, and relies on a lack of transparency to encourage people to take risks they wouldn't if they fully understood what they were getting into. The idea is that it's a collective action problem that only works if everyone jumps on board, and so the opacity helps persuade people to do that and achieve a better overall result than if they made an individually-rational choice. The model seems like it might explain other odd features of our social world.

Run a MapReduce job across five billion web pages for 25 cents - I have a massive data-crush on Common Crawl, and this is a fantastic practical demonstration of why I'm so excited. 

Clickjacking - The web's security model is more like Windows' than Unix's. It's been grafted onto an underlying system that was designed without any security foundations, and there's lots of gaps where different components interact in exploitable ways. This page explains how there's no reliable way to prevent malicious sites from hosting your site as an invisible frame and tricking users into taking actions by unknowingly clicking on it. Luckily we're in a world where software can be frequently updated, unlike 90's desktop software, so at least if this becomes widespread we might quickly see some fixes.

Muse - A noble experiment in mining useful data from your own email archives. It's still a bit too buggy to really get a feel for how interesting the results could be though.

Photo by Let Ideas Compete

Rust - A trap to ensnare unwary web crawlers, by Tim McNamara. It creates pathological patterns of input data that will slow down naive robots by the sheer volume of processing required, whilst using minimal resources on the server thanks to elegant event-driven code. It's effectively a reversed denial-of-service attack, designed to overwhelm malicious or thoughtless crawlers of your site. Well-written and robust robot scripts will cope with malformed input of course, but the odds are that any crawler that's bringing your site to its knees with an unreasonable number of requests won't be a masterpiece of engineering!

Seeing like a database - Written by another fan of Seeing like a State, this has a great quote from Jay Owens at the end, noting "the asymmetry of personal data, open for the 99% & deep analytics for the 1%".

HttpBin - Echoes back information about HTTP requests you send it, including things like headers, data, and forced result codes. I'm just thankful it introduced me to the 418 (I'm a teapot) status code, I can't believe I've been writing web code for so long without checking for that possibility.

Drone landscapes, intelligent geotextiles, geographic countermeasures - I'd never realized how deeply adding processing to landscape structures could change our world. This is a compelling exploration of some of the possibilities, and I'm especially struck by the possibilties for a robot-readable world.

An end to bad heir days - The copyright on James Joyce's work finally expired! The enforcement process became a poster child for how the combination of insanely-long copyright terms and ornery heirs can derail the enjoyment and exploration of an artist's work. Thankfully scholars are now free to quote Joyce's work and letters, and I've just downloaded A Portrait of an Artist as a Young Man to re-read in celebration.

Photo by Richard Paterson

The Ugliest Map in the World - Such an eyewatering color scheme, you'd think I'd designed it. The swimming-pool bottom caustics for the ocean areas really clinches it.

The Life of a Typeahead Query - An exploration of how hard it is to make an easy interface. Great to see a practical example how someone architected a real-world system with messy requirements.

Ending the Infographic Plague - Visualizations are an excellent hack for getting publicity, which inevitably leads to pollution by bad actors.

The Mess that is NPM - I really, really want to use Node.js, but the library ecosystem isn't quite mature enough for me to use in production. There's a lot of non-technical community hacking that you need to do to create a strong set of modules, and responsible maintainership isn't something I'm perfect at with all my projects, so I know how hard it is.

Brain Grain - Tasty little HTML5 visualization of world-wide migration. It's pretty simple, but has some innovations I've not seen elsewhere and uses animation effectively.

And last but not least, Jetpac is now rounding out a fundraising round, so if you're on Angelist any comments or recommendations would very welcome.

I spent this afternoon wandering the British Museum's Mesopotamian collection, and I was struck by what the humanities graduates in charge of the displays missed. The way they told the story, the Sumerian's biggest contribution to the world was written language, but I think their greatest achievement was the invention of data.

Writing grew out of pictograms that were used to tally up objects or animals. Historians and other people who write for a living treat that as a primitive transitional use, a boring stepping-stone to the final goal of transcribing speech and transmitting stories. As a data guy, I'm fascinated by the power that being able to capture and transfer descriptions of the world must have given the Sumerians. Why did they invent data, and what can we learn from them?

First you get the data, then you get the power

The Sumerians were a nasty lot. Their idea of a fun time was wheeling a bunch of caged lions into an arena so the king and his friends could shoot them from a chariot. One of the perks of working for a king was the opportunity to drink poison and join him in his grave. They created seals and cuneiform writing as tools of power. They kept track of who owed them what, in a way that left evidence that could be used to convince a third party of the obligation. I could swear blind that you'd verbally promised me three lambs in the spring, and it would be your word against mine. With a written record of the transaction, I could convince the rest of the community that it was true. If you don't hand over those lambs, some of them might help me stick that dagger between your ribs. Since these sort of obligations are the foundation of any state, the earliest writing was a potent source of power.

That's still true today. Gathering data is not a neutral act, it will alter the power balance, usually in favor of the people collecting the information.

Power corrupts data

"The inscription on this stone is a statement of grants and privileges bestowed on the sun-god Shamash's temple by the Akkadian king Manishtushu (2269-2255 BC). It was actually written many centuries later. The object was clearly a forgery designed by the Sippar priesthood for their purposes."

As soon as records become vital in arguments about who gets what, people will figure out how to falsify them. The more important the outcome, the more temptation there is to fudge or fake them. Written records remove the problem of fallible memories, but replaces it with a second-degree question of provenance. How do you know the data accurately reflects what happened?

It's a good reminder that the map is not the territory. We still have a disturbing tendency to trust anything that's recorded, without understanding the subjective process that went into creating the record.

(Pre-)Digital Rights Management

This stone was planted in the ground to mark a property boundary, and the top section records the details of the claim. The bottom third is covered with threats of supernatural retribution against anyone who moves or alters the marker. The main way Sumerians protected the integrity of their data was through curses. This may seem laughable to a modern audience, but I don't think we're so different. Does you expect the FBI to actually raid your house if you copy that VHS tape? The warnings are a way of forcefully expressing society's norms, rather than a credible threat of punishment.

As geeks we'll often roll our eyes at a technically-ineffective mechanism for preventing the copying or alteration of data, but the longevity of useless curses should make us think twice. Violating the rules is a decision taken by a person, so sometimes hacking the human element of the process is the most effective prevention.

Reading the future with data

Many of the tablets archaeologists have recovered are elaborate instruction manuals on how to interpret omens. The idea was that you'd observe events that were happening now and use them to predict what was going to happen in the future. All the examples I saw at the Museum were obvious nonsense, using inputs like the shape of animal entrails, but what struck me was how respected they must have been despite their lack of results.

We've created science as a much more elaborate process for predicting the future from data, but in many ways that's lulled us into a false sense of security. The media prominently features 'scientific' studies showing that everything gives us cancer, thanks to our insatiable appetite for certainty and reassurance in the face of something terrifying and unpredictable. The lesson for me is that the results of any data-driven project will be accepted or ignored based on people's needs and fears. In the absence of real answers, we'll take bogus ones painted with a veneer of data, just like the Sumerians.

All data matters

We actually know more about everyday life in Sumeria five millenia back than we do in Europe fifteen hundred years ago. The Sumerians recorded everything on stone or clay tablets, most of which were discarded after use with no thought for posterity. As it happened, the clay tablets proved remarkably resilient and so archaeologists and scholars have found and decoded hundreds of thousands of them. This data exhaust gives a rich view into trade, worship, life, death, medicine and almost every other aspect of the Sumerian's world.

This is a big reason why I'm so fanatical about opening up data sources. It's great to see Twitter taking steps to archive our public conversations in the Library of Congress, but it's taken a year and they're still not finished. Even when they're done, storing the records in a single location and on a single system is a terrible long-term plan, the only approach that's proven to last centuries is wide distribution of many copies on a range of mediums. Craiglist is another bad example, holding information that could be vital to understanding details of our social and commercial lives in the future, data that's been on view to the public, and yet refusing to discuss archiving any of it and actively blocking anyone who tries. If there's any way you can, please think about how to open up data you control, it's the best way to pass it on to posterity.

See for yourself

I had an amazing time in the British Museum's Mesopotamian galleries, I'd highly recommend it if you're ever in London, and it's completely free. Data was the aspect that fascinated me, but there's so much more held in the treasury of beautiful objects their scholars have collected, I guarantee you'll come away with a feeling of awe, and maybe a fresh view of the world around you too.

Pigs' ears, anyone?

The longer I've lived in America, the more of a stranger I feel when I return to England. What struck me this visit was how exotic the local street markets feel. When I was a kid, market stalls were an absolute last resort when your parents couldn't afford to buy something in a proper shop. They were the homes of cheap batteries that never lasted, and brand-name clothes surrounded by a aura of suspicion. In San Francisco "farmer's markets" are at the opposite end of the scale. If the Ferry Building slid into the sea on a Saturday morning we'd lose half the Bay Area technology workforce.

These same stripey-neon gloves were there in the 80's

I walked around Bury St. Edmund's and St Ives' markets this weekend, and I was struck by how much things had changed. There was still a fine selection of dubious clothing, but there were attractions for the gourmet too, with game pies and Spanish hams.

The rain in Spain probably isn't this grim

It was a good reminder that England isn't as backwards as I sometimes assume based on my memories. My brother has been involved in market stalls on-and-off over the years and he gave me some interesting background. In Bury there used to be a waiting list for space but now there are empty spots. That's opened up opportunities to new traders with less-traditional merchandise, but it's because the older stalls are closing.

I was sad when I saw the local butchers had just closed down too, but then my sister described a visit with her husband's family, where the meat looked extremely unappetizing. I was reminded of that when I saw a stall in Bury selling meat from cardboard boxes.

It's easy for me to slip back into nostalgia, but supermarket meat counters are better than the average butcher's shop I grew up with. I'm hopeful that the best traders with something unique to offer will do well, and I shouldn't mourn the passing of the rest too much.

It was a good reminder of the limits of my knowledge of Britain these days. I left because I was frustrated at the resistance to change, but progress still happens, even if it's not at the pace I'd like. There's also usually a complex story behind the surface, and as an outsider I'll often miss it. The UK changes a lot more than you'd think, because the British do a great job of transforming things while maintaining the appearance of continuity, in street markets and everything else.

If you want to experience them for yourself, I highly recommend looking at the small towns near where you'll be in Britain. The network of cities was built up around fairs and markets, and you'll still hear medium-sized rural places described as "market towns". You'll often find a morning or two a week the center is closed off and stalls set up. It's Saturday and Wednesdays for Bury St. Edmunds and Monday for St. Ives.

Photo by Ian Dolphin

As we're getting more traffic to the Jetpac home page (thanks AllThingsD!) optimizing our conversion rate has become a priority. In our case, the action we want people to take is connecting with Facebook, so we're having to work quite hard to figure out what messages work best to persuade people. In previous projects, it's often been quite a surprise exactly what sentences work, and so the only way is to test a lot of different approaches and see which strike a chord.

I love KissMetrics as a measuring tool for that sort of experimentation, but I couldn't find a good example of how to do the sort of tests we need. Ideally it should be all data-driven, so that the even less-technical members of the team can edit the copy options and try out new variations without my involvement. I built out a small framework that does everything we need, and have just open-sourced it as a github project:

https://github.com/petewarden/copyoptimizer

For example I have a headline with the class name 'action_header', and a link I want people to click with the class 'action_button', so I have an example data structure like this:

g_copyChoices = { 

  'action_header':['Please click me!', 'If you wouldn\'t mind, click here', 'I\'d really like you to<br>click below'],

  'action_button': ['Start Here', 'Next', 'Sign up']

};

When the page is first loaded, the inner HTML of the elements with those class names is replaced with a randomly-selected string from the array, and the choice is stored with KissMetrics so we can see which ones convert best in the reports. I also store the choices in cookies so that repeat visitors see the same text, and we don't pollute the metrics with varying choices.

Once the data has had a chance to percolate through Kiss's servers, you can choose the class name from the drop-down menu below 'Funnel Overview' on the report page and see which of the messages had the best conversion rate.

http://petewarden.typepad.com/searchbrowser/2011/12/how-to-easily-optimize-your-landing-page.html tag:typepad.com,2003:post-6a00d83454428269e20162fdc263bf970d 2011-12-13T15:05:18-07:00 2011-12-13T15:05:18-07:00

Photos by Tang Yau Hoong The tourists have left - Despite the early-stage hype, there's fewer VCs around than ever. Converting addresses into lat/long coordinates in Excel using the Data Science Toolkit - I love seeing the creative way people...

pwarden

Photos by Tang Yau Hoong

The tourists have left - Despite the early-stage hype, there's fewer VCs around than ever.

Converting addresses into lat/long coordinates in Excel using the Data Science Toolkit - I love seeing the creative way people use open-source projects once they're out in the wild.

Strata - A good overview of what's on offer at the Big Data conference, featuring your correspondent with "Embrace the Chaos", and with a 20% discount.

Google code prettify - A beautiful little Javascript hack for syntax-colored display of all sorts of computer languages in web pages.

12 Things Brad DeLong Got Wrong in his Career - A bit like a VC firm's anti-portfolio, acknowledging and even celebrating your mistakes is a fun way to keep yourself intellectually honest. I always loved the idea of the slave at a Roman Triumph whose job it was to whisper to the honored general "Remember you're mortal".

http://petewarden.typepad.com/searchbrowser/2011/12/five-short-links-2.html tag:typepad.com,2003:post-6a00d83454428269e2015394443041970b 2011-12-09T19:15:56-07:00 2011-12-09T19:15:56-07:00

Photo by Andrew Hudson EntityTagger - A pleasantly practical natural-language processing paper, via Nat Torkington How prostitution and alcohol make Uber better - A clever tabloid hook for an interesting data story. One thing I've heard that might explain part...

pwarden

Photo by Andrew Hudson

EntityTagger - A pleasantly practical natural-language processing paper, via Nat Torkington

How prostitution and alcohol make Uber better - A clever tabloid hook for an interesting data story. One thing I've heard that might explain part of the pattern is that police shifts vary regularly by day, which can impact arrest times.

Social Network Analysis for Telecoms - I've repeatedly heard this used as an anecdote, but it wasn't until I was sitting at an event next to Mike Driscoll this week where it was mentioned that he was able to point me to his original research. It's great to see the original research, I can understand why it's now a classic example of how useful data science can be.

Hue Histograms - A charming way of visualizing image color characteristics by another friend's company. I'm lookin at good ways of anonymizing image data in a way that still preserves enough signals to be useful for machine learning, and this has given me some ideas.

Break an image into tiles - On the topic of images, I was pleasantly surprised at how easy ImageMagick was to install on OS X through MacPorts, I used to dread the failed dependencies. I used the recipe in the article for a hack I'm quite proud of. I needed to generate 'percent of the world seen' thumbnails for Jetpac public profiles shared on Facebook, so I manually created the HTML for a page with a grid of one hundred of the elements, one for each number, took a screenshot and then ran it through the grid command to get the numbered images I needed. You can see it in action if you like this sneak peek of my public profile page - you can unlike it afterwards if you don't want my new pensive portrait in your stream.

http://petewarden.typepad.com/searchbrowser/2011/12/five-short-links-1.html tag:typepad.com,2003:post-6a00d83454428269e20154380a7627970c 2011-12-08T15:03:35-07:00 2011-12-08T15:03:35-07:00

Photo by Andrew Magill One lesson I learned from Eric Ries is how powerful an incremental, reactive approach to testing can be. It's really hard to balance resources between development and testing, especially as a starving startup, but if you...

pwarden

Photo by Andrew Magill

One lesson I learned from Eric Ries is how powerful an incremental, reactive approach to testing can be. It's really hard to balance resources between development and testing, especially as a starving startup, but if you build tests to catch errors that have actually happened, you know you're focused on high-priority issues.

We started Jetpac with a very minimal deployment process with few automated checks, but two stages so we could eye-ball the test environment before pushing it to the final set of live servers. Yesterday that manual process finally failed after we accidentally pushed a completely broken build to the main site and took it down for a few minutes. That gave me a strong reason to add the first automatic checking to our deployment scripts to make sure we couldn't push to production if the test environment wasn't responsive.

To start with I just wanted something very basic that will catch glaring errors that stop our Ruby app from running entirely, since that was what actually happened and they're pretty simple to detect. To do that, I wrote a short Ruby script that can be invoked from the command line and will spot empty responses, 404's and other obvious problems with a URL. We invoke it like this in our deployment bash script, after calling Capistrano to do the actual push:

smoketest.rb "http://testingenvironment.example.com"

if [ $? -gt 0 ]; then

    echo '*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!'

    echo "Deployment not allowed, test server is not responding"

    echo '*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!*$!'

    exit 1

fi

It will print out information to stderr about any problems it encountered, and handles both http and https URLs. I'd imagine as our needs grow we'll turn to something more complex like Capybara, but for now this simple script is a very quick and easy way of catching a lot of common problems.

http://petewarden.typepad.com/searchbrowser/2011/12/how-to-run-simple-smoke-tests-in-ruby.html tag:typepad.com,2003:post-6a00d83454428269e2015437f691c8970c 2011-12-07T01:00:48-07:00 2011-12-07T01:00:48-07:00

Photo by Mugley Mailana, Inc is dead. This week I've been going through the formalities, squaring away the legal paperwork and returning the tiny amount of money I'd raised, but truthfully it had been dead a long time, I just...

pwarden

Photo by Mugley

Mailana, Inc is dead. This week I've been going through the formalities, squaring away the legal paperwork and returning the tiny amount of money I'd raised, but truthfully it had been dead a long time, I just hadn't faced it. I'm over the moon about my new startup, but it feels like time to raise a glass to the last three years of my life, and the majority of my savings.

It began as a dream while I was still at Apple. I knew I wanted to strike out on my own, but by a strange kind of luck the painfully-slow green card process kept me living the corporate life for five years while Apple's shares kept rising, and my tiny windfall from technology I sold them when I joined became enough to live on for a few years. Within a couple of weeks of getting my permanent resident's status I handed in my notice, and set out to Build A Startup.

Technology risk

The hardest lesson to learn was how obsessed with technology I am. I had a problem in mind, one that I'd lived with at Apple - how to identify experts in large companies - but to be honest I chose that because I already had a solution that involved interesting engineering. I spent a year building a pipeline that could semantically hundreds of millions of email messages on a shoestring budget, seamlessly interface with Exchange, and present the results as beautiful visualizations. The only thing I failed to do was sell it to the enterprises I claimed were my customers. I wasn't a complete idiot, I spent time flying to boardrooms and talking to mid-level executives, creating demo videos, I even wrangled a few free pilot programs, but fundamentally I didn't care enough.

Shiny things

That meant when I'd proven my technical point, and faced a mountain of distribution problems instead, at some level I started to look for ways out. I'd already been using Twitter as a source of hundreds of millions of public messages for my demos, and then the public versions of the visualizations started to get some attention. I'd soured on the enterprise sales experience, so I started to explore what I could do on the consumer side. The trouble was I'd completely lost sight of what problem I was tackling. At least with the original version I'd set out to fix an issue I'd spent years living with. Now I was driven purely by curiosity, hoping I'd find neglected data that was so useful the problems I'd apply it to could be an afterthought.

Lonesome founder

I knew myself well enough to spot some of this at the time, and that the best prescription was a business and product-focused founder. I spent a lot of time dating potential partners, especially as I went through Techstars, but there was never a good enough fit. I needed somebody who was willing to bet on what we'd now call Big Data, who'd believe that there was a coming revolution that would bring data-processing problems that had previously required millions of dollars of investment within the reach of early-stage startups. Without external validation, nobody non-technical was convinced of either the concept in general, or my particular ability to execute on it.

Chasing the dragon

Muttering to myself in true mad scientist fashion, I set out to prove them all wrong by adding so many awesome features to my by-now-Gmail-addon Mailana that the world would have no choice but to sit up and take notice! One of these features was an email-contact-to-social-network-profile connector that involved me indexing public Facebook profiles, and then getting in a legal kerfuffle. Nightmarish as the situation was, the publicity and validation I received from that visualization work was addictive. I set out to explore that more, with the excuse that it provided distribution opportunities for my business, but if I was honest with myself it was because I found the whole area fascinating.

Startup neglect

I wandered farther and farther from my nominal business, first as I launched OpenHeatMap, and then as I delved into data arcana through books and journalism. It felt great, because I was actually making a difference to the world, I was having an impact! Sadly I wasn't building a company. I took one final stab at that with the Data Science Toolkit as the final product from Mailana, but in my heart I knew that it had a lot more potential as a long-term open-source project than as a revenue-generating business.

Closure, and a new start

It took a lot of soul-searching to accept, but I knew Mailana was over. I'd originally given myself an allowance of two years to spend with no revenue, and it had been over three. I was lucky enough to have a circle of trusted friends who were working on interesting projects, but Julian and Derek were particularly appealing. I'd been working with them for months as an advisor and fell in love with the idea behind Jetpac. It gave me the chance to keep exploring a lot of the technology ideas I was fascinated, but within the context of an actual business, with a revenue model, funding and more than one employee!

I don't regret any of the time or money I sank into Mailana, I've got so much to be thankful for over the last three years. The people I've met alone make it worth every minute, and I feel like I've now got a lifetime's worth of mistakes to learn from! I'm sorry to say goodbye to Mailana, but glad I had the chance to try something crazy and fail.

http://petewarden.typepad.com/searchbrowser/2011/12/death-of-a-startup.html tag:typepad.com,2003:post-6a00d83454428269e20162fd755231970d 2011-12-06T18:59:25-07:00 2011-12-06T18:59:25-07:00

Photo by Jannis Andrija Schnitzer On being wrong in Paris - A great general meditation on the slippery nature of facts, but the specific example is very resonant. We tend to think of places having clear boundaries, but depending on...

pwarden

Photo by Jannis Andrija Schnitzer

On being wrong in Paris - A great general meditation on the slippery nature of facts, but the specific example is very resonant. We tend to think of places having clear boundaries, but depending on who I was talking to I'd describe my old house as either in "Los Angeles", "Near Thousand Oaks" or "Simi Valley". Technically I wasn't in LA, but the psychological boundaries aren't that neat.

The devil in the daguerrotype details - The detail you can see on this old photograph is amazing, and I love how they delve into the capture method. I was disappointed there was nothing on the role of lenses as a limiting factor on resolution though, I'd love to know more about that.

Katta - A truly distributed version of Lucene, designed for very large data sets. I haven't used it myself yet, but I'm now very curious.

Hbase vs Cassandra - An old but fair comparison of the two technologies. This mirrored the evaluation I went through when picking the backend database for Jetpac, and I ended up in the same place.

It's cheaper to keep 'em - Your strategy is sometimes pre-determined by what numbers you're paying attention to. If you start off with the assumption your job is to get new users as cheaply and fast as possible, you'll never realize how important retaining existing customers can be.

http://petewarden.typepad.com/searchbrowser/2011/12/five-short-links.html tag:typepad.com,2003:post-6a00d83454428269e20153941311b1970b 2011-12-05T19:06:04-07:00 2011-12-05T19:06:04-07:00

A friend inspired me to dig around in my digital attic, and resurrect a video of one of my live VJ performances. It's playing off the music of Richie Hawtins and Pete Namlook, and created on the fly using my...

pwarden