eddology • by edd dumbill

Your week would have been even better if you'd read these

edd@me.com (Edd Dumbill) — Sun, 12 Feb 2012 17:21:00 -0800

And so would mine. Every week, my Instapaper queue fills with awesome links that I rarely get time to read. I figured I’d share some from the last 7 days.

History

“Albert Einstein, whose theories exploded and reshaped our ideas of how the universe works, died on April 18, 1955, of heart failure. He was 76 … Armed with his camera and a case of scotch — to open doors and loosen tongues — Morse compiled a quietly intense record of a 20th-century icon’s passing. But aside from one now-famous image — of Einstein’s office, exactly how he left it, taken hours after his death — the pictures Morse took that day were never published.” http://life.time.com/history/the-day-einstein-died-a-life-photographers-story/#1

Business bits

The book is dead, long live the book! “Lean Publishing is the act of self-publishing a book while you are writing it, evolving the book with feedback from your readers and finishing a first draft before optionally using the traditional publishing workflow.” http://leanpub.com/manifesto

“A software-defined network, which originated in government spy agencies, is similar to server virtualization, and because of that is quite likely bad news for networking equipment makers like Cisco and Juniper Networks.” http://bits.blogs.nytimes.com/2012/02/06/start-up-nicira-plans-to-disrupt-networking-giants/

Brain bits

Any writer, programmer or artist gets “flow”, and gets how illusive it can be. Is it possible to induce the flow state electronically? http://www.newscientist.com/article/mg21328501.600-zap-your-brain-into-the-zone-fast-track-to-pure-focus.html

“When you are solving a difficult problem re-ask the problem so that your solution helps you learn faster. Find a faster way to fail, recover, and try again. If the problem you are trying to solve involves creating a magnum opus, you are solving the wrong problem.” http://www.azarask.in/blog/post/the-wrong-problem/

Programming

This week, a somewhat philosophical take on the discipline.

The 10 rules of a Zen programmer – of which I think the most important has to be “no ego”. http://www.grobmeier.de/the-10-rules-of-a-zen-programmer-03022012.html

There is no root cause of failure, says John Allspaw. Be suspicious of clearly attributed single “causes”. http://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/

In the same way, you shouldn’t have just one teacher. Including yourself. http://raganwald.posterous.com/autodidacticism

Awesome things I might have read this week

edd@me.com (Edd Dumbill) — Fri, 03 Feb 2012 07:06:15 -0800

Every week, my Instapaper queue fills with awesome links that I rarely get time to read. I figured I’d share some from the last 7 days.

Self improvement

Take nothing personally: don’t let your emotions hold you back
http://bennesvig.com/2012/01/27/taking-nothing-personally/

Why every professional should consider blogging: this is an excellent read. Even if nobody ever reads you, you’re still doing yourself a great favor. Writing on G+ counts, too!
http://technicalblogging.com/why-every-professional-should-consider-blogging/

Silicon Valley’s Productivity Secret: I include this because mostly I’m amused at a basic technique being heralded as a secret. Accountability is a superpower.
http://blog.idonethis.com/post/16736314554/silicon-valleys-productivity-secret

Design

“We do more reading on the screen today than we did even a year ago. If we are ever to have a golden age of reading on the screen, this might be the start of it.”
http://coding.smashingmagazine.com/2012/01/30/the-future-of-screen-typography-is-in-your-hands/

A Beginner’s Guide to Pairing Fonts
http://webdesign.tutsplus.com/articles/typography-articles/a-beginners-guide-to-pairing-fonts/

Programming

Java Sucks, by Jamie Zawinski. Written in 2000 but pretty instructive still. “I think Java is the best language going today, which is to say, it’s the marginally acceptable one among the set of complete bagbiting loser languages that we have to work with out here in the real world.” I have no idea if JWZ loves Clojure, but I hope he does.
http://www.jwz.org/doc/java.html

Cloud computing and big data

A Tour of Amazon’s DynamoDB: in-depth briefing on the new database
http://www.paperplanes.de/2012/1/30/a-tour-of-amazons-dynamodb.html

The Ups and Downs of the “America-Proof” Cloud
http://www.readwriteweb.com/cloud/2012/02/the-ups-and-downs-of-the-ameri.php

Google’s Data Center Engineer Shares Secrets of “Warehouse” Computing
http://m.wired.com/wiredenterprise/2012/01/google-man/

The Great Disk Drive in the Sky: How Web giants store big—and we mean big—data
http://arstechnica.com/business/news/2012/01/the-big-disk-drive-in-the-sky-how-the-giants-of-the-web-store-big-data.ars/1

Frameworks, freedom and the cloud

edd@me.com (Edd Dumbill) — Tue, 31 Jan 2012 16:25:00 -0800

Open source web frameworks have been a big driver of web productivity in the last 5 years, and having them cloud-hosted “out of the box” is highly desirable for most people. But is there a downside?

In one sense, frameworks are endangered by a cloud future. We’re slowly moving to a platform-oriented cloud (PaaS), as opposed to a virtualized infrastructure one (IaaS). In this future we’ll be using platform-as-a-service solutions such as Amazon’s DynamoDB or Google’s App Engine. In this world, the databases and frameworks will run out of the box. But where will the new open source frameworks come from? With little opportunity for development in the broader ecosystem, few will have the incentive to improve or create new frameworks.

We should be as be wary of cloud “platform-as-a-service” as much as the closed world promoted by iOS. By abstracting away the moving parts of the operating system, platform-as-a-service offerings lock out many inventors from improving and innovating—except perhaps, for on top of the platform, which they have no control over.

On the shoulders of standards

Standards and open source are both the safeguards of freedom and the catalyst to innovation. In the mobile world, having Android be open source is already proving beneficial for diversity and innovation.

We’re living in a world built on the innovation made possible by open standards and open source. The standardized IBM PC gave birth to many new businesses and technologies, the most influential among these being Windows and Linux. On the internet, open standards have been crucial. Web frameworks such as Django and Ruby on Rails just wouldn’t have happened without the web and Linux.

In order to preserve a strong software future for web development, we need to take care that our cloud infrastructure is open. Projects such as OpenStack are leading the way in this.

The bargain of convenience

How does this tie into offerings such as Google’s AppEngine? On the one hand, AppEngine is marvelously progressive. Why bother with the troubles of virtualized physical machines, load balancing and so on, when the platform can do it for you? But on the other hand, we’re always trading off freedom for convenience.

That’s not, on the face of it, a bad thing, except when it turns out to be a bad trade—and you don’t always know when that’ll be. If you don’t control the platform on top of which you’re innovating, you’re at the mercy of its owners changing the rules of the game without reference to you.

It’s in the long-term interests of the industry to keep all the layers of the stack open, so we can continue to innovate on them and try different approaches.

Even if your cloud provider is the smartest company in the world with the best record on uptime, security and redundancy, it’s not always enough. Control, or even merely the illusion of it, matters. One reason that EC2 is a wildly popular cloud hosting platform is because of the control developers get—even if the resulting architectures end up all looking much the same.

Open source is our guarantee

When developing services that run in the cloud, you really do need to be careful not just about technical issues, but about those of direction and control. The IBM PC was changed slowly and relatively predictably. There aren’t the same restrictions or guarantees for software platforms.

You may be thinking “so, you’re saying I should make my life less convenient so as to protect the interests of the following generation?” However great this sounds, we know from life this doesn’t happen very often, and exceeding rarely in business.

Fortunately, competition comes to our aid. We’ve seen repeatedly that open source is a powerful weapon in the hands of a competitor. If you commoditize your opponent’s platform, they have nowhere to go.

The advantages of open source in giving developers control, fostering innovation and seeding ecosystems mean that it is here to stay. There will be several open source Google AppEngines in the future, and who knows, perhaps even from Google themselves.

So for now, recognize that open source is far from obsoleted by the cloud. It’s more important than ever. As long as there’s open source, there’s a place for innovation in our industry.

Yak shaving

edd@me.com (Edd Dumbill) — Wed, 18 Jan 2012 16:13:00 -0800

If your brain works the way mine does, you often find yourself doing a variety of related but not necessary tasks along the way to fulfilling a larger goal. For some reason, the history of which I’m not entirely sure, this is called “yak shaving.” Some of my finest work has been done that way, so I’m happy to embrace it, even if I do add hours to the job at hand.

This week’s yak shaving has neared epic proportions, but proved eminently satisfying. I’ve been working on collecting together an overview of big data products. It’s early days yet for Hadoop centered offerings, but 2012 will be the year that Microsoft and Oracle will join IBM and EMC Greenplum in having launched products in the marketplace.

Surveying who-offers-what is a bit like writing code. I naively started off with a spreadsheet, but I quickly realized that real life data doesn’t fit into grids. Even if you can coerce it that way, the effort required to get there is massive. Instead I wanted a structured description of the products that I could amend and refactor as the whole landscape came into perspective.

To get this done, I revived an old interest. Some years ago I contributed a little to Dave Beckett’s Redland RDF project, and so I returned to this technology. RDF is simply a machine readable way of writing descriptions about things.

The task at hand was to collect knowledge about each vendor and their products, and have this in a format that could be used to create feature tables and comparison charts.

In my little project, each vendor has a file in which I’ve written a machine-readable description of their products. Here’s a little excerpt from my description of Cloudera:

<http://www.cloudera.com/hadoop/>
    a :HadoopDistribution ;
    dc:title "Cloudera's Distribution including Apache Hadoop" ;
    :homePage <http://www.cloudera.com/hadoop/> .

You can probably figure out this is the beginning of a description of Cloudera’s Hadoop Distribution. This is in the Turtle dialect of RDF, which makes the whole thing a lot more readable. I’m not an RDF purist, and don’t have plans yet to publish this data publicly. I was simply using the technology as a shorthand for writing down a large amount of semi-structured information.

After collating all the data together into a Redland database, I was able to write a small amount of Python code that queried the data, fed it into some templates, and spat out some chunks of HTML for inclusion into my article. For the curious, I used a combination of SPARQL and the regular Redland APIs to query the data, and chose the jinja2 templating engine to create my HTML. Excerpted screenshots are attached to this post.

Yes, I could have chosen XML & XSLT for this, or JSON, or, or… but that’s not really the point. The point was to have fun and learn stuff at the same time as achieving my goals. I brushed up on my Python, got some SPARQL experience in, and have content that can be easily kept up to date and repurposed.

Oh, and if you’re going anywhere near RDF technology, I can recommend Redland as a pragmatic and straightforward toolkit.

Look out for the Hadoop survey to be published tomorrow!

Writing style in ClojureScript

edd@me.com (Edd Dumbill) — Sat, 07 Jan 2012 11:54:00 -0800

Matt Might’s shell scripts for checking writing style are simple and effective, as is Benjamin Beckwith’s port of them to an Emacs mode. They pick up on some common defects in technical writing, such as use of the passive voice, weasel words and lexical illusions (inadvertent repetition of words.)

As a fun exercise, I decided to port this code to a variety of languages. To start with, I chose ClojureScript. I’d ultimately like to get to the point where I could create a bookmarklet or browser extension for checking my style when writing prose straight into the web browser.

My project’s called Wascal, and it’s the perfect way to eliminate those wascally weasel words, passive voice and duplicate words from your writing. Check out the code here on GitHub.

Here’s a screenshot of the Wascal in action. Below that I’ll present several observations about ClojureScript, discovered while writing the code

A few observations

ClojureScript’s regular expression support is that of JavaScript’s, as opposed to the Java regular expressions used by Clojure. This is fine, but you’ll need to hunt down a decent reference to them on the web—which turns out to be hard. Most material with a high Google ranking omits to mention the use of \1, \2 etc for back-references in matching expressions, and also that, in substitution expressions, $& is the back-reference for the whole matched expression.

ClojureScript’s default library for manipulating the DOM comes from Google Closure. While I love the minification and dependency handling parts of Closure, it does not present a nice interface for manipulating the DOM. Fortunately a variety of alternatives exist, none of which I’ve tried yet. The most promising to me seems to be Chris Granger’s pinot, wonderfully named as a complement to his noir Clojure web framework.

JavaScript interoperability is one of ClojureScript’s incredibly strong points, but will also always be a source of confusion for the beginner. For instance, in order to call the JavaScript e.preventDefault() I had to write (. e (preventDefault)), as opposed to (.preventDefault e). The latter returns the preventDefault method itself, whereas the former invokes it.

Hands-on with Dart

edd@me.com (Edd Dumbill) — Sat, 31 Dec 2011 11:39:00 -0800

Dart is a new programming language for the web, developed by Google. It’s intended to bring more structure to programming for the browser.

To get a taste of Dart, I re-implemented my ClojureScript coin-tossing experiment in the language, learning as I went along. The source code is on GitHub, take a quick look at it before reading my comments.

Pre-alpha geekery

Before you read on, it’s worth noting that Dart and its associated tools are not even in alpha release. They’re changing all the time and bugs are an expected part of the experience. The point of evaluating it now is to get a glimpse of the future, not to see whether Dart’s something to use in your next production system.

Dart, as currently experienced, compiles to JavaScript. The ultimate idea is that there’ll be a Dart interpreter in your browser, which will run Dart programs natively. The Dart team don’t foresee a Dart virtual machine, much in the same way that JavaScript doesn’t have a VM. Seen in one light, Dart and JavaScript interpreters are VMs themselves—this is already in evidence with languages like ClojureScript and CoffeeScript compiling down to JavaScript.

IDE from the get-go

Though it’s possible to use a command-line compiler, the Dart project pushes you strongly in the direction of using the Dart Editor. The editor itself is a specialized version of Eclipse. Despite not being an habitual IDE user, I had little trouble adjusting to the Dart Editor.

The most valuable feature of using the IDE is the incremental compilation, which enables you to catch errors early on, especially if you use static types. Another advantage of using types is that it brings the IDE autocomplete into play.

Static and dynamic types

Dart solves the issue of whether to use static or dynamic types by letting you do both. I am having a tough time deciding whether this is the smartest thing ever, or too much power for mere mortals to wield. For example, I can declare a variable either as simply a var or as an int and work with it in the same way. Here are my initial thoughts.

Having types is great. It lets me find more errors at compile-time. It reminds me of how productive I felt when I first started using C#. In fact, Dart has as much affinity with C# than as with Java in my mind.
Not having types is great. There are moments when a programmer finds themselves putting 80% of their effort into dealing with artificial class hierarchies than actually doing work. I can duck the question of which class or generic to use and get on with writing code.
This could get messy. I hope those Dart guys know what they’re doing!

The more you can use types, the more errors you’ll catch before execution. Here’s a fuller explanation of Dart’s optional type system.

Generics

I’m old and grumpy enough not to like generics very much. They raise a lot of questions and make programmers do more work than they need to. If you’re unlucky in which examples of Dart code you encounter first, you might run away screaming because of the similarity to Java generics.

However, it’s not as bad as it looks. Dart’s optional typing manages a compromise that, though I don’t yet truly understand how it works, lets me apply as much strong typing as I need to avoid errors, without making me dance in circles trying to keep up with it.

I’ll conclude my desperately unscientific analysis of Dart types by saying that they’re doing something clever that makes typing feel more natural and gets less in the way, but because I don’t understand it yet, I worry that it might explode in my face one day.

Classes and Interfaces

Dart isn’t prototype-based, like JavaScript. Instead, compound behaviors are achieved through classes and interfaces. Much like Java, an interface defines a contract that gets implemented by an actual class.

This is probably the most confusing thing about Dart right now. For example, List is an interface. But you can use it directly, because it has a default implementation, Array. However, that’s really not clear to any newcomer just reading source that throws lists around.

I was completely stumped as to how I’d go about adding behavior to a List. As I wrote the coin-tossing example, I wanted min() and max() functions for my list items. I could extend the List interface OK, but then I had absolutely no idea how to go about providing a default implementation for these new methods. In the end I plumped for writing my functions in a Util class.

Mixins, or something similar, would be an ideal way to solve these kind of issues. Apparently they’re under consideration.

As a side note, my being forced to use an auxiliary utility class mightn’t be a bad thing. This is one of the most pernicious things about object oriented programming, you get caught up trying to fit your code into its arcane strictures rather than getting things done.

The two most valuable things Dart could add to List, or its parent Collection, would be map() and reduce() functions.

This ain’t JavaScript

While Dart seems really familiar to a JavaScript developer, there’s quite a lot of peace of mind to be had from knowing you don’t have to worry about the crazy bits of JavaScript semantics.

One of the most-needed holes in JavaScript that Dart fills is string interpolation, e.g. “Hello ${name}” does just what any Perl or PHP programmer would expect it to.

Dart also involves a whole lot less boilerplate. Rather than

function() { }

for a closure, you can write

() { }

And one-line function definitions have a shorthand with =>, for example:

bool toss() => Math.random() < 0.5;

Though a little cryptic at first, there’s a great shortcut for constructors that do little more than initialize member variables.

Foo(this.bar, this.baz);

Interacting with the page

Dart are working on a browser API that’s more abstracted than the browser DOM, in much the same way that jQuery is. They’ve sensibly adopted the jQuery selector syntax, and it was easy to figure out how to add event handlers, e.g.

document.query("#mybutton").on.click.add((e) => print('hi!'));

Miscellaneous observations

Dart is hard to search for in Google right now, being a generic term. I found adding the word “language” to my searches did the right thing in most cases.

The API docs badly need a search. Google’s Seth Ladd says it’s coming.

It was frustrating not to have an interactive Dart console to play around with stuff. Dart’s going to need either this or a debugger as part of its 1.0 release.

Dart vs ClojureScript

As I’ve written this program in both Dart and ClojureScript, there are ways in which I find them contrasting.

ClojureScript’s optimized code is much smaller than Dart’s.
Dart’s browser API libraries being part of the core feels more natural than ClojureScript’s delegation to Google Closure.
In ClojureScript I found it easier to focus on the problem I was trying to solve rather than the mechanics of object-orientation.
In Dart, I really miss having a REPL.
Dart’s IDE makes it easier to get started.

Conclusion

Dart looks promising. It feels web-native, and manages to bring in the advantages of modular organization and static typing without sacrificing the flexibility of a dynamic language. Dart is fun to write, which is for me an important test of a programming language.

As a pre-alpha technology, there’s a lot of flux and change in Dart right now. With Google’s backing, I give it a good chance of being a key part of web development in a few years’ time. If you want to have a say in Dart, or be a part of it, it’s a good time to start splashing around.

Check out the Dart source code for my coin-toss experiment.

For more Dart news, follow these folk on Google+: Seth Ladd, Bob Nystrom, Dart Bits.

A busy person's introduction to Clojure

edd@me.com (Edd Dumbill) — Wed, 21 Dec 2011 17:37:00 -0800

Have a few hours spare and fancy getting to know Clojure and ClojureScript?

Sometimes all you need to get learning a new programming language is the right size problem. Nothing too trivial or too daunting. Fortunately, while reading NN Taleb’s book “The Black Swan”, I got a right-sized idea.

Taleb describes a coin-tossing game you can play to derive a Gaussian distribution for yourself. If the coin lands heads, you win $1. If tails, you lose $1. After a suitable number of tosses, record your net winnings. Repeat this again some number of times. Plot the frequency of each final amount and you’ll find the familiar bell curve. (You can get a similar curve just by recording the number of heads, but it’s not as much fun without the highs and lows of gambling!)

I decided to implement a simulation of this game in Clojure. And beyond that, to package it up in a way that other people can use, and then to use it as an example of how to create a ClojureScript application that runs in the browser.

The coin toss demo running in ClojureScript

Buckle up, the whole thing will take a few hours, but by the end you’ll have an idea of what a Clojure program looks like, and how they’re run and compiled. If you want to skip to the end result right now, check out the coin-tosser source on GitHub.

Install Clojure

I’m not going to reproduce installation instructions here, but you can find them on the Clojure Wiki. You need Clojure and Leiningen installed. Leiningen, often referred to as “lein”, is an excellent build tool and dependency manager that most Clojure programmers use.

This is a conversation

You really can’t learn a new language without trying to converse in it. We don’t figure out what a word means just because we’ve seen a definition, but because we’ve heard or seen it used in several contexts and can deduce a meaning and appropriate usage from that experience.

The same goes for programming. As you read I’m guessing you’ll be doing the same things I was when I was writing: Googling for information, trying code samples out. I’ve tried to write this so it’ll work as a standalone article, but you’ll have much more fun if you use it as a source of starting points.

Tackling the problem

I’m just a beginner: this article is simply the story of how I went about modeling the coin toss game, mixed in with things I’ve learned by reading about Clojure for a few months. I wrote most of the code interactively in the Clojure REPL, that is the interactive immediate execution mode of running Clojure. You can play with one of these online at TryClojure, or fire one up yourself once Clojure is installed.

I began by modeling the fundamental action in the game, the coin toss. To keep it simple, always a virtue, I decided to craft a function that returned true for heads and false for tails. The expression (= 1 (rand-int 2)) does just that. Here’s the function:

(defn coin-toss []
  (= 1 (rand-int 2)))

This expression defines a function, with defn, that has no arguments, hence the empty parameter list []. Run it repeatedly to convince yourself it works.

user=> (coin-toss)
false
user=> (coin-toss)
false
user=> (coin-toss)
true

You can experiment with unfair coins by altering the definition of coin-toss to make a heads result more or less probable.

Settling scores

Now we’ve got a coin, how much do we win in each round? The rules of the game say we win a dollar for heads, and lose a dollar for tails.

(defn toss-score [toss]
  (if toss 1 -1))

The result of the coin toss is passed as a parameter, and the value of the function is the winnings. Clojure’s if expression simply takes the first value if the passed expression is truthy, or the second value if it’s falsey.

Truthy? Falsey? As with many programming languages, values other than the Boolean true or false can be used in a boolean context in Clojure. To put it simply false and nil are falsey, and everything else is truthy.

Let’s check we get the expected results.

user=> (toss-score true)
1
user=> (toss-score false)
-1
user=> (toss-score (coin-toss))
-1
user=> (toss-score (coin-toss))
1

Repeat offender

Unregenerate gamblers that we are, we’re going to be tossing that coin more than once. Clojure’s repeatedly is the ideal way.

user=> (repeatedly 5 coin-toss)
(true true false false true)

What we get back is a list of results. It’s also a seq. A variety of data structures in Clojures are seqs, such as vectors, lists and hashmaps. It’s a basic interface for sequence types.

Of course, we don’t care as much about the results as how much money we’re getting from them. So, let’s count our net winnings.

The net winnings are the total of each individual winning. First, we want to calculate those individual winnings by applying our toss-score to the result. Try this:

user=> (map toss-score [true false true])
(1 -1 1)

The map function applies another function to a collection repeatedly, returning a sequence of the results. Now, we need to total them with our friend reduce.

user=> (reduce + [1 -1 1])
1

You’ve probably figured out that reduce applies a two-argument function to a collection in order, taking the result of the previous calculation as one of the arguments along with the next item in sequence. In the example above, it’s performing the calculation (+ (+ 1 -1) 1).

A little digression on syntax: you’re probably also wondering why I’m using square brackets, Clojure’s notation for vectors, instead of parentheses above. The short answer is that parentheses tell Clojure to treat the first item in the list as a function and the rest as arguments. Try (reduce + (1 -1 1)) and you’ll get an error message. If you want to use a list, write it as (list 1 2 3) or in the quoted form ‘(1 2 3).

Let’s put this all together and formulate our function for scoring a run of coin tosses.

(defn count-winnings [results]
  (reduce + (map toss-score results)))

It’s easy enough to test out and get some sample scores.

user=> (count-winnings (repeatedly 5 coin-toss))
-3

Finally, it’s going to be a chore to write all those repeatedly expressions out, so we’ll define a function that says what we actually mean: find the winnings from an experiment with a specific sample size.

(defn sample-score [n]
  (count-winnings (repeatedly n coin-toss)))

Creating all these small functions might not be strictly necessary, but they provide a clarity to the program. I prefer to be working in terms of what the code means rather than what it does.

Run and run and run

We’ve now got enough machinery to run a coin-toss game. In order to gather results, we’d like to run the game repeatedly. Again, it’s about as simple as writing down what we mean.

(defn sample-scores [n m]
  (repeatedly n #(sample-score m)))

What’s with the #()? It’s Clojure shorthand for creating an anonymous function. We could have written the following instead.

(repeatedly n (fn [] (sample-score m)))

Let’s run 20 games of 100 tosses each.

user=> (sample-scores 20 100)
(8 18 -10 2 -6 14 2 0 10 -6 20 0 -4 6 -12 0 2 -12 -6 10)

The final part of the exercise is to count up how many times we get specific results, and chart them. We’ll use a hash-map to store this frequency table. Let’s say we got $0 six times, $-2 twice and $2 four times, we’d end up with a map like this:

{0 6, -2 2, 2 4}

We’re going to look through the scores and do something with each one. As this isn’t just a straight transformation of the list, it’s a clue that we could use reduce as part of our solution. We’re going to start off with an empty map {}, and end up with one full of frequency counts. The function we’ll pass to reduce should take the map as its first argument, along with a score, and then return an updated map. Let’s imagine what this might look like.

user=> (reduce tally-score {} [8 18 -10 2 -6 14 2 0 10 -6 20 0 -4 6 -12 0 2 -12 -6 10])
{0 3, 2 3, -4 1, -6 3, 6 1, 8 1, -10 1, 10 2, -12 2, 14 1, 18 1, 20 1}

Now we just need to figure out the mechanics of tally-score. There are two scenarios to handle for each score.

If the results map has a key for the score we’re processing, we must increment the corresponding value by 1.
If the result map has no key for the score, we need to add the key with the value set to 1.

These scenarios can be handled with the following code, assuming results contains the results map, and score is the current score we’re processing.

(if-let [prev-count (results score)]
  (assoc results score (inc prev-count))
  (assoc results score 1)))

Here’s a chance to use a few new Clojure features:

(results score) looks up the key score in the hash-map results. In Clojure, maps are functions. You pass the key, and you get the value. For example, evaluating ({1 "one", 2 "two"} 1) yields "one".
if-let performs the bindings in the square brackets, and if they’re all true evaluates the first expression, else the second.
assoc returns an updated map with the key-value pair passed as arguments

I typically define these little fragments of code in the REPL and then put them together into a function when I’ve figured out how it should work.

Tally ho!

Putting together the above tallying process, we can create a function that took a seq of scores and returned a map of frequencies.

(defn tally-scores [scores]
  (let
      [tally-score (fn [results score]
                     (if-let [prev-count (results score)]
                       (assoc results score (inc prev-count))
                       (assoc results score 1)))]
    (reduce tally-score {} scores)))

Because we don’t need tally-score anywhere outside of this function, I’ve declared it with let, which makes it visible only inside the scope of the let expression. This example also gives you a clue to what you may have suspected by now, that (defn foo []…) is shorthand for (def foo fn [] (…)). To make it even more concise, we could use letfn.

Let’s try a sample run.

user=> (tally-scores (sample-scores 100 20))
{0 15, -2 18, 2 17, -4 19, 4 8, -6 9, 6 5, -8 2, 8 3, 10 3, 12 1}

Hooray!

Off the charts

To make the bell curve apparent, we can use time-honored basic ASCII art. For each score, we can use a string of asterisks to represent its length. Let’s try with 10.

user=> (repeat 10 "*")
("*" "*" "*" "*" "*" "*" "*" "*" "*" "*")

Great, but that’s a seq of strings, not a string of 10 asterisks. Fortunately, we can use join. This function lives in the clojure.string namespace. Until now we’ve only be using functions from the core namespace. So, we’ll grab join.

user=> (use '[clojure.string :only (join)])
nil
user=> (join (repeat 10 "*"))
"**********"

We can use this in a function chart-line, which we’ll use to represent a line in our chart. Because we want to plot a continuous axis of the scores, we need to allow that our results tally map might have no value for certain scores, so chart-line needs to accommodate a nil.

(defn chart-line [tally val]
  (str val
       "\t"
       (if-let [c (tally val)]
         (join (repeat c "*")))))

Concatenating strings in Clojure is done by passing them as arguments to str. To finish, we’ll create the whole chart. For every score, we want a line of asterisks representing the frequency. This is a task for map. To create every chart line, we want something like the following.

(map chart-line (range lowest-value highest-value))

After a bit of tinkering to get a good-looking graph, this is what I came up with.

(defn chart-lines [scores]
  (let [tally (tally-scores scores)
        low (apply min (keys tally))
        high (apply max (keys tally))
        r (inc (max (Math/abs low) high))
        chart-line (fn [val]
                     (str val
                          "\t"
                          (if-let [c (tally val)]
                            (join (repeat c "*")))))]
    (map chart-line (range (- r) (inc r)))))

You can see I decided I wanted my graph with 0 in the middle, so did a bit of calculation to get there. The interesting construct that you’ve not seen before in this function is apply, which simply applies a function to a seq of arguments. In other words, these two are equivalent:

(apply min [1 2 3 4]) and (min 1 2 3 4)

You’re probably also concerned about Math/abs. Because Clojure is based on the JVM, we have all of Java at our disposal. Here we’re borrowing the abs function from java.lang.Math. Let’s not worry too much about that now, as this is the only Java we bring in.

Let’s run this, and join up each line with a carriage return to get something pretty.

user=> (println (join "\n" (chart-lines (sample-scores 50 10))))
-7	
-6	**
-5	
-4	*****
-3	
-2	************
-1	
0	**********
1	
2	*************
3	
4	*****
5	
6	***
7	
nil

Ta-da! The bell curve. Or something a bit like it. Mission accomplished. One final function to make this easier to run, and we can try as many sample sizes as we like.

(defn chart-scores [scores]
  (println (join "\n" (chart-lines scores))))

Tinker time

If you followed along and want some easy places to start fiddling, try the following exercises.

What if the coin was unfair, and returned tails twice as often as heads?
Would you get the same graph if you only counted the number of heads results?
Adjust the program to count only heads, and alter the charting range so it looks pretty again.

There are a couple of fundamental Clojure concepts that I’ve not really touched, and if you want to at least bluff your way as Clojure-credible, then you should learn about these.

Laziness, the idea of not evaluating expressions until they’re needed, and
tail recursion, the way you loop.

Get smart

Now we’ve got a bunch of code that works, let’s figure out how to make this into a program that anybody else can use. There are few things that help make look Clojure code smart enough to go out in public.

The first of these is namespaces. As hinted earlier when we looked at clojure.string/join, Clojure lets you organize functions by placing them in a namespace. This stops them stomping on top of other functions with the same name. Because I’m a Brit with a schoolboy sense of humor, I picked the namespace tosser.

Second, though Clojure doesn’t mandate a certain directory hierarchy in the same way Java does, there’s a convention most people adopt that helps things stay readable.

Check out the source code for the experiment we just developed. You’ll see a Leiningen project.clj file in the root, and then the source tucked away in a directory corresponding to the namespace I decided to use, tosser.

Third, in order to make our coin tossing system usable outside of the Clojure REPL, we need to write a “main” entry point that will take command-line arguments and run the experiment in response.

You can find the source in main.clj, but here’s the essence.

(defn -main
  "Main entry point"
  [repetitions sample-size]
  (chart-scores (sample-scores (Integer/parseInt repetitions)
                               (Integer/parseInt sample-size))))

Notice we’re using java.lang.Integer.parseInt to conver the string argument from the command line. Don’t forget that Clojure is a typed language, even though it’s dynamic and doesn’t require type declarations.

The declaration :main tosser.main in project.clj tells the compiler, via Leiningen, which namespace contains the main function of the program.

Once you’ve built the code with lein compile you can invoke it via lein run.

$ lein compile
Copying 1 file to /Users/edd/work/git/coin-toss/clojure/lib
Compiling tosser.main
Compilation succeeded.
$ lein run 50 10
-7	
-6	***
-5	
-4	*********
-3	
-2	**********
-1	
0	**************
1	
2	*********
3	
4	*****
5	
6	
7

If you want to understand what’s going on at the Java level here, you can also run the program like this:

$ java -cp ./classes:lib/clojure-1.3.0.jar tosser.main 50 10

If you want to hand your program to somebody else, regardless of whether they’ve installed Clojure, use lein uberjar.

$ lein uberjar
Cleaning up.
Copying 1 file to /Users/edd/work/git/coin-toss/clojure/lib
Created /Users/edd/work/git/coin-toss/clojure/tosser-1.0.0.jar
Including tosser-1.0.0.jar
Including clojure-1.3.0.jar
Created /Users/edd/work/git/coin-toss/clojure/tosser-1.0.0-standalone.jar

The program can now be run like this:

$ java -jar tosser-1.0.0-standalone.jar 200 20

If you’re going to check your code into source control, be sure to ignore the classes and lib directories that Leiningen generates.

And that’s about all there is to it. Welcome to computer programming, circa 1995. In the next section, we’ll move into the web age a little.

Getting webby with it: ClojureScript

ClojureScript is a Clojure-like language that compiles to JavaScript. I say “Clojure-like” because ClojureScript isn’t Clojure. It’s a lot more like Clojure than JavaScript is like Java, but they’re not the same.

However, it’s easy enough to port across our Clojure program to give ourselves a first taste of running ClojureScript. It also makes the coin-tossing experiment feel a lot more interactive!

If you browse through the source of the ClojureScript coin-toss experiment, you’ll find it much the same. However, there are a few things to notice.

Firstly, reinforcing the point that it’s a different language, the source code files have the suffix .cljs.

Second, I’ve marked functions that I want to be available to JavaScript with the ^:export notation.

(defn ^:export chart-lines
  "Given a list of scores, create a simple ASCII frequency graph"
  [scores] ... )

The ClojureScript build process uses JavaScript “minification” for compression. The export metadata tells the compiler to leave the name of that function alone, so it’s callable from outside the ClojureScript program.

Third, you might be quietly fretting about the small bit of Java we call, Math/abs. Obviously we’re not going to be calling Java from this program, so what gives? Instead, this is turned into a JavaScript invocation. Fortuitously, JavaScript also has a Math.abs function that does exactly the same thing. We should really write js/Math.abs to call the JavaScript object, but ClojureScript does “the right thing” for built-in JavaScript objects like Math and RegExp.

How do we drive this thing?

First, we need to compile the program. Thanks to our project.clj definition and the lein-clojurescript plugin, this is as simple as invoking lein compile.

ClojureScript compile to JavaScript. In order to run the program, we need to create an HTML page that will load the script and call it. This page looks something like this:



  Coin Toss Demo
    
    
    
    
    
  
  ...

The important bits have been highlighted in bold. I’m also using jQuery as part of the demo, but you can ignore that for now. The three script elements achieve the following: they load in the Google Closure JS library, load in the compiled ClojureScript program, and loads our program’s namespace plus any of its dependencies.

This is where it gets a bit silly. ClojureScript uses the Google Closure JavaScript library as a substrate. Yes, “Closure” is pronounced the same as “Clojure”. Just smile politely and move on.

The namespace tosser.main refers to that defined in main.cljs. Rather than have a “main” entry point as we did for the command-line version, I created a new file with the entry point I want for my web page version. It defines a function chart that we can call from JavaScript to get the ASCII chart.

(ns tosser.main
  (:require [tosser.core :as toss]
            [clojure.string :as string]))
(defn ^:export chart [m n]
  (string/join "\n" (toss/chart-lines (toss/sample-scores m n))))

ClojureScript doesn’t yet support Clojure’s use, so we import things we want from other namespaces as part of our ns declaration.

To run the coin-tossing experiment from JavaScript, it’s as simple as calling tosser.main.chart(). If we put the parameters into a form, as seen in the screenshot, we can use jQuery to call the experiment when the user hits the submit button, and then write the results back into the page.

Here’s what the jQuery code to do that looks like, with the invocation to our ClojureScript highlighted. Note that we needed to convert string input to integers. I could equally well have done that in ClojureScript in the definition of chart.

$(document).ready(function(){
    $("form").submit(function(){
	var tosses = parseInt($("#tosses").val());
	var runs = parseInt($("#runs").val());
	$("#results").text(tosser.main.chart(runs, tosses));
	return(false);
    });
});

For a peek at how the ClojureScript program appears to JavaScript, pop open the JavaScript console from your web browser and evaluate a few things. Try this, for example:

> tosser.main.chart(20,5)
"-4	
-3	**
-2	
-1	*******
0	
1	*********
2	
3	**
4	"

When it comes to putting your web page live, you obviously don’t want a million little JavaScript files. Instead, you can recompile with optimizations to end up with just one file, tosser.js. You then don’t have to load Google Closure or perform the goog.require instruction either, as in tosser.html.

$ lein compile '{:optimizations :simple}'
Compiling clojurescript in src ... compiled 2 files to out/ and 'tosser.js' (took 40979 ms)
$ ls -l tosser.js
-rw-r--r--  1 edd  staff  383078 Dec 21 16:13 tosser.js

However, 380k is hardly a small filesize for a simple program. Fortunately, we can specify :advanced for optimizations too, which also minifies the resulting JavaScript. That makes it pretty useless for debugging, but gets the filesize down to 55k. If you use advanced optimization and you want to use jQuery, you’ll need to remember to use jQuery.noConflict() for things to still work.

So, that’s a bit messy!

There’s not actually any reason that I should have complicated my demonstration by using jQuery, except for that I’ve spent the last three years exclusively using jQuery for my JavaScript and it felt a comfortable place to be.

In fact, ClojureScript has enough in it for you to do all the things that jQuery can do using just ClosureScript and the Google Closure libraries. In that case, you may want to keep the Closure Library API Documentation to hand.

In keeping with the rather fluid nature of JavaScript, there’s also a third way. Instead of keeping your jQuery with your HTML files, you can call jQuery directly from your ClojureScript via the js namespace. For example, the jQuery statement

$("body").append("hello
");

could be rendered like this in ClojureScript

(.append (js/jQuery "body") "hello
")

More idiomatically, we can use the -> (thread macro) from Clojure to chain functions in the same way jQuery does.

(-> (js/jQuery "body")
    (.append "hello")
    (.css "background-color" "red"))

So, pick your poison. For a new application, I’m inclined to try and stick with Google Closure and ClojureScript throughout, but jQuery has many riches that it would be a shame to ignore.

In conclusion

I hope you’ve enjoyed the high-speed romp through Clojure and ClojureScript. I find Clojure to be an enjoyable and natural way of programming.

There’s plenty more to explore, but if you’ve followed along with this project you’ll have a decent grounding. Feel free to use the comments on this article to send questions, bug reports and improvements.

Finally, I recommend picking up a copy of The Joy of Clojure as well. It’s a good read, quite often challenging, but written by people who really enjoy the language.

All the source code for programs developed in this article is available on GitHub.

The key to being a data-driven organization

edd@me.com (Edd Dumbill) — Thu, 15 Dec 2011 17:41:00 -0800

Let’s talk about being a data-driven organization. Becoming data-driven requires systemic change, where the data science is part of the feedback and decision loop of a business.

However, many fewer people than we’d hope know how to use and understand data. Even simple concepts such as the difference between mean and median, or causation and correlation, can be hard to convey.

So, maybe we should teach everyone about numbers? It worked for Google.

(photo credit: ansik on flickr)

Numeracy’s not enough

Even if it were possible, making everyone speak the language of numbers is not a recipe for success. Financial crashes tell us numeracy alone won’t save us. The wonderful thing about numbers is that we can use them to back up whatever plan we already decided to embark upon.

It’s still the case that to achieve a data-driven organization, one where data science drives real product and business benefit, the organizational culture needs to be more quantitatively oriented. But we cannot pursue a “you must go away and understand numbers, then we can talk” approach. Unless rigorously enforced from the top-down, that’s not going to fly—nor is it sufficient.

Something more collaborative is required, allowing communication across disciplinary boundaries. I’ve found that in software, the best way of having others understand what can be done is to have them work alongside you. You get domain expertise, and they get a feel for the medium.

Collaboration and exploration

A practical and feasible way to achieve a goal of organizational numeracy is to create social, collaborative and exploratory tools and processes, which people can use to get the fingertip feel for the data without having too learn much ahead of time. This is advantageous in two main ways:

social and collaborative systems facilitate the two-way flow of domain and business expertise and feel for the data
exploratory systems are able to constrain the tool to reduce the need for numeric expertise. We can create systems that guide exploration. It’s important not just to give bland reports, but to engage business users with the data by permitting experimentation and interactivity

Batteries included—and humans are too

Though useful, the phrase “data-driven” is an overstatement—it implies humans aren’t required. The ideal situation is when the data science enables and augments human capability, and there’s a collaboration between the business owners and the number crunchers.

The data-literate need not be frustrated that business doesn’t understand their language, but instead build directed tools to allow business to engage in their world. In turn, they will understand more about the business and be better able to meet its needs.

Big data and data science: it's about the who and the how

edd@me.com (Edd Dumbill) — Tue, 13 Dec 2011 10:48:00 -0800

(This post first published on Google+).

Two areas that I work in, “big data” and “data science”, both have somewhat nebulous definitions. As with many technology trends, they are neither wholly novel or well-defined. Many business analysts or statistical programmers will lay claim to doing what is characterized as data science, and managers of large BI (business intelligence) deployments will assert they’re handling big data.

What is the point of such labels?

The utility of the terms big data and data science does not lie in their watertight definition. Though many, including me, seek to both educate others and profit by defining labels, a clear ontology of a developing phenomenon is incorrect as soon as you publish it.

Beware the vaunted four Vs of big data (volume, velocity, variety and variability) for they will have little practical use outside of a consultant’s white paper. You do not buy a cow by specifying that you wish for a bovine ungulate that consumes monocotyledonous herbaceous plants. You want something that will produce milk and that will live on your farm.

Instead, the value of the terms is in highlighting a change in the who and how, a migration of technologies and practice into broader applicability.

In the case of big data, the web and social software mean that even modest businesses have large and expanding datasets on their hands. This was the sort of problem that only larger retailers or search engines found it profitable to face previously. Now—the change in “who”—anybody can use technology to process these large and messy datasets. This is thanks to commodity hardware, the open sourcing of projects like Hadoop, and on-demand cloud computing resource—the change in “how”.

For data science, the chance in “who” is in the spread of exploratory and entrepreneurial analysis, historically performed for example by physicists or investment industry quants, into a business and product context. Data scientists (or data science teams) embody domain knowledge, programming ability and mathematical aptitude. Rather than sectioning them behind the IT firewall, they are brought into the conversation of the business as a whole—a change in “how”.

Feeding on themselves

Another interesting aspect of both big data and data science is that, once started, they exhibit positive self-reinforcement. Organizations find that once Hadoop, for example, is in an organization it can be used in an increasing number of contexts. Such scalable computation is enabling many mobile and sensor applications that would previously have been impractical.

Likewise, data science as a phenomenon is self-reinforcing. Its existence and value within companies is causing its recognition not just as a compound skillset and job title, but also as a leading edge of the transformation of corporate culture. To be driven by data necessitates much cultural change: successful data science teams provide an example and learning ground for effecting such changes.

Craze 2.0

Inevitably, the terms big data and data science will be used by anybody wanting to jump on the bandwagon, for there is considerable financial value in these trends. Cloud, social software, web 2.0 all suffered a similar fate.

But behind the labels lie important changes in the way that we think about both computing and its place within organizations. Just don’t let anybody tell you they know exactly what big data or data science means!

===

And to conclude, always be prepared for the data to show you that you’re not in the business you thought you were.

http://xkcd.com/539/

"The first problem for all of us, men and women, is not to learn, but to unlearn"

edd@me.com (Edd Dumbill) — Sat, 17 Sep 2011 15:59:00 -0700

“The first problem for all of us, men and women, is not to learn, but to unlearn”

- Gloria Steinem

Webapp review: Checkvist outliner and checklist editor

edd@me.com (Edd Dumbill) — Sun, 04 Sep 2011 23:32:00 -0700

Checkvist is an online outliner, similar to applications such as OmniOutliner on the Mac.

Fundamentally, you can use it to create hierarchical outlines. Checkvist has a lot of extra bells and whistles that facilitate its use for basic project management. Items can be assigned to other users, given due dates, and have annotations from members of your team. You can also tag an item with a hashtag.

None of those fetaures get in the way of straightforward outlining, however. Plentiful keyboard shortcuts mean you can use Checkvist to quickly set down your thoughts without having to pause to hunt and click with the mouse. The use of shortcuts is so extensive in fact that users of the vi editor will feel quite at home!

Crucially for a web app, Checkvist speaks the standard file format of its genre, in this case OPML. You can both import and export OPML. Other import formats include plain text and Confluence Wiki syntax. Those formats are supported for export, along with simple HTML.

Printing is also well supported through an HTML view that gives you the options of hiding or showing the checkboxes and notes.

Most of the functionality is available for free, but you need a Pro account to unlock the groupware features and advanced tag management. Along with that, Pro gives you HTTPS access, backups and UI customization. The Pro subscription is $15 for 6 months.

Checkvist’s UI is clean and functional, though a little ugly to my eye. A Pro subscription lets you tweak the UI some, or you could install a browser extension similar to Stylebot and do it yourself.

The low-down

Checkvist is a capable outliner program. You may or may not want to use it for project management in addition, but it’s worth a look as a entry-level groupware. If you’re just a casual maker of lists, then look no further.

Checkvist is also available in the Chrome web store.

Two business book reviews: Blueprint to a Billion and the Five Dysfunctions of a Team

edd@me.com (Edd Dumbill) — Sun, 04 Sep 2011 18:03:00 -0700

I’ve been able to do a fair bit of reading this holiday weekend. I posted a couple of mini-book reviews up on my Google+ feed. Click through to the reviews below.

Blueprint to a Billion — by observation of successful companies, David Thomson has distilled seven practices that all of them had in common. Great reminder of good business sense.

The Five Dysfunctions of a Team — through the telling of a story, Patrick Lencioni illustrates the ways in which teams need to function to all work together to a common goal.

Picking business books can be hard, there are so many, but both of these are easy to read and will leave you with some principles to learn from.

Software development completely in the cloud

edd@me.com (Edd Dumbill) — Sat, 03 Sep 2011 18:57:00 -0700

We’ve passed an important landmark in software development. Using only a browser, it’s now possible to develop, debug and deploy software in the cloud.

To prove it, I spent a few hours today teaching myself how. Using just my Chromebook I wrote, tested and deployed a simple Node.js program using Cloud 9 IDE, Heroku and github.

Cloud 9, based in Amsterdam and San Francisco, just received their Series A funding and are growing fast. Their product is maturing quickly: the Heroku support was just added, for example. Cloud 9 is available as both a hosted service and an open source project.

If you wish to use Cloud9’s debugger, then at the moment you must restrict yourself to JavaScript applications using Node.js. The IDE supports host of other languages though, including Ruby, PHP and Python.

I was able to set up a one-click deploy to Heroku. You can also deploy to Joyent’s servers.

To find out more about Cloud 9, read this excellent article about Why Cloud 9 Deserves Your Attention.

Below is a screenshot of Cloud 9 running on my Chromebook. It’s a little rough around the edges yet: documentation is a little lacking and at times it got confused making new files.

This is seriously impressive work, though. I wouldn’t be surprised to see an acquisition by one of the larger cloud companies soon: Salesforce, Joyent or VMware seem obvious candidates.

The Cloud9 web IDE running on my Chromebook

edd@me.com (Edd Dumbill) — Sat, 03 Sep 2011 18:37:12 -0700

The Cloud9 web IDE running on my Chromebook

Are You A Victim Of Phantom Vibration Syndrome?

edd@me.com (Edd Dumbill) — Sun, 07 Aug 2011 22:29:41 -0700

Are You A Victim Of Phantom Vibration Syndrome?:

Apart from habit, Phantom Vibration Syndrome is also about not being fully present. As much as we all believe we’re skilled multi-taskers, for the record, we’re not. Quite simply, we’re no longer fully present. By this I mean emotionally as opposed to physically. We think we are, we think we’re participating in the conversation, but in reality, we’re not.

AwayFind

edd@me.com (Edd Dumbill) — Sun, 07 Aug 2011 16:56:14 -0700

AwayFind:

Interesting service

Let us find your urgent messages. When you receive an urgent message, AwayFind will notify you via your mobile device, a voice call, IM, or even delegate the message to someone you specify.

"Just as cats have evolved to imitate the cries of newborn babies to insinuate their way into our..."

edd@me.com (Edd Dumbill) — Sun, 17 Jul 2011 09:14:57 -0700

“Just as cats have evolved to imitate the cries of newborn babies to insinuate their way into our homes, so corporations have learned to affect the trappings of personhood to reach deeper into our social lives and conversations”

- A comment from Matt Edgar on a recent Google+ post of mine

Being a data-driven person

edd@me.com (Edd Dumbill) — Mon, 04 Jul 2011 17:49:18 -0700

I’m intrigued by the idea of being data-driven. Even more so by the frequent times that I fail to achieve it.

If you read Steven Levy’s excellent history of Google, In the Plex, you get the idea that few decisions are made in Google without data to back it up. As a geek, that had me shouting “Yes, I want that!” — here is a company that has apparently made good by the relentless application of analysis, combined with a clear basic mission.

We are frequently exasperated by the decisions of others that defy what seems clear evidence: by governments, companies, our family members. As a nation and as a race, we remain stubborn in the face of stark medical research that smoking and drinking are harmful to us, that obesity places a burden on the national purse.

Being data-driven sounds practical in a geek utopia like Google, but in real life even the most obvious of discoveries seems to have a minimal impact on our behavior. Public opinion and subjectivity can and do trump the evidence.

All is not lost. The situation isn’t binary. Most people know smoking could kill them and steer clear. Data and its interpretation can help push the needle toward making better decisions, especially in closed systems such as companies.

I will leave a discussion of becoming a data-driven company for another time. It’s going to form a big part of September’s Strata conference. And it’s probably more easily done in a company than in the unconstrained wilds of our personal existence. But what about us as people?

Can a person be data-driven?

As humans we have the impulse to learn, to know. We’re curious about ourselves, each other. We understand that knowing more can help us make better decisions. Looking at Google as a company, and how its products have contributed to our lives, we wonder whether we can ever be like that ourselves: finding our way through the distractions to achieving our goals.

For me as a person, I now have a large amount of data which I did not have before. Thanks to my Fitbit, I understand how active I’ve been, and I regularly track my weight. My inbox statistics, thanks to SaneBox, let me understand my backlog, and RescueTime lets me know what I’ve been doing on my computer.

Yet, with apologies to St. Paul, I still do not do the things I want to do, and I do the things I do not want to do.

Just knowing isn’t enough. The whole self-improvement book market is testament to this fact. We need support, external motivators and coaching to attain our goals. In the corporate world, we call such things “consultants”, whose real role is in overcoming internal inertia, not in pointing out anything new.

If knowing isn’t enough, it makes me wonder if we are truly happier and better off for knowing more. Ignorance is bliss, yet we are hard-wired with an instinct for discovery and self-analysis.

I’ll close with what I think is the “secret” to it all. Our lives are steered by a crooked and crazy rudder. We have control, but the currents and tides are strong. The combination of our goals and data give us a course, but we still have to practice and learn to steer.

By acclaiming and romanticizing our heroes, who make it all look so natural, we often miss the years of failure and practice it took them to achieve their goals.

In this sense, knowing more isn’t enough. Failing more, and so learning more, is.

“If a thing is worth doing, it is worth doing badly.”

— G.K. Chesterton

"If Google offered a WordPress comment integration, I would give G my comments in a heartbeat."

edd@me.com (Edd Dumbill) — Sun, 03 Jul 2011 08:01:46 -0700

“If Google offered a WordPress comment integration, I would give G my comments in a heartbeat.”

- +1 from me!

More from Chris Brogan in The Google Plus 50

R ecosystem continues to grow: CRdata is a web service for R

edd@me.com (Edd Dumbill) — Sun, 03 Jul 2011 07:49:43 -0700

R ecosystem continues to grow: CRdata is a web service for R: