Freebase 101 I’ll be giving an overview of the various parts of Freebase and how they fit together, from Freebase.com to MQL to our API, Acre, Gridworks, and more. If you’ve ever felt unsure what it’s all about, or just need a recap, this will help you put it in perspective. [NB: highly recommended if you're unclear on what this Freebase thing is all about or why you should care!]

Gathering human judgements with RABJ Shailesh Kochhar will give a brief primer on RABJ (Redundant Array of Brains in a Jar), which Freebase uses to gather human judgements and do data QA. If you’ve ever played one of our “data games” like Genderizer, you’re using RABJ. Soon, you’ll see even more RABJ in our public data tools.

Open source Acre and Freebase.com We’ve been working on making the Freebase.com website more open, and in this talk Michael Masouras will be talking about our latest steps in this direction, including open sourcing the Acre platform itself.

The talks run from 5:30-7pm then afterwards we’ll be adjourning to Gordon Biersch (right downstairs) for some beer and food and general Freebase/open data/etc chat.

If you’re interested in coming, you should RSVP at meetup.com. It’s important to let us know you’re coming, as we need to give a list of attendees to security so you can get into the building.

See you there!

This seems quite similar to the OTW’s Archive Of Our Own “Tag Wrangling” project for fannish metadata (fandoms, characters, etc).

Does anyone know of any other curated folksonomies out there? That is, a freeform tagging system with a layer of curation over the top to even out some of the stranger quirks of a purely anarchic system.

Yesterday Metaweb, my employer and the creators of Freebase, announced that we’ve been acquired by Google.

The announcement was pretty exciting, not least because I got to be the person to post the official blog post on the Freebase blog.

I’m also excited that we launched this video along with the announcement, explaining what Metaweb/Freebase is all about:

Transcript:
You know what drives me crazy about words? They have a million different meanings.

Like, check this out: someone says, “I love Boston.” Now, they probably mean, “I love Boston, the big city in Massachusetts”, but they could be referring to one of the twenty-six other Bostons that are scattered around the globe. But, if it’s during the playoffs, they’re probably referring to the Celtics [basketball team]. Of course, you and I both hope that they’re talking about the Boston. You know. [Image of rock band, sounds of electric guitar.]

But, I guess there’s really no way of knowing. The problem is that the same word can mean so many different things. Because of that, when it comes to finding, linking, reconciling, or organising multiple layers of information, words are not the best solution. The guys at grocery stores figured this out back in the sixties when they started putting barcodes on everything, so that products with the same name wouldn’t get confused.

So how come on the web, so many sites still try to organise stuff with words? Say you’re a product guy at a big music site and you want to pull in feeds of lyrics and videos and photos from all of your data suppliers. But everyone uses different names for things, and a lot of the feeds don’t even match up, so you’ve got to reconcile them, and pull in updates, and deal with merges and deletes and splits. It’s a nightmare.

But what if there was a better way?

Welcome to Metaweb. Metaweb is a service that helps you build your website around entities, and not just words. Whoa, what’s an entity? Well the simple answer is, it’s a singular person, place, or thing.

OK, well, let’s compare that to text. Did you know that on the web there are more than 50 different ways people write “U. C. Berkeley”? [Examples listed: Cal Berkeley, Berkeley University, UCB, California, U of Cal, etc.] And they’re really just talking about one single place, one entity. By mapping all those words to a single entity, as if it had its own barcode, you can combine all that information about U. C. Berkeley into one place.

But that’s just the beginning. Because entities represent unique, real-life things, we can build a map that shows how they’re related. So, you can look for things that share certain attributes, like “actresses under 20 from New York”. Can you imagine trying to find that with a keyword search? [Shows typical keyword search results, with keywords highlighted: "NY blogger under fire for criticizing actress", "March 3 2004: New! 20 steps to be an actress", "Kid actress eats 20 York peppermints".] Entities are just smarter than words.

So, Metaweb’s been in the process of identifying millions of these entities and mapping out how they’re related, and what words other sites use to refer to them. And it’s really cool because they have a totally collaborative process that involves the online community. This thing will always be expanding and improving.

So, how is this going to help you? Well let’s say you’re that guy writing the movie review. If you tag the review with an entity in Metaweb, it’s like you’re looking at a menu saying, “Hey, Metaweb, give me the movie poster and a trailer and some links and maybe some other information like the release date and who was in it.” And BAM, it’d be right there. And now, your page looks awesome!

Or, say you’re that product guy at the music site. Instead of spending months doing messy integrations and maintaining all those feeds, you can just plug in to Metaweb, and suddenly everything just works. It’s like a switchboard for content on the web. [Various logos related to web content: eg. Twitter, Facebook, Audio Scrobbler, WordPress.] And not only that! When your site’s built on entities, new things get magically connected. Like, if one of your users adds a band to her profile page, or tags them in a comment, that can show up on the band page, because they’re all linked under the hood to the same entity.

Are you kidding me? This stuff sounds impossible! Well, that’s what they said about the barcode.

And it’s not just movies and bands. Metaweb has millions of entities in thousands of categories: twelve million and counting!

Metaweb makes your site smarter. It’s time to connect to the web. Metaweb.com.

I think a lot of friends and family are finally going to be saying, “Oh, so that’s what you do” :)

The other good news with the announcement is that Freebase is going to be staying free and open, and we’ll be working with Google to make it bigger and better (and you know with Google, bigger means bigger). So that’s pretty exciting. I’ll be continuing on over there doing community/developer relations stuff.

I’ll also be at OSCON next week, where I’ll be giving a presentation on Open Source, Open Data where I talk about how we apply open source ideas and processes to open data. Come see my talk!

I decided I was going to update one of my CPAN modules, the woefully unmaintained Metaweb.pm. I set out to do this on my work machine: a Macbook running OS X 10.5.5 (Leopard), on which I’d never done any serious Perl development before.

I started by checking out my code from my own repository. I was sure I’d need some stuff installed from CPAN, so I started by attempting to run the Build.PL that comes with my own module. It told me it needed a pre-requisite library (JSON), so I ran cpan install JSON.

Several hundred lines of text scrolled by, and it told me it couldn’t install anything. I dug back a fair way through a lot of non-specific “make failed” before I found anything approximating a meaningful error message: “Perl v5.10.0 required–this is only v5.8.8″. Fair enough. I need to upgrade my Perl. How do I do that on OSX? Well, I’m not sure, but I heard that perl.org had recently had a highly-acclaimed makeover, so I decided to check there. It tells me that Perl is included in Mac OS X. Yes, thanks. And upgrades? I couldn’t find binaries anywhere via the site.

A google for “perl 5.10 on osx” led me to perl.com’s download page which tells me:

If you want to upgrade, you can build the Unix source if you have the developer tools installed.

Luckily I happen to know that “the developer tools” means XCode, which provides such tools as make.

Hang on… wait a minute… how come I got so far without even having make installed? Surely the CPAN install script should have bailed out if there was no make? In retrospect, I realised that I’d just pounded blithely on the enter key while CPAN was configuring, and no doubt had skipped past the bit that asked me where my make was. I deleted my ~/cpan/CPAN/MyConfig.pm and ran through it again. Yup, sure enough, there was a warning. A single line, plain text warning, that I didn’t notice as it scrolled by.

Now, perhaps I’m just being picky here, but I’d expect that if there’s no make, it should fail a bit more loudly than that. But whatever. I went to download XCode, noting as I did so that nowhere had any error message or webpage actually provided a URL for where to get that, or even mentioned the name of it. Just so you know, here’s the XCode for OSX 10.5 Leopard download (binary file).

It takes me about 15 minutes to download the almost 1GB XCode via work’s network connection, but had I been at home, or worse yet back in Australia, it might have taken me hours. (It also comes on the Leopard install disks, but I don’t have them handy.) It was long enough for me to write this blog post up to this point, and if I hadn’t been fairly dedicated to getting the job done by now, I probably would have given up by now.

Another 20 minutes to install the package (during which I bitched to friends on IM), and I have make. I run cpan install JSON again and everything succeeds. There are 1,988 lines of output, of which many hundreds are Date::Manip’s file listing. Why do I need Date::Manip? I don’t even know, but that doesn’t stop it vomiting gibberish all over my screen.

Anyway, at long last I have the pre-req libraries I need to play with the simple, non-binary package I wrote two years ago. It took me a little over an hour in total.

Oh, and by the way, I didn’t need to upgrade to Perl 5.10 — I guess that was a red herring.

Recommendations

So, these are the things I think the maintainers of the CPAN module, and of various high-profile Perl websites, and of Perl itself, should be doing:

• Fail louder at configuration time if vital CPAN install tools aren’t available.
• Fail louder at module install time, ditto.
• Suppress about 90% of CPAN output unless a verbose flag is set, to make it easier to see warnings etc.
• Provide links to required developer tools (i.e. XCode, for OSX) in warnings, on webpages, etc.
• Provide a binary package of recent versions of Perl for recent versions of OSX.

Some of these I can do something about, some I can’t. I’ll go nag the right people after I post this, and see if I can get things moving, write some patches, or something. But why are we even in this situation? CPAN has got hairier and hairier over the last, say, five to seven years, to the point where it is a serious impediment to anyone starting Perl development. If you are an experienced developer, you probably don’t notice it anymore, but trust me: I’d been doing Perl for a decade and been publishing modules to CPAN for eight years before I took two years off, and I found this intensely frustrating. For new developers on anything other than a traditional(ish) Unix system on which they have root (and that includes most web hosting providers), CPAN — which was meant to simplify Perl development — is undoubtedly a major barrier to entry.

This needs fixing.