Google Refine is a power tool for working with messy data sets, including cleaning up inconsistencies, transforming them from one format into another, and extending them with new data from external web services or other databases. Version 2.0 introduces a new extensions architecture, a reconciliation framework for linking records to other databases (like Freebase), and a ton of new transformation commands and expressions.

Freebase Gridworks 1.0 has already been well received by the data journalism and open government data communities (you can read how the Chicago Tribune, ProPublica and data.gov.uk have used it) and we are very excited by what they and others will be able to do with this new release.

Check out the post and the screencasts on Youtube: one, two, three.

For those of you in the San Francisco Bay Area, David will be demoing Google Refine 2.0 at tonight’s San Francisco Freebase meetup. Be sure to RSVP if you’d like to attend!

This past Saturday I had the pleasure of presenting the Freebase API at the HackTO event in Toronto. For those unfamiliar with this type of event, its basically a day of intense hacking where programmers get together, learn new skills and try to build cool apps all in a single day. Freebase was one of 6 APIs which were presented in short 5 minute pitches after which the participants got 30 minutes to brainstorm ideas and form teams. Once the teams were formed the real learning began. People jumped in and started playing with the API and quickly discovered many of the strengths and weaknesses of the Freebase API. It was really cool to have people come up to me with an idea and see the look on their face when I showed them how to pull the data they needed out of Freebase using the query editor. I found it interesting to see how quickly they picked up the basic concepts and moved from understanding what a graph of entities was to learning how to do basic reconciliation using the search API (a surprisingly frequent request).

A common first impression seemed to be that Freebase looked really cool but that there was a lot to absorb at all once and many people weren’t sure where to start. After my experiences this weekend, I’m even more convinced that the best way to address this is to just sit down with someone and help walk them through their problem from the perspective of a Freebase expert. It’s so much easier to understand the difference between ids, keys, types, and properties when you can see it explained in the context of a problem that you’re trying to solve. As an expert this often means repeating many of the same basic concepts over and over but I think the end result is a much deeper understanding of how Freebase concepts are applied to real world problems.

By the end of the day, there were 17 teams with apps to demo; 5 of them using the Freebase API. Only one other API (OANDA) had that many apps. Although none of the Freebase apps placed in the top 3, they were all well received and I heard afterwards from the judges that some of them were very close.

These are the apps that were built using the Freebase API (sorry no URLs yet):

• SuperTwit – used the Cadmus & Freebase API to tweet out the ages of people from your top conversations.
• Trendeme – took the top trending words from Twitter to figure out if your friends tweet interesting things (used the Freebase API)
• What To Do – looked up where you are and gave you suggestions of where to go (used the Yellowpages & Freebase API)
• MediaGorge – applied the 4Square check-in concept to books, movies etc. using the Freebase API.
• Multiplayer Online Trivia – pulled in a list of movies from Freebase API and made a trivia game by guessing multiple choice questions.

All in all, I think it’s a net benefit to the community to have more people developing apps based on Freebase data since each new Freebase developer becomes one more person with a vested interest in the depth and quality of Freebase data. I hope that through more events like HackTO we can continue to grow the community and make Freebase an even more powerful resource for everyone to use.

Shawn

Thanks Shawn, and thanks to all the hackers at HackTO. If you are having a Hack Day or other event and would like Freebase to be part of it, please contact us!

We’ve been encouraging developers to use MIDs for some time now, but our own data dumps had not kept up with the times. The good news is, as of Friday October 13th (just over six weeks from now) our data dumps will start using MIDs instead of GUIDs. This will apply to all formats of our dumps — the TSV, link export, and simple topic formats. The not-so-good news is, if you are using our data dumps you may need to modify your code to deal with it. However, we believe that the benefits of MIDs will outweigh the disadvantages of this change.

If you have any questions or concerns, please let us know, by commenting here or on the freebase-discuss mailing list.

Until now, Acre development has always been tied to Freebase.com, meaning that you need to develop your Acre apps on our server, using our app editor. But we know that most software developers prefer to use their own native development environments — their favourite text editor, version control system, and so on — so lately we’ve been working on ways to make Acre work with source code that’s not stored in Freebase.

Today we’re pleased to announce that we’re releasing the Acre platform as open source software. This means that you can run Acre on your own machine, pulling templates and other files from your local disk and using your own development environment. While Acre still has close ties to Freebase (such as API hooks for easily making Freebase queries), this also means that you’ll be able to develop standalone, non-Freebase apps using the platform if you want. And, by running Acre on your own platform, you can avoid the resource limitations that are necessary in a shared environment.

If you’re interested in server-side JavaScript platforms, you may also be interested in some of the technical details of Acre.

• Acre is based on Rhino, Mozilla’s implementation of Javascript in Java. (In fact, “Acre” stands for “A Crash of Rhinos Evaluating”.) Acre, by default, uses the Jetty servlet engine as its HTTP server, but can be run in any servlet container.
• Acre includes a module system which supports high-latency source retrieval using extensive caching. Although Acre was originally designed to fetch data only from Freebase itself, it can fetch data from disk and will support a wider range of require() options such as WebDAV.
• Acre is capable of running on Google AppEngine, with support for synchronous and asynchronous HTTP requests, and the Keystore. Soon, Freebase’s own Acre installation will run on AppEngine.

Please download Acre and try it out, and let us know what you think!

This month’s meetup will be held at Google’s San Francisco offices, at 345 Spear St near the Embarcadero. Full address and instructions are given on the registration page (linked below).

Along with a new venue, we also have a new schedule. The meetup will begin at 5:30pm with technical presentations, including:

• Freebase platform overview
• Short tech talks about specific Freebase features
• Previews of what’s in store for Freebase in the near future
• Q&A

Then from 7pm onwards, we’ll adjourn to Gordon Biersch downstairs and kick on with an informal social gathering for Freebase developers and anyone else who’s interested. Both the tech talk space and Gordon Biersch have wifi, so bring your laptops.

For full details and to register, go to http://meetup.com/sf-freebase/ It’s vital that you RSVP for this event, as building security will have a list of attendees and will only let in those who are on the list. (No RSVP is required for the social gathering at Gordon Biersch, however.)

See you there!

Our planned cutover date is Wednesday November 10th, 2010. Anyone using Freebase’s API will need to update their code by that time. (If your client code can handle a redirect, it will continue to work, but will probably suffer a performance hit.)

We’ll be doing our best to contact as many API users as possible, and help them get transitioned across over the next three months. Please help us spread the information about his changeover throughout the developer community. We will be posting reminders on the official Freebase mailing lists and blog, but if you know Freebase API users who do not read those, please point them at this blog post to keep them in the loop.

If you have any questions about this, please post to the freebase-discuss mailing list.

You can find our fresh, tasty data dumps at download.freebase.com/datadumps. They are available in multiple formats:

• A TSV dump for quickly seeing data about a single type in tabular format, importing into Excel, etc
• A “quad” dump, for loading into a graph-shaped data store
• A simple topic dump, containing basic information about every topic in Freebase

There’s more documentation on the Freebase wiki.

Another related dump is our WEX (Wikipedia Extract) dump, which is a semi-processed version of Wikipedia’s own dumps, which can help you if you want to work with Wikipedia’s data (if you’re interested, see our announcement blog post from 2008). WEX is currently produced every two weeks. You can download the latest WEX from http://download.freebase.com/wex/.

If you have any questions about any of our data dumps, please post to the freebase-discuss mailing list.

“Whoah… what’s going to happen to Freebase?” you might ask. Well, we’re also extremely pleased to be able to say “nothing”…. or rather, nothing other than getting better, and yes, even more open. As a first step toward that last point, we’re also happy to announce today that we have increased the frequency of our downloadable database dumps from quarterly to weekly.

We hope both announcements will encourage even more individuals, institutions and companies to use and contribute to Freebase. We believe that the more we can all work together on cataloging and connecting the “graph of everything,” the smarter the web will get.

As for the Metaweb product line, we will continue to support all of our existing partners, but we won’t be taking on new ones while we work with the Google team to deliver new and innovative ways to make the web smarter by linking content with entities. No idea what we mean by that? Then check out the fun video below.

You can also read more in Google’s announcement on their blog.

Yours,
The Metaweb Team

• “View Source” links on every page written in Acre. One of the features that made the Web great will now work server-side too. Wonder how a feature on the site works? Soon you will be able to find out for yourself with a single click.
• You can make a difference. Different parts of freebase.com will each be their own Acre app. If you want to suggest a change, just clone the app, edit it, and then post a link to the mailing list… or just continue
• “Eating our own dog food” with Acre will make it better for you. This goes for both the platform itself and the ecosystem of libraries that run on top of it.

The release of Acre that just went live is the first evidence of that last point as it adds a bunch of new features. But before we get to those, it’s worth noting that we’ve also made a pretty fundamental change to how you refer to resources in Acre… which is rather than using Freebase IDs to refer to apps and files, you can now refer to them in the APIs with Acre URLs. For example:

acre.require("http://service.freebaseapps.com/lib")

The goal is to make it easier for developers to remember the relationship between apps and their source. However, in order to reduce the number of keystrokes required for using URLs, we’re also supporting a special “relative” syntax in which you can omit both the protocol and the Acre host suffix:

acre.require("//service/lib"); // released version
acre.require("//service.libs.freebase.dev/lib"); // current version
acre.require("//5.service.libs.freebase.dev/lib"); // specific version
acre.require("lib"); // relative file (unchanged)

Some things that may not be obvious from just these examples:

• The old graph ID format is still supported, so nothing should break.
• There is no longer a second version argument to acre.require as that is now implied by the hostname
• The new syntax makes doing the right thing easier in that requiring the released version of a library is now less typing than requiring the development version
• App Editor also supports this syntax. So you can just paste in any URL served by Acre into the path argument to view its source.

So on to the new goodies…

Re-usable Queries

The traditional pattern for loading a base MQL query and then customizing it in an Acre script goes something like this:

var q = acre.require("query").query;
q = acre.freebase.extend_query(q, {"id":"/some/id"});
var result = acre.freebase.mqlread(q).result;

That’s all well and good, but where things start to break down is when you want to customize the same base query more than once. The problem is that extend_query modifies the original query object returned from require. To get around this limitation, you either have to create a deep copy of the query or require the query twice. Ugh.

So in dev/29, we’ve tried to make these common use cases much simpler by adding some new methods to the query object returned by a require. So now our simple example can be written like this:

var result = acre.require("query").extend({"id":"/some/id"}).read().result;

The read method — and, of course, there’s a write method too — is just a wrapper around acre.freebase.mqlread and has all the same functionality.

The good news is that unlike extend_query, every call to .extend() returns a new deep copy. So now, querying two distinct variations on the same base query is as simple as:

var q = acre.require("query");
var result1 = q.extend({"id":"/some/id"}).read().result;
var result2 = q.extend({"prop":"/some/other/constraint"}).read().result;

We think that’s a lot nicer and hopefully will save you a lot of boiler-plate code… which has always been a design goal with Acre.

Async Fetching

Acre is all about presenting data. However, the current serial approach to fetching data combined with the 30 second limit per page constrains the amount of data you can actually fetch in a single request… especially if you have any slow queries (like big counts or sorts).

Now you can fetch data in parallel so that one slow query won’t hold up all your other queries. This can be a huge savings for some use cases. For example, some of our “dashboard”-style pages that used to timeout, now run in a couple of seconds. However, there is a limit to this goodness… you’re only allowed a maximum of 10 simultaneous requests (you can fire more off, they’ll just have to wait on each other).

The two new APIs for async are:

acre.async.urlfetch(url, options);
acre.async.wait_on_results(timeout);

Where the options for acre.async.urlfetch is a dictionary that can contain:

• callback – function to call with response object if successful
• errback – function to call with error object if not successful

…as well as the standard HTTP request options from sync urlfetch — method, headers, content, and sign. Plus one new one that can come in especially handy with aysnc:

• timeout — #of ms to give the request before throwing (or calling errback with) a timeout error

acre.async.wait_on_results fires all active async fetches and then blocks until they are complete (IMPORTANT: async fetches will not run at all unless you call this). While that might seem like an unnecessary extra step, it can actually be quite useful and we felt the tradeoffs were worth it:

1. Makes it easy to upgrade existing templates and scripts to using callbacks (don’t need to nest template calls in callbacks). Just call it in between setting up your fetches and running the parts of your page that depend on the data from those fetches (same as today).
2. acre.aysnc.wait_on_results can be called multiple times, so it can also be used as a simple request grouping mechanism.

One more bit of good news… all of the acre.freebase.* APIs already support async too. All you have to do is include a callback and/or errback in each API’s options dictionary and the call will automatically be done asynchronously.

For more detail, we’ve already updated the Acre API docs and there’s also a small demo Acre app with some basic examples.

Routing

Now you can also take a lot more control over request handling in Acre… but only if you want to. If you do, just create a file named routes in your app and then all requests will go to that file first. From there, it’s up to you to handle the request. We are providing two new APIs, however, to give you some additional tools to work with:

acre.route(url)
acre.get_metadata(url)

Calling acre.route is nearly identical to doing a redirect (presuming the target is also served by Acre)… except that the URL the user sees does not change. However, that does mean the URL you route to can include extra path and/or parameters and that those will appear in the acre.request object of the file being routed to.

Finally, acre.get_metadata allows you to introspect an app before calling acre.route. Since it defaults to the current app, so you can even do things like the following:

var md = acre.get_metadata();
if (!md.files[filename]) {
// do some error handling
}
acre.route(filename);

…to ensure that you don’t route to a file that doesn’t exist. We’ll be talking more about routing in the future as we’re using it heavily in the development of freebase.com; it can be very useful for stitching together disparate apps into a single whole… or making shorter, prettier URLs that include Freebase IDs. :-)

Whew… and this is just the beginning of what should continue to be an interesting journey.