Mainly Data

Blogging for the Yoga Bear Cause

Fri, 29 May 2009 12:53:00 -0700

I’ve been lucky to watch Yoga Bear grow from an idea to a nationwide organization of hundreds of volunteers, yoga studios, and participants recovering from cancer. The Blogging for a Cause campaign from Zemanta is looking for nonprofits to help out, and I can’t think of one more worthy than Yoga Bear. From events like Yogapalooza and the Meeting of the Minds to their commitment to hilighting research on the effectiveness of alternative medicine in the treatment of cancer, Yoga Bear is enhancing the lives of many on a shoestring budget. Let’s hope Zemanta recognizes the great work Yoga Bear is doing and awards them the grant!

The image above is used by Google to implement a common...

Thu, 29 Jan 2009 20:11:00 -0800

The image above is used by Google to implement a common technique known as CSS spriting. This technique is used by most major websites to minimize HTTP requests and hence improve page load times. As an example, Facebook loads all of the News Feed icons in a single PNG file, displayed below. And yes, reblogging this post was an excuse for me to display Facebook’s icon set, which is totally awesome—Facebook’s design team is superb.

davemorin:

This is crazy genius.

superamit:

“Google’s search pages load this 1 single image for their logo and all the UI parts and pieces, using CSS to crop the images as needed, which seems to me as 100% more efficient than managing a pool of tiny images.” - Dressed In Value: Google (via Chris Glass)

Smart!

Hadoop for a living

Thu, 30 Oct 2008 22:33:24 -0700

Hadoop for a living:

Hadoop has been good to me, so I’m going to try to be good to Hadoop. Come say hello to us in Burlingame.

Hive is alive!

Sat, 16 Aug 2008 03:25:40 -0700

Hive is alive!

bigdata

Wed, 23 Jul 2008 01:18:02 -0700

bigdata:

If you’re at OSCON, please attend this talk and let me know if the hyperbolic claims in this abstract are substantiated.

August in San Francisco

Sat, 19 Jul 2008 16:08:05 -0700

zachklein:

I’m renting a place near Alamo Sqaure for a month in order to be closer to a handful of friends I’ve been distant from for too long, to recruit developers, to learn the geography of the Bay Area, and to have a convenient jumpoff point for a few backpacking trips in the Cascades.

yay.

Cassandra

Wed, 02 Jul 2008 15:52:00 -0700

Cassandra:

Congratulations to Avinash, Prashant, and Karthik.

MMDS 2008 and CIM

Mon, 30 Jun 2008 02:29:00 -0700

Last week the second Workshop on Algorithms for Modern Massive Data Sets was held at Stanford University. This workshop had an incredible density of prestigious speakers from the field of machine learning; I guess spending a weekend in California during the summer is an easy sell.

Each of the four days of the conference had a theme. The official themes:

Data Analysis and Data Applications
Networked Data and Algorithmic Tools
Statistical, Geometric, and Topological Methods
Machine Learning and Dimensionality Reduction

After attending several talks by these machine learning luminaries over the course of four days, I tried to pull together a few common themes of my own for further exploration:

Incomplete Dyadic Data
Distributing Data and Computation
Manifolds with Noise

I’ll try to put a detailed post up on each of these three topics this week.

Over the past few years I’ve become a regular conference attendee. It seems plausible that I will continue to attend KDD, VLDB, SIGMOD, and MMDS for years to come, leaving me with a major question: where can I find a longitudinal analysis of the content of these conferences?

It seems that these conferences provide an excellent yearly cross-section of the state of their respective fields, but I’m much more interested in the deltas. What are the new topics, which topics are making forward progress, and which topics are losing steam? Taking time to collect this information would provide some fascinating insight into the progress of science.

The KDD community has done some analysis of the DBLP data set, but not with the aims proposed above. From the VLDB community has come some work that is a bit closer in intent: AnHai Doan’s Cimple project. So far, they’ve produced the moderately useful DBLife, and they’ve outlined a promising research direction.

I’ve recently started a conference type on Freebase and have a simple tool to monitor conference progress using their API. I’d like to make this tool more general and robust. Any CIM students or conference geeks looking for a summer project?

My good friend, musical mastermind Matt O’Malley, cooked...

Fri, 27 Jun 2008 02:51:42 -0700

My good friend, musical mastermind Matt O’Malley, cooked up a theme song for the Facebook Data Team. You can also check it out on my Muxtape. It’s just so, so good. If you would like to create an animation to accompany the music, please let me know.

REPL

Thu, 26 Jun 2008 22:30:55 -0700

Python is an amazing language for many reasons, but code prototyping via the interactive interpreter has crippled my development speed in languages that do not have a REPL.

For this reason, I was ecstatic when I joined Facebook and learned they had developed their own interactive shell for PHP, phpsh. My Javascript development was hastened when I found Mozilla’s spidermonkey.

I also enjoyed learning Ruby and Erlang, because in Ruby, you have irb, and in Erlang, you have the Erlang shell, Eshell.

I recently came across CERN’s CINT, a REPL for C/C++. Finally! I have come to dread coding in these languages, especially C++, but I’m now looking forward to my next big C/C++ project. If you’ve used CINT, drop me a line.

I’ve also started looking for a nice Java REPL. There’s DynamicJava, and the Groovy interactive shell. Anything else I should try?

Wired Magazine: (Brief) Dispatches from the "Petabyte Age"

Wed, 25 Jun 2008 01:22:00 -0700

I tend to geek hard when a publication drops a series of articles about petabyte-scale data analysis. Frequent culprits include the SIGMOD Record or Teradata Magazine, but this week, Wired Magazine splashed into the pool.

Chris Anderson (the long tail one) starts things off by declaring that the data deluge spells trouble for the scientific method. Clearly this claim is false; Google’s success is almost entirely due to rapid product iterations via thousands of hypothesis tests every month. To give Chris the benefit of the doubt, he seems to be asserting instead that the practice of model development should be done in close concert with the collection of empirical data. I couldn’t agree more.

Near the end of the article, Chris makes a common (if frustrating) mistake: he claims that the cluster used for the NSF’s CluE program will be running the Google File System, as well as crediting Google and IBM for building the software that will power this cluster. In actuality, Yahoo deserves almost all of the credit, as they have done an incredible job scaling the Hadoop project to thousands of nodes and making the NSF program possible. Just another testament to the effectiveness of Google’s PR machine.

The rest of the articles cover many application areas of large scale data analysis in brief, including agriculture, astronomy, high-energy physics, politics, epidemiology, and insurance. There’s also a startlingly incoherent attempt to describe how MapReduce works that probably should have been left out.

Wired’s intentions were noble but their execution was not up to expectations. I’m hoping that tomorrow’s MMDS Workshop will have a bit more substance.

Bloomberg for the Web? We Need Real-Time News, Data, and Analytics.

Mon, 23 Jun 2008 23:03:00 -0700

The NYT had an article today about the battle for efficient financial services information delivery currently heating up between Bloomberg and Thomson Reuters. A few weeks ago, there were articles about the NASDAQ and NYSE making their stock quotes available in real time to multiple information outlets. And just last week, Tibco, once a wholly owned subsidiary of Reuters, purchased Insightful. Insightful makes the S-Plus statistical analysis software on which the R project is based. Tibco already owns Spotfire, a firm that makes excellent software for data exploration.

These developments made me think back to my time in financial services. Trading floors have the highest concentration of numerate people I’ve ever been around. They also have ready access to superb information manipulation software.

Now consider the current state of analytics software for the web. I won’t bother to list the major competitors, as they are all pretty mediocre. As the online advertising space grows in sophistication and mechanisms to promote a more efficient market are introduced (see, for example, Right Media and ContextWeb’s ADSDAQ), a workbench for the real-time exploration of news and data related to the web will be a necessary tool for many quantitative marketers.

As Hal Varian points out, marketing is the next field to be overrun with quants, and I expect that the tools most useful in finance will be brought along for the invasion.

Yoga Bear: Now Using YUI and App Engine

Mon, 23 Jun 2008 02:08:00 -0700

Yoga Bear: Now Using YUI and App Engine:

My girlfriend Halle works hard at her nonprofit, Yoga Bear. I help out by maintaining the website. Today I ported the website from Django (hosted by WebFaction) to Google’s App Engine; I also ported the CSS to use YUI’s reset, fonts, and grids.

As a result, we no longer have to pay hosting fees and the site renders properly in IE 6. Furthermore, the transition will allow me to explore the YUI AJAX libraries and App Engine data store to add some new features to the Yoga Bear website. The web is beginning to feel less hostile to rapid development.

I made a wordle from my delicious tags. I found this site in...

Sun, 22 Jun 2008 17:59:30 -0700

I made a wordle from my delicious tags. I found this site in the App Engine gallery.

I really dislike web programming due to its many accidental complexities, but the simplicity of developing a web app in python via App Engine has me experimenting with the web once again.

TIME Digital's 1998 "Cyber Elite"

Sat, 21 Jun 2008 04:51:00 -0700

TIME Digital's 1998 "Cyber Elite":

A late-night browsing session landed me on this gem from ten years ago. Some highlights:

It’s been said before, but reading the description of GeoCities from 1998 really drives home the fact that online social networking dates back to the beginning of the web.
DoubleClick is really old school. It’s crazy to think of them integrating with Google. Also, there’s a reminder of how controversial cookies were on the early web: “Using controversial software called cookies, DoubleClick sites can snoop on users’ browsing habits, sometimes picking up such critical information as zip codes”.
Rupert Murdoch’s internet ambitions were already apparent.
Joe Nacchio was not on trial.
Even before the iPod, Jonathan Ive was recognized for his visionary designs.

TIME also showed an early awareness of the web’s global reach, highlighting Japanese, Chinese, and Brazilian leaders.

Historical artifacts like these have helped bring perspective to my first two and a half years in the Valley. For another historical reference, check out an old Forbes article about the aftermath of Yahoo’s first major acquisition spree.

Creative Commons Technology Summit 2008

Sat, 21 Jun 2008 02:32:00 -0700

On Wednesday, Creative Commons hosted their first ever technology summit at Google. I’ve been into CC for a while now and I was relatively pleased by their progress since I last checked in.

Joi Ito started the day off with a lucid delineation of CC’s major components. He pointed out the technical side of CC is hoping to create a set of standards for digital media exchange in a similar spirit to what the IETF does for the internets as a whole. The political side of CC is more akin to the Open Internet Coalition that is fighting to keep these standards in neutral hands.

Ben then gave a great outline of the technical components of ccREL. I am impressed with the refinements and flexibility introduced by the full-scale adoption of RDFa for semantic markup.

The later sessions started to drift away from my core interests, but I was intrigued by the proliferation of digital copyright registries: Registered Commons, SafeCreative, and Noank Media, for example. It was great to see Attributor have a presence at the summit. They’re also heavy users of Hadoop—I am once again impressed by what Jim has built.

I’ve enjoyed watching Creative Commons evolve over the past several years and I’m still holding out some hope that I’ll be able to have a material impact on their success some time in the future. For now, it’s great to keep up with the team; I trust they’re in good hands with a fellow Cavalier leading the way.

Hive JIRA ticket opened!

Thu, 19 Jun 2008 19:09:49 -0700

Hive JIRA ticket opened!:

We’re one step closer to getting our data warehousing framework built on top of Hadoop into the hands of the community.

I learned yesterday that Google’s ads data warehouse,...

Sat, 07 Jun 2008 19:01:00 -0700

I learned yesterday that Google’s ads data warehouse, built on Netezza, is named “Everest”—the same name Yahoo chose for their modified Postgres data warehouse. Awk-ward…

Repeatability, again

Fri, 06 Jun 2008 15:50:00 -0700

In a recent post I mentioned the idea of reproducible research. It turns out that this year’s SIGMOD conference, where we’ll be presenting a new approach to structured storage, has conducted an experiment in reproducible research. You can read a fascinating account of the experiment in this month’s SIGMOD Record. While you’re there, be sure to check out the “Data Management Projects at Google” article as well!

Hadoop at Facebook

Fri, 06 Jun 2008 13:48:00 -0700

There’s a post on the Facebook Engineering blog today from one of the Data team engineers, Joydeep Sen Sarma, discussing how we use Hadoop here at Facebook. Check it out.

For more on Hadoop at Facebook, you can check out the slides from a set of lectures I gave at IBM’s Cloud Computing Center in Dublin.

For a deeper look at the architecture of HDFS, check out the presentation [PDF] that dhruba, a recent addition to the Data team, gave at the IBM storage team’s recent offsite.