Luigi Montanez

QR Codes In China And The Troubling State of Tech Reporting

2013-05-31T00:00:00-04:00

A slide deck by Mary Meeker, partner at Kleiner Perkins, has been making the rounds:

KPCB Internet Trends 2013 from Kleiner Perkins Caufield & Byers

One graph in particular caught my eye: Slide 64 reports a 4X year-over-year increase in QR code usage in China. There were around 2 million scans in March 2012, and 9 million scans in March 2013.

That certainly looks impressive! The audience is left to conclude that QR code adoption in China is turning out to be quite different from what it’s been like in the U.S.

But something about this smelled fishy to me. China is a country of 1.3 billion people, so 9 million of anything isn’t that big. If those 9 million scans were performed by 9 million different people – a presumption that’s highly optimistic – that would mean just 0.7% of the population scanned a QR code.

But surely, the 4X year-over-year growth is something notable, right? Not when you consider that QR code usage will go up as more people use smartphones. A mere three slides later in the deck, we see how much iOS + Android usage has grown in China:

In February 2012, China’s active base looks to be around 70 million, which grew to nearly 250 million in February 2013. So year-over-year growth in nearly the same time window is about 3.5X, which almost fully accounts for the 4X growth in QR code scanning.

Based on this quick analysis, QR codes look to be as much a dud in China as they have been here in America. Certainly not as impressive as the slide makes it out to be.

I searched Google to find corroborating data, and instead saw that the tech press has been breathlessly reporting on that same slide I had noticed:

TechCrunch claims that “people actually use” QR codes in China, repeating key points from the slide deck, and concluding with a personal anecdote about a recent trip to Beijing.
BizReport says that QR code scanning is “taking off” in China, then repeats the key points from the slide deck, and at least does some additional research to dig up links on QR codes in cemeteries and watermelons.
MIT Technology Review wrote an article on the broader points about China that Meeker made, but also uncritically repeats what was stated in the deck.
And VR-ZONE lazily rewrites the TechCrunch article.

All these articles simply repeat something that was stated, in some cases verbatim, without even a hint of any critical thought going into the production of content.

So who’s at fault here? On the individual level, Meeker should have done a little more digging into why QR code usage has risen, especially considering that the answer was right there three slides later. Each of the articles linked to above were written by tech reporters, and they bear the brunt of responsibility for not doing any due diligence for their readers.

But the real fault lies in the broken institution of journalism itself. The tech industry, particularly around consumer Internet and mobile, is severely underserved by its reporters. The quality of journalism is consistently bad – lacking depth, insight, and expertise. Original, long-form reporting is rare. Pundits with opinions have disproportionate mindshare, particularly those who write about Apple and Google. They certainly drive pageviews. But punditry alone isn’t good enough.

The Wall Street Journal is the most widely circulated newspaper in America. Imagine if it only consisted of its right-wing editorial section. Sure, those pundits might be able to attract some readers, but not nearly enough to be sustainable. The reason the Wall Street Journal is so successful is because it produces extremely high-quality journalism around business and finance. It serves its industry very well.

So what the tech industry needs is an equivalent of what the Wall Street Journal is to those in the finance industry. Or maybe – and excuse while I hold my nose – what POLITICO is to Washington, D.C. and political junkies.

The current outlets just don’t cut it. The newspapers in Silicon Valley don’t have much reach. On the Web, Ars Technica and The Register produce quality material, but not enough. The hugely popular gadget blogs are more about consumerism as entertainment instead of the technology that’s really at the heart of the industry: software.

Imagine a news organization that had the technical expertise and industry connections to really dive into the Heroku/RapGenius kerfuffle, or the Adria Richards incident, or why exactly Ruby on Rails had so many critical security patches a few months ago. This industry is so important, and it deserves better journalism.

It’s time to do something about that.

What I'll Miss Most About Google Reader: Speed and Magic

2013-03-17T00:00:00-04:00

Last week, Google announced that it plans to shut down Google Reader later this year. I’m a passionate fan of Google Reader – using it several times a week – and I was very disappointed to read about the decision.

I’ve been heartened to see the flurry of activity around building a replacement, culminating with the announcement that Digg is planning their own Reader.

In the midst of the hand-wringing and petition-signing on Twitter, I saw an astute observation:

Lots of boo hoo about Google Reader. Not much mention of any specific feature that dies with it.
— Sam Schenkman-Moore (@samsm) March 14, 2013

Must be frustrating. Google Reader competitors reading this whining get no insight into what to create to replace it.
— Sam Schenkman-Moore (@samsm) March 14, 2013

So as a sort of PSA to whomever builds the next feed reader I wind up using, here’s what I think makes Google Reader so special: speed and magic.

Speed

For an app that displays lots of content from many different sources, Google Reader is consistently fast. It’s snappy. I click – and boom! – I see what I wanted to see. The load times are nearly imperceptible. Sadly, the alternatives I tried this week were decidedly un-snappy.

But more important for me – as a practitioner of a mouseless lifestyle – is Google Reader’s extensive support for Vim-like keyboard shortcuts.

Those jumping shortcuts (g then _) are game-changing. So, so good. The movement key choices are actually a bit quirky. I find myself hitting the ‘N’ and ‘P’ keys more than the home keys like ‘J’ and ‘K’. But over the years I’ve developed muscle memory, and I go into a zen-like state when processing my feeds.

The next feed reader I use needs to work as fast as I can think. It needs to load content quickly, and it needs to have Vim-like keyboard shortcuts baked in.

Magic

To subscribe to a blog through its feed, one must find the RSS (or Atom) link somewhere on that blog. This takes some technical savvy. There’s a <link> tag that makes it browser-discoverable. If the blog doesn’t have that, it may have an RSS link or that orange icon somewhere in the header, sidebar, or footer.

In recent years, I’ve noticed that some blogs just don’t provide any sort of link to a feed. No <link> tag that gets detected by a browser. No RSS icon. And that’s where Google Reader’s magic comes in.

Because in that “Subscribe” box, I can pop in the URL of the blog I want to subscribe to, and Google Reader just knows the feed for it. It’s never failed me. It. Just. Works.

The next feed reader I use needs to be a little magical when it comes to feed detection, because I don’t have the patience to hunt down feed URLs. Taking it a step further, I’d like a browser extension that automatically adds the blog I’m viewing directly to my feed reader and doesn’t interrupt my flow. It should Just Work.

Two Virtues of Great Web Apps

There’s a larger point here: Speed and magic are the two most important virtues of great web apps. Be as fast (or faster) than the user can think. Don’t make the user wait. And delight the user by providing magically pleasant experiences.

Google understands this; those virtues are what make the best Google products so great. I consider Google Reader to be one of the best web apps ever made. Here’s hoping that the creators of the next great feed reader keep speed and magic on the front burner.

The Rise of the Activist Engineer

2013-01-08T00:00:00-05:00

I’ve had a unique career for a software developer. An idealistic Howard Dean supporter during college, I went to work for his PAC Democracy for America immediately after graduation. One brutally cold Vermont winter is one too many, so I then spent a few years building websites as a consultant for political campaigns, advocacy organizations, and labor unions. I proceeded to work at the Sunlight Foundation for a few years before settling into my current role at Upworthy. I’ve always been employed to build web apps, but I’ve done it exclusively for organizations that aligned with my particular worldview.

Obama 2012

This mission-driven approach to a career is commonplace for those of us inside the Beltway, but until recently wasn’t very common for those who work in the larger tech industry. That appears to be changing. A tipping point seems to have been reached with the staffers of the Obama 2012 re-election campaign. The campaign’s Tech Team has been the subject of glowing press profiles in the aftermath of their victory.

Previous presidential elections, particularly on the Democratic side, have spawned tech companies. Consultancies like Blue State Digital and EchoDitto formed in the wake Howard Dean’s 2004 primary campaign, and Barack Obama’s victorious 2008 effort yielded startups like Optimizely and NationalField. While many of the Obama 2012 tech staffers came from outside the political sphere, it stands to reason that quite a few will continue to be involved in politics and advocacy.

Beyond the several dozen engineers employed directly by the Obama campaign, the fawning press coverage about them will surely inspire many others around the country, and maybe around the world, to focus their talents towards politics and advocacy.

From a software developer’s perspective, what the Obama Tech Team did wasn’t technically groundbreaking. Instead, it’s clear that the team was thoroughly competent, given the time and resources needed to succeed. That sort of work environment is highly appealing. The Obama Tech Team’s lasting contribution will be the example they set for the future.

Changing the World

The prospects for more activist engineers are bolstered by a failed promise of Silicon Valley. A long-time refrain has been that VC-backed Silicon Valley startups can indeed change the world. But the current crop of consumer web and mobile startups don’t live up to that ideal. SnapChat and Facebook Poke aren’t changing the world, they’re enabling people to be even more self-centered and insular. Airbnb and Uber, two terrific services that are profoundly disrupting their markets, don’t make that dent in the universe Steve Jobs once spoke of.

The change the world lottery is even harder to win than the startup lottery. Facebook, Twitter, Apple, Google. And winning the change the world lottery is fleeting. The average engineer who spends a year working for Google or Facebook will do more good for the world by donating her considerable year-end bonus to various charities. If you want to change the world, apply your talents to actually changing the world.

As noted by Catherine Bracy, Silicon Valley has culturally created such an enormous bubble around itself that it’s blind to issues experienced by the rest of the world. Yes, SOPA and PIPA were bad, but our current prison system is horrific.

The situation isn’t completely dire, of course. Bright spots are emerging. It’s important to identify and organize those engineers who want to contribute their talents directly, and that’s what Catherine Bracy did as she led the Obama campaign’s San Francisco Tech Field Office.

Civic Hacking

The movement that best organizes software developers towards change is the one centered around open government and open data. Civic hackers directly apply their skills to make government work better and be more accountable. The organization that best harnesses developer energy is Code for America, focused on improving city governments and now working towards a fascinatingly anarchist definition of citizenship. Anarchist is a scary word, and I don’t mean that Code for America promotes disorder or nihilism. But they are building tools that begin to obviate the need for government, replacing it with civic action. To me that’s a profoundly disruptive version of civic hacking.

Literally hundreds of web and mobile apps have been built out of this movement, and it’s still in its early stages. As uber civic hacker Mark Headd recently wrote, the barrier to even more activity isn’t a lack of energy, it’s a lack of raw government data to work with. Governments need more convincing, which will require more activism.

Civic hacking is a sort of gateway drug for activist engineers. The concepts behind open government like transparency, engagement, and accountability cut across partisan lines. It broadly appeals to many people, particularly when contrasted with putting your life on hold to work for a presidential campaign for 16 months. The organizations that lead the movement are all prolific at publishing their work on GitHub, setting examples for all to see.

Open Source and Free Software

Speaking of GitHub, it’s worth considering the open source and free software movements. Does Richard Stallman fall under the banner of activist engineer? Linus Torvalds? The Ubuntu and Apache teams? Have they not changed the world?

I’d argue that most open source software is about craft rather than seeking societal change. Rails and Django are terrific at improving the lives of the developers who work with them every day. But they’re ultimately tools used to build things, not outcomes in and of themselves.

By contrast, Richard Stallman, the fierce advocate of free software, is literally on a mission to change the world. He’s an activist engineer, but his activism narrowly focuses on free software. While he did build the original GNU tools and Emacs, these days Richard Stallman is more like a community organizer, spreading the gospel of free software around the world.

Building a Sustained Community

Community will be key for determining whether the rise of activist engineers can be sustained for the long run. Lots of pathways have been cleared. It’s now up to the activist engineers out there to build on top of the foundations that have been laid.

Progressive technologists Jim Pugh and Nathan Woodhull recently pleaded for someone, anyone, to keep the technology created for Obama 2012 maintained and ready for the next election. But I don’t think it’ll be so bad if the code is abandoned. What would be disastrous for the Democratic side is if the community of activist engineers built up to support it collapses.

Technologies change. An app built in 2012 shouldn’t necessarily be used in 2016. But people matter, and we don’t go out of date nearly as fast as Rails code.

Activist engineers have arrived. We’re here. Now will we recognize our collective power, grow our ranks, and shape the future?

A Reminder: The Web Exists

2012-09-26T00:00:00-04:00

Lost in the last week’s reporting around the new Apple Maps app in iOS 6 has been one salient fact: Google Maps continues to be available to all iOS users at maps.google.com. It’s the same Google Maps that iOS users have grown to know and love, with the exception of Street View. And it even prompts for “installation” to the home screen, so access is quick and easy:

Image by CNET

The tech press, which presumably exists to serve its audience, has largely ignored the existence of maps.google.com, instead opting to focus on the corporate drama behind Apple’s decision to dump Google’s maps and roll their own solution. Take this piece from the NY Times, which reports that a Google-made Maps app is still months away from completion. A reader of the story would be left dejected: Google Maps is months away, and still has to be approved by Apple for inclusion in the App Store.

But that’s simply not the case – Google Maps is available to all iOS users right now at maps.google.com. And because it’s just a web site, it’s available to any device that has a web browser. In this way the tech press is similar to the political press, valuing conflict and process over long-standing realities. The end result is sadly the same: the public loses out.

John Gruber, the Rush Limbaugh of Apple punditry, similarly neglects to mention the alternative of maps.google.com on Daring Fireball. He mentions the Maps issue in four separate posts:

A link to an article on how iOS 6 Maps is a huge improvement in China.
A link to an article on Counternotions that rebuts the criticisms.
A link to the Amazing iOS 6 Maps Tumblr.
A link to a process article on the prospects for a Google Maps app.

While Gruber’s audience is the devoted Apple faithful, the lack of a mention of maps.google.com is a clear disservice to his readership.

The larger problem here is a general lack of respect for Web-based solutions. Apple’s push for native apps on iOS has severely hurt the Web. Users become paralyzed, thinking that if there’s no solution available on the App Store, it must not exist. Startups like Lift and Simple require a native iOS app to use their Web-based services. That’s a trend that needs to be reversed.

We can’t let ourselves forget about the Web.

On Starting a Startup

2012-07-03T00:00:00-04:00

We launched Upworthy 99 days ago on March 26. We started as a company two months earlier on January 30. There were various news stories written about us as the team was being built, as we launched, and a month after we launched. Read those if you’re not familiar with our story. What follows are some of my thoughts and reflections on the experience of starting a startup.

Choosing a Technology

The engineering team started out as a team of two: the CTO and me. I was anointed Founding Engineer. My background is in Ruby on Rails, but in recent years I’ve soured on it a bit. Eric, a colleague of mine when I was at the Sunlight Foundation started using Sinatra for all his projects, and he seemed quite happy. So after some evaluating and thinking, Tim and I settled on Padrino, a framework built on top of Sinatra that incorporates most of the best parts of Rails.

At the end of our first day as a company, I made sure to create a paid private account on GitHub, generate a skeleton Padrino app, and push the initial commit.

I’ve been very happy with our decision to build on top of Padrino. The built-in admin interface was a huge time saver when building towards the minimum viable product, and nearly every major Ruby library out there is compatible with Padrino and Sinatra. Some of the technologies we use are:

Mongoid and MongoDB for our main data store.
Redis for caching and Resque for background processing.
Twitter Bootstrap and LESS for stylesheets.
Jammit Sinatra for asset packaging, though Sinatra AssetPack looks appealing.

We launched on Heroku, and while I’m not as happy with that decision as Padrino, I’d still recommend Heroku for startups like us. There’s been some widespread downtime in recent weeks, but some of the blame can be shared with AWS. For a startup with limited sys admin resources, Heroku makes much more sense than directly managing AWS services. Yes, we do have to architect our app with Heroku in mind, but it’s not nearly as severe as hosting on Google App Engine. There’s always been a reasonable solution for what we want to do.

Buying the Domain

Naming our website and acquiring the domain was probably the hardest thing to accomplish. We stumbled on Upworthy very late in the process. It was preceded by weeks of head scratching and hair pulling because the domain name industry is completely broken. It’s been reported that if Obama wins a second term, he’ll try to reform The Drug War. Well, I hope he also tries to reform the domain name system and disband ICANN. There are too many good domains being squatted on. Offering seemingly large amounts of money to those domain squatters is fruitless. They call themselves “domain investors” and believe that they’re a legitimate market force. But all they’re doing is stifling innovation, enabled by a broken institution in ICANN.

When we settled on a name (not Upworthy) a few weeks into the project, as the first technical hire, I took on the responsibility of securing the domain for us. It was expiring in the middle of February. After doing some research, it was clear that I wouldn’t be able to just register for the domain name manually on the day it became available. So I dutifully registered on the Big 3 backorder services. The domain backorder market is a rigged game, where those companies have deals with registrars that give them the first crack at registering domains the millisecond they expire. And when multiple people register a backorder on the acquiring backorder service, there’s an auction held for a few days and the domain goes to the highest bidder.

It’s capitalism at its not-so-finest. We braced ourselves for an auction.

On the expiration date, the moment of truth came (for .com domains it happens at 2pm ET). And it went. None of the backorder services I registered with picked up the domain. Instead the domain name was snatched by a service called BackorderZone, which I had never heard of. And we were left out of the auction, because only people who bid on the domain on the winning service before it’s snatched up get to participate. I felt like a complete failure. We tried to negotiate with the ultimate winner of the auction, but it didn’t pan out.

At that point, we already had a design, a brand, an app, and a logo. But we had no domain name. After some desperate brainstorming sessions, the name Upworthy came to Eli as he was on a run, and we quickly went with it. It was close enough to our initial choice that we were able to launch just two weeks later.

So what are the lessons here?

If you’re going to try to negotiate for a domain name or snatch an expiring one, realize that the chances of you obtaining it are actually very small, unless you’re willing to spend six figures. If you try to snatch up an expiring domain, use BackorderZone. The operation is just one guy in Pennsylvania and apparently he knows what he’s doing.

But I recommend not playing that game at all. Make up a new word like we did. Combine two short words. I think we lucked out by not getting that first domain name, because I think Upworthy more accurately connotes what it is we’re trying to do.

Facebook: One-Fifth of the Web

So what exactly are we trying to do? We want to make stuff that matters as shareable and viral as funny pictures of kittens. And these days, that means optimizing for social networks. Particularly Facebook.

As a software developer and early adopter, my social network of choice is Twitter. And Upworthy certainly has a strong presence on Twitter: a following of 10,000 built up in the last 100 days. If you do a real-time search for “upworthy”, you’ll find a constant stream of tweets linking to us.

But Twitter traffic simply doesn’t compare to the traffic we see from Facebook. I could tell you that Facebook is like the Sun, and all the other social media sites (Twitter, Reddit, Pinterest, StumbleUpon) are like the planets. But then you’ll probably picture something like this:

That’s actually not to scale. Because in reality, the Sun is unfathomably huge. This more accurately depicts Facebook’s dominance:

One in five web pages loaded is a Facebook page. Until I started at Upworthy, I didn’t appreciate how nearly everyone is on Facebook. Normal people. It’s truly a form of mass media, and the other social networks we frequent like Twitter and Reddit aren’t in the same league. Facebook, whether you like it or not, is the present and future of the Web. If you’re in the business of spreading ideas, knowledge, and information, you’d be sorely mistaken to ignore it.

You know about the Like button, but here’s a tip: It’s not very effective. Use their (deprecated) Share pop up instead. Shared items make it into friends’ news feeds, not liked items.

A Lean Startup

I love the taste of the Lean Startup Kool-Aid. I’ve found the books so important that I own both Kindle and paperback versions of them:

The Lean Startup by Eric Ries
Running Lean by Ash Maurya
The Startup Owner’s Manual by Steve Blank and Bob Dorf

The Lean Startup methodology is a great way to think about building a business. It provides much needed structure in treating a business as an experiment. Make hypotheses. Run tests. Prove hypotheses with data and results.

As an engineer, I’m not too involved in the day-to-day of running the business. But I’m glad for the knowledge gained from the Lean Startup movement.

Two Memoirs

At the beginning of my time at Upworthy, I relaxed at night by reading iWoz, the autobiography of Apple co-founder Steve Wozniak. Woz was responsible for engineering the Apple I and Apple II. He essentially invented the personal computer as we know it.

I never quite adored a book until reading iWoz. It’s probably the favorite book of my adult life. Woz is kind, honest, and intelligent, and that all comes through on the page. People like Steve Jobs end up running multi-billion dollar companies, but they can’t do it without people like Woz. He simply wanted to make awesome stuff. There was one passage in the book where Woz talked about his creative process: He’d think and think on a problem for days, and when he came to a solution, he’d implement it quickly in a matter of hours. That’s how I work, and it was re-affirming and inspiring to read that I shared that in common with him.

I read another book, just in the last week, called The Boy Kings. It’s a memoir of Facebook employee number 51, Kate Losse. She started out in customer service and worked her way up the company over the course of several years. It’s an engaging, well-written look inside Facebook as it grew, and it most clearly describes the “brogrammer” culture that pervades tech startups.

It’s a bit of an anthropological study of the “boys” of Facebook, as Kate refers to them, viewing them and their project with a critical eye. Obviously, Mark Zuckerberg is described in great detail, but I was most fascinated by the descriptions of co-founder Dustin Moskovitz, the world’s youngest self-made billionaire. He’s portrayed in the vein of Woz: genuinely kind, perceptive, and a hell of an engineer. While Mark schemed up plans for world domination, Dustin was busy making sure the site stayed up. The memoir is from the view of just one person, but I can’t help but think about how different Facebook would be if Dustin was the CEO and not Mark. And it led me to think about how different Apple would be if Wozniak led it and not Jobs.

Those two memoirs drove home two important points for me: First, that I want to be an engineer in the mold of Dustin Moskovitz and Woz. I respect them infinitely more than I respect Zuckerberg or Jobs. And second, technology companies really do have the power to change the world. Apple and Facebook aren’t just companies; they’re institutions. Both books tell the story of how those institutions were built.

Upworthy is about to complete its first hundred days. I can’t wait to see what the next hundred days, and then the next thousand days bring for us. I’m thrilled to be at the ground-level, as we build something that’s trying to change the world in our own way.

Actually, Social Media Buttons Work Really Well

2012-05-31T00:00:00-04:00

Yesterday, an epic rant against social media buttons titled Sweep the Sleaze made its rounds on the Web. The gist: Don’t bombard your audience with buttons to share stuff on social networks. Instead, make good content that they’ll be compelled to share organically.

The advice is well-reasoned and generally correct, and was covered by The Oatmeal a few months back and by 37signals a few years back.

But this part smelled fishy to me:

But do these buttons work? It’s hard to say.

And this:

Some people probably do use those buttons. Maybe even a lot of people. And maybe you do and think I’m dead wrong about this. Maybe I am. And maybe someone needs to do some serious research to know for sure.

It’s quite easy to do a bit of research and find out just how many people use those buttons, particularly the Twitter button thanks to Twitter’s data-generous API. So I sought to answer the question:

If a website displays those annoying Twitter buttons, just how many tweets linking to it are made by people using those sleazy-ass buttons?

Here’s a gist of the script I used:

It scans through the latest 500 or so tweets that link to a particular website, and checks the source field of each tweet, looking for the clearly marked Tweet Button.

I first ran the analysis on a new website called Politwoops that was launched yesterday by my former employer, The Sunlight Foundation. The site is a good test subject because it displays the Twitter button prominently in the header, it’s about Twitter, and it went a bit viral yesterday.

Search term: politwoops.sunlightfoundation.com
Total tweets analyzed: 498
Tweets from button: 111 (22.3%)

So here we have a Twitter-savvy audience, and 22% who linked to the site did so using that flippin’ Twitter button on the website.

How about the 37signals blog, an even savvier audience made up of folks who’ve been on Twitter for half a decade? Surely, none of them will find use for that superfluous tweet button that appears on the bottom of each post:

Search term: 37signals.com/svn
Total tweets: 499
Tweets from button: 70 (14.0%)

Smaller, but still significant.

Switching audiences, let’s look at viral media website BuzzFeed. They serve content optimized for sharing on social networks, and they prominently display social media buttons on each post.

Search term: buzzfeed.com
Total tweets: 495
Tweets from button: 191 (38.6%)

Huge! BuzzFeed’s audience consists of more normal folks, not the ones who get worked up over a topic like this. The buttons are meant to remove friction, and here we see that they really appear to be doing so.

It’s time for a confession: I knew those buttons were effective well before running this analysis, because my startup Upworthy is all about spreading stuff that matters through social media. Here’s the analysis for our website:

Search term: upworthy.com
Total tweets: 499
Tweets from button: 319 (63.9%)

You can tell by viewing one of our nuggets that we really, really optimize for sharing. Yes, we have buttons that are hard to miss. But we’re also very thoughtful about crafting share text that will (a) make you want to use it to share and (b) make others want to click.

It’s true that a social media strategy simply isn’t about sticking buttons on a website. It requires high quality content, and just as importantly, packaging that content in a shareable way. It’s not sleazy to try to get your audience to share your stuff. Share buttons are well-placed reminders that sharing is an option.

You can look at a page on Upworthy and think, “Boy, that share button stuff is so offputting. People who fall for it must be gullible buffoons!” Or you can look at it and think, “This content is good and important and should be shared. Buttons that make it easier to do so are worth the pixels and bandwidth.” If you’re in the latter camp, I should mention that we’re hiring.

And if you think this piece of content is worth sharing on Twitter, please go ahead and click the button below…

Developer Happiness and MongoDB

2012-05-23T00:00:00-04:00

MongoDB continues to be the most popular of the NoSQL data stores, and it continues to generate disgruntled blog posts from former users. I’ve been using MongoDB since the summer of 2009, and my experiences have been consistently positive. In fact, MongoDB is my current default data store when building a new web app, and has been since shortly after I started using it. I understand all the criticisms being levied at Mongo, but they’re not going to change my position.

Why not? Because using MongoDB makes me happy.

Matz’s Revolution

“Ruby is designed to make programmers happy.”

Yukihiro “Matz” Matsumoto created a programming language optimized for developer happiness. Ruby wasn’t optimized for technical goals like speed or scale (insert Rails-can’t-scale joke here). It was optimized for the happiness of the human beings who would spend all day looking at it.

DHH, creator of Ruby on Rails, adopted Matz’s philosophy. The rise in popularity of Rails had a lot to do with how good the out-of-the-box experience was. Developer happiness leads to observable benefits in productivity. Here’s an excerpt from the 37signals ebook, Getting Real:

Happiness has a cascading effect. Happy programmers do the right thing. They write simple, readable code. They take clean, expressive, readable, elegant approaches. They have fun.

Left out of that quote is that happy programmers are more creative. And happiness and creativity build off another, creating a virtuous cycle of collaboration. The Django committers were already quite content with their toolset when they borrowed some ideas from the Rails core team, and vice versa. Sinatra’s creator wasn’t happy with some aspects of Rails, so he made his own thing, and shared his happiness with like-minded Ruby developers who realized they felt the same way. The creator of CoffeeScript borrowed aspects of Ruby and Python and drastically improved the experience for those of us who weren’t having much fun writing JavaScript.

The Freedom of Choice

Programmers tend to be quite passionate about the tools they use: operating systems, text editors, IDEs, languages, frameworks, libraries. And they want to switch when they’ve discovered a better tool for the job. Freedom of choice is incredibly important. But strangely for web developers, the choice for data store was always made for them: an RDBMS.

The NoSQL movement introduced genuine choice in data stores for the first time in the Web era. The differences between MySQL and Postgres and Oracle started to look small when compared to the new paradigms available. The realization that a relational database isn’t the only way to persist data is an epiphany.

Decades of RDBMS dominance led to dogma. Incontrovertible truths. Data belongs in the rows and columns of a table. Tables must be normalized. Repeating data is bad. Tables are joined. Transactions are necessary and schemas must be defined, otherwise chaos will ensue. NoSQL freed developers from those chains. And the newly liberated are happy.

Choice and Control in MongoDB

MongoDB is schemaless, which means the data schema is defined by the application, not the database. Data migrations are run to actually transform data, not merely for a simple change in the database schema. As a result, they’re run far more infrequently, which in turns speeds up the development process. If your application needs a middle_initial field, you add it to your application. MongoDB doesn’t need to be told about it.

MongoDB is document-oriented. Data isn’t confined to rows. It instead lives in a richly structured JSON document. Thanks to that, the dogma of normalization can be thrown out the window. A blog post has many comments. Well then, those comments should be stored inside the blog post, not in a separate table. A blog post has many tags. Store those tags inside the blog post and forget about the rule that says you need a join table named blog_posts_tags.

The schemaless, documented-oriented nature of MongoDB translates into a newly found sense of developer control. Your data schema mirrors the actual look of your application. You can glance at a mockup for a new feature, and quickly envision how the data will be persisted.

For example, a blog post has an author. If the application has user accounts, those authors are persisted in a users collection. The old dogmatic way of thinking dictates that in order to display that user’s name on a blog post, there must be a foreign key and a join query. But in MongoDB, the developer has control over the way the data is stored. I’d store a reference to the user, but I’d also store the user’s full name right there, inside the blog post. The blog post document is a faithful representation of what makes it to the screen. It’s not bound to the rules and conventions of the data store it’s contained in.

The tradeoff with such denormalization is that you’ll have to write a little bit of code to update all the blog posts by the author if the name is ever changed. But that’s a trivial bit of housekeeping that quickly becomes second nature. It’s much more natural than adhering to Boyce-Codd Normal Form.

MongoDB is flexible. It puts the developer in control, instead of dictating how it must be used. A breath of fresh air.

JSON

JSON is a near-perfect format for persisting data in object-oriented applications. Because it’s literally an object notation, there’s little impedance mismatch between the native objects in the programming language and how they’re stored in Mongo. ORMs were created because objects needed mapping to relational tables. That’s no longer the case with MongoDB, though object-document mappers do exist, and provide useful utilities like callbacks and validations.

JSON is a joy to work with because it eliminates so much friction. Unlike XML, it’s simple, succinct, and quickly scannable. It’s self-documenting. A JSON document represents a fully-formed object, so no parsing is necessary. It’s become the lingua franca for exchanging data on the Web. And it’s universal: every programming language understands it.

Queries and commands in MongoDB are done via calls to JavaScript functions, which take in a JSON object for the various conditions and options. This results in a magical side effect when interfacing with MongoDB from a driver in another language. Querying in any object-oriented language remains straight-forward and natural because the ideas in JavaScript and JSON are easily translated. A JSON object easily maps to a hash in Ruby, or a dictionary in Python. There’s little mental overhead needed.

Every developer I know loves JSON. It makes them happy. When using MongoDB, you get to think in JSON all day long.

Minimum Viable Features

MongoDB ships with an assortment of useful utilities that make working with it just a little bit nicer. There’s GridFS for file storage. Geospatial indexing to do lat/long calculations. Map/reduce for aggregation. There’s even a way to get full-text search capabilities using an index on an array of strings.

These aren’t full replacements for more robust solutions. They’re more like minimum viable features. There’s always a networked filesystem or S3 for file storage and PostGIS for serious geospatial work. There’s Hadoop for map/reduce and Elastic Search for full-text search. But the ability to do those things in MongoDB for simple use cases makes the development process a little more pleasant.

There are also neat little tidbits scattered around. Documents in MongoDB have a unique identifier, automatically indexed, called an ObjectId. By default it’s generated as a GUID with the timestamp embedded inside. No longer do you have to store a separate created_at timestamp. Or you can if you want. You have the choice, after all.

Unintentionally Delivering Happiness

It seems to me that the creators of MongoDB never set out to optimize for happiness. I’d guess that Redis is the one NoSQL data store where the creators actively think about developer happiness.

Instead, 10gen’s marketing has been focused on scaling. The brand itself, Mongo, came out of the word humongous. And much of the criticism of MongoDB has come from the technology choices made in the name of scaling. But the reality is that scaling any technology is difficult, and there’s no silver bullet.

The criticism that most strikes fear, uncertainty, and doubt in the hearts of developers is that MongoDB loses data. I’ve never experienced it. Turn on journaling, use a replica set in production, and make solid backups. You’ll be fine.

I build web software for a living, and I want to choose the tools that will make me happy while doing it. That’s the value most important to me, and I find myself sticking with MongoDB.

How rel=nofollow Works

2012-01-12T00:00:00-05:00

There has been a furious storm around Google’s new Search, Plus Your World features. The hubbub is centered around a complaint by Twitter that links shared on Twitter are not surfacing in the search results.

Google responded on Google+ with a terse statement:

We are a bit surprised by Twitter’s comments about Search plus Your World, because they chose not to renew their agreement with us last summer (http://goo.gl/chKwi), and since then we have observed their rel=nofollow instructions.

Before ascribing evil motives to Google, it’s important to understand exactly what rel=nofollow does, and how Twitter is currently using it. The rel=nofollow attribute is placed inside the HTML link tag. It’s an anti-spam measure that tells the Googlebot to not follow the link when crawling the Web. So <a href="http://example.com" rel="nofollow'>Click here</a> will be rendered as a clickable link to the user, but the Googlebot will not follow the link as it is crawls the web page containing that link.

Twitter.com incorporates rel=nofollow in all outbound links contained in tweets, most likely in an effort to deter spammers. The HTML for a tweet on Twitter.com looks something like this:

<div class="tweet">Check out this link: <a href="http://t.co/123"
rel="nofollow">http://t.co/123</a></div>

This means that while the Googlebot will index the text of the tweet, it will not follow any link in that tweet. Thus, Google will never know what a particular tweet links to because Twitter explicitly told it not to follow any links.

So Google launches Search Plus, and we start to see results that indicate that “John Doe shared this on Some Site”:

But we’ll never see “Jane Smith shared this on Twitter” because Twitter uses rel=nofollow on all outbound links in tweets. So even if @JaneSmith shared a relevant link, Twitter explicitly told Google not to index that link. Google will still index the text of the tweet. But without knowing what link the tweet points to, Google has no way inform us that a link was shared on Twitter.

It’s frustrating to see tech pundits severely misinterpret the facts and come to false conclusions. Here’s Danny Sullivan, who’s an expert on this stuff:

My take is that Google, by mentioning this in its post, is trying to suggest that Twitter is hurting itself by blocking its own pages using nofollow.

That’s clearly not what Google is saying. This has nothing to do with pages on Twitter.com. Twitter is telling Google to not index (outbound) links in tweets, and Google is complying. The consequence is that a link shared on Twitter will never be known to Google, and won’t be displayed.

MG Siegler perpetuates the misinterpretation:

But even more interesting is what Google said after blaming Twitter for Google screwing them: “since then we have observed their rel=nofollow instructions.” It’s subtle, but it sure sounds like Google is implying that they’re no longer crawling twitter.com — at Twitter’s request — so they don’t have the access needed to include Twitter in Search+.

But that doesn’t actually mean this at all. Sullivan also points this out in his post and also notes that it sounds like bullshit misdirection. All it means is that in some cases, Google won’t follow links from and to twitter.com. But as Sullivan notes, Google has over 3 billion Twitter pages in their index.

Once again, this is completely off the mark and insinuates all sorts of bad intention. Google is simply complying with Twitter.com’s directive to not follow outbound links in tweets it crawls, and the consequence is that there will never be “… shared this on Twitter” in the search results.

The same issue would affect any website that chooses to put rel=nofollow on all its outbound links. Usually the directive is used in places like the comments section of a blog post (to deter spam), but Twitter has decided to use it for all tweets. There is no great conspiracy against Twitter here. Google is simply following the rules of rel=nofollow.

MongoDB 2.0 Should Have Been 1.0

2011-11-07T00:00:00-05:00

No open source project has received more criticism in recent years than MongoDB. Most of the flak has revolved around technical implementation decisions made for the project, and perceptions of how the company behind MongoDB, 10gen, has used advantages gained from those design decisions in marketing their product. It seems that more blog posts have been written on why one should not use MongoDB than on why one should use it.

In a now-404ed July 2010 entry (mirror), Mikael Rogers, at the time an employee of the company behind CouchDB, wrote a highly-trafficked critique of MongoDB’s asynchronous writes and lack of single-server durability. Because of the design decisions made regarding persisting to disk, sending a hard kill signal to a standalone MongoDB instance could result in the database becoming corrupted beyond recovery. This was no good, of course. To their credit, engineers at 10gen worked to address the problem and added in single-server durability in the 1.8 release.

A few months later, Ethan Gunderson wrote a post titled “Two Reasons You Shouldn’t Use MongoDB”. Copping to an over-the-top title, he described the implications of the design decision to have a global write lock, as well as the lack, at the time, of single-server durability. His message was that developers need to understand MongoDB’s design decisions and the associated tradeoffs.

Being document-based datastores, Riak and CouchDB are the most direct competitors to MongoDB. The employees of Basho, the company behind Riak, seem to have several bones to pick with both MongoDB and 10gen. At JSConf, Basho employee Sean Cribbs was giving a presentation on Riak. When asked a question comparing Riak to MongoDB, he responded, “Mongo loses data.” UPDATE: Sean has just written a post titled MongoDB and Riak, In Context (and an apology).

A week later, in a weird self-congratulatory blog post, Basho COO Antony Falco laid into the marketing of MongoDB by 10gen. Strangely, he never once named the technology or the company while simultaneously accusing them of spreading FUD. His message: 10gen is flat out lying in its marketing of MongoDB, and it’s hurting the entire NoSQL community. Light on technical specifics but heavy on philosophical generalities, Falco’s screed seems to function as a rallying cry for his troops at Basho.

Several days ago, Michael Schurter, an engineer at Urban Airship, detailed his company’s trials and tribulations with MongoDB. Among them were the global write lock, the daunting complexity associated with running a MongoDB cluster, and the possibility of data loss.

Most recently, an anonymous ex-user, apparently compelled by social responsibility, left a Pastebin titled “Don’t use MongoDB”. Eight reasons are given, all of which rehash points made in the posts above. Most damaging are the accusations that MongoDB randomly loses data. But there’s no hard evidence given. And because of the anonymous nature of a Pastebin posting, the 10gen CTO is left to respond point-by-point on the comment thread in Hacker News with no real dialogue possible.

And so the critiques range from “understand the design tradeoffs and take appropriate steps” to “you’d be crazy to fall for the marketing hype – never ever use MongoDB”. Evaluating who is most correct is left as an exercise for the reader.

There is no doubt that MongoDB has benefitted from an aggressive marketing push. There are more MongoDB conferences held (organized by 10gen) and MongoDB books written (mainly by 10gen employees) than for the other NoSQL datastores combined. Here’s a chart from Google that compares relative search interest over the last 12 months:

I’ll close with a criticism of my own: MongoDB 2.0, released in September, should have been version 1.0. Looking back, it’s clear that certain features added post-1.0 should have been added before a 1.0 was declared:

Configurable fsync time in 1.2.
Background index creation in 1.4.
Sharding and replica sets in 1.6.
Single-server durability as an option in 1.8
Single-server durability as the default in 2.0, as well as a standalone compact command.

The single-server durability issue was a huge one. Declaring a 1.0 version well before the issue was addressed was a major misstep by 10gen. Replica sets were the second attempt at a replication solution, much improved from the previous master/slave setup. However the two replication solutions were drastically different, and the transition should have been happening between 0.X releases, not 1.X releases. In an alternate universe, holding off on the 1.0 release may have tempered some criticism MongoDB has received.

MongoDB is on its way to becoming the default datastore for web apps. At version 2.0, it is finally a stable product free of unexpected surprises. That is, a proper 1.0 release. With this stability, developers should seriously consider working with it. Its developer experience is unmatched in the world of datastores, though Redis comes in at a close second.

The focus of my next blog post will be on developer happiness and MongoDB.

Disclosure: I have given presentations at three 10gen-sponsored MongoDB conferences. At the New York City event in May 2010, I ate a 10gen-sponsored turkey sandwich for lunch. At the D.C. event in November 2010, I had pre-conference drinks on the company’s tab and received the O’Reilly MongoDB book as a thank you gift.

Apps for Failed Institutions

2011-10-04T00:00:00-04:00

It’s been a rough few years. Our once-great institutions, many built up in the aftermath of World War II, have crumbled. The public has lost faith. Take a look at this recent Gallup poll:

It seems Americans only like two things: technology and food. And they dislike nearly everything else, with Big Oil and the federal government at the bottom of the list. The Gallup poll, covering industry, omits other institutions like higher education, organized religion, and professional sports. But it’s easy to guess how those would fare in a public poll.

Americans have acted out on these frustrations. Politically, there have been wave elections for three consecutive cycles, in starkly different directions. The Tea Party tapped into the stewing anxieties of middle America. And young protesters are now occupying Wall Street, and will soon occupy city centers across the country, in response to unemployment and unchecked corporate dominance.

There are two lessons we software developers can take from the poll above. First, we chose the right profession. Not only are we highly employable, but people actually like us for what we do, at least as much as they like Mario Batali and Chipotle. But second, and most importantly, we have an incredible opportunity to remake society’s institutions through the software and services we can create.

This is a big, ambitious undertaking. The good news is that it’s already happening. Consider banks. Even before the Great Recession, online banks like ING Direct, and online brokerages like E*TRADE democratized personal finance, making it more straightforward. Their presence alone, of course, was not able to stave off financial disaster. They didn’t shake things up enough. Maybe BankSimple will help solve the problem, but it’s likely to be only the tip of the iceberg. We need to fundamentally remake the way we save and invest money, and we need more than one software company to do it.

Healthcare is another failed institution. America spends dramatically more per capita on healthcare than the other Western countries, yet Americans are sicker and die of illness earlier. And despite the rhetoric, the truth is that healthcare will remain a private enterprise no matter what party is in power. It’s time for a drastic rethinking of health insurance, preventive medicine, and personal wellness. And software will have an enormous impact on how we get there. FitBit and MassiveHealth are a start, but again, there needs to be more.

The other industries on the bottom of that list need to be rebooted as well. The bust of the housing market led to the Great Recession. How can we lead people to make responsible decisions about where they live and how they get around? Sure, we may like flying on some airlines like Virgin America or Southwest, but air travel still generally stinks. Is the educational system truly preparing students for life in the 21st century? The Khan Academy is on to something, but they could probably use some help. If a web app can help us connect to old classmates, another web app can surely help us get rid of our dependence on fossil fuel. It just hasn’t had a Mark Zuckerberg to invent it yet.

In the past decade, no institution has fallen out of favor more than the federal government:

It’s a tragedy of democracy.

Everyone has formed an opinion on what exactly is wrong with our government. And many people are working earnestly to fix it. But no one will actually succeed. Why? Because no one has had a novel idea about politics and government in decades. The same old coalitions, the same old parties, the same old advocacy groups are fighting against each other using the same old tactics. The political system is prime for something completely new and different. Something that will come in and fundamentally change the game. It won’t look like anything Washington has seen before. And it’ll certainly be built on top of software. Software that any of us could conceive of and create.

Software developers need to own up to the responsibility of making the world better. No more working on small things. We have a unique skillset that’s perfect for this moment in history. Such endeavors aren’t just the domain of the big thinkers. We don’t need to be invited to TED. All the ideas we need are right in front of us. We live them. We can fire up our favorite text editor, hack away during nights and weekends, and deploy to Heroku for free.

Ideas are out there. Time to build them. Time to change the world.

DROID 3 Review

2011-07-11T00:00:00-04:00

There are barely any DROID 3 reviews on the Web, so I thought I’d try my hand at gadget blogging. I ordered the DROID 3 from Verizon Wireless on Thursday July 7, the day it was announced, and it was delivered the next day. The DROID 3 is my third DROID. I used the original DROID from the summer of 2010 until a month ago. I’ve used the DROID 2 Global for the past month. And now I’m the owner of a DROID 3.

Here are my DROID, DROID 2 Global, and DROID 3:

The Original DROID

I love the original DROID. Released seven months before the iPhone 4, it sports a high resolution display with nearly invisible pixels. It’s well-built and solid. “It feels like a gun,” remarked a colleague upon holding it. The DROID literally saved Motorola, and propelled Android OS to the pole position it occupies today in the smartphone market. And it’s the first and only phone on Verizon to run stock Android. I was thrilled to switch to it after two years on AT&T with an iPhone 3G.

But earlier this year, the DROID started showing its age. The interface ran slow. Very slow. I rooted it, ran a custom Gingerbread ROM, and dabbled with overclocking. But the frustrations continued to mount. Battery life was abysmal. I had planned to wait for the DROID Bionic, but its launch was delayed with no firm date announced.

The DROID 2 Global

About a month ago, I bought a used DROID 2 Global on eBay. It has two notable qualities. First, it’s fast. The 1.2 Ghz CPU and dedicated GPU provide a welcome upgrade over the original DROID. Second, the DROID 2 Global’s build quality is atrocious. Unlike the original, it’s not solid. Its back is covered in cheap plastic, and the plastic front bezel is painted as fake chrome. The power button is sticky and soft. In other words, the DROID 2 Global doesn’t feel like a gun. It feels like a cheap plastic toy.

The MOTOBLUR skin is unwelcome. The typography is sloppy, particularly in the status bar. Overall, the DROID 2 Global’s CPU covers up for a lot of unforced errors on the part of Motorola. I planned to upgrade as soon as a better phone came out.

The DROID 3

Not wanting to wait for the DROID Bionic, or Samsung Galaxy S II, I pulled the trigger and bought an unsubsidized DROID 3 on launch day. Overall, I’ve been happy with my purchase. The build quality is closer to the original DROID. This phone feels solid, and the larger screen size (increased to 4 inches from 3.7 inches) is a plus. The slide-out keyboard feels great, with an additional fifth row dedicated to numbers. Because I incorporate numbers into my passwords (using password recipes), signing into my various accounts was much more pleasant with the new keyboard. The DROID 3 runs Gingerbread 2.3, a step up from Froyo 2.2 on DROID and DROID 2 Global. And its internals, featuring a dual-core 1 Ghz processor, enable a smooth user experience.

The most controversial aspect of the DROID 3 (and other Motorola phones of this generation) is it’s qHD (quarter HD at 960x540) display. A high resolution display is usually very welcome, but the PenTile screen technology Motorola incorporates is decidedly unwelcome. It’s significantly poorer than the display shared by the DROID and DROID 2 Global. Pixels on solid colors are visible. Gradients look jagged. Scrolling white text on a black background results in a hideous effect. When playing games like Angry Birds, or scrolling through home screens, there’s a shadowing that occurs with fast moving elements across the screen.

Motorola seems to have tailored its MOTOBLUR skin with the shortcomings of the screen in mind. The PenTile technology does best when displaying shades of blue, and so MOTOBLUR has ditched Gingerbread’s green and black motif for a blue one. Where Android likes use white text on a black background for Settings, MOTOBLUR has inverted that to hide the problems associated with that color combination. Speaking of colors, greens aren’t at all vibrant, like on the Samsung Nexus S. Instead, bright greens end up looking like pea soup. The pixelation on reds and yellows is especially painful. The home screen interface on MOTOBLUR is silly, so I switched to GO Launcher EX on the recommendation of Leo Laporte.

But after several days, I think I can live with the inferior qHD screen. The pixelation is only noticeable when looking for it, though I wish it weren’t noticeable at all. It’s disappointing that the original DROID, released in the fall of 2009, looks significantly better than the new DROID 3.

Battery life is yet to be determined. It was excellent for the first two days, and then rapidly dropped from 100% to 40% in the course of a few hours on the third day, despite not being actively used. I’ve disabled GSM support (the phone is global). That may help things.

One disappointment that may be a fluke: My DROID 3’s main camera stopped working after the first day. All I get is a black screen in the camera app, even though I do get a working picture when switching to the front-facing camera. I’ll have to ask Verizon for a replacement unit.

I do plan on keeping that replacement unit for the long term. The DROID 3 is a good phone, despite its screen, and it’s a worthy successor to the original DROID. To read first impressions from other DROID 3 owners, head on over to the Android Forums.

When Steve Jobs and Local Government Collide

2011-06-09T00:00:00-04:00

Soon after he unveiled iCloud at the WWDC keynote, Steve Jobs went to a Cupertino City Council hearing to present Apple’s plans to build a new campus. The city council recorded the visit, and has made it available on YouTube.

For many reasons, it’s a remarkable thing to watch:

While not as polished as his keynotes, Jobs still gives a brilliant performance. He forcefully but respectfully conveys his message: Apple wants to stay in Cupertino and build this campus. Let us do that.
His slide deck complements the serene images of the new Apple campus. When displaying text, he uses a light beige background instead of the slate grey standard for Apple keynotes (and in Apple Keynote). There’s a California cool to it all.
The main building may resemble a spaceship as Jobs mentions, but it’s also very reminiscent of the Pentagon outside Washington, D.C. The Pentagon can hold 26,000 employees. This new Apple Ring will accommodate 13,000. Apple, infamous for its secrecy, wants to build a gorgeous fortress in Cupertino.
There’s a remarkable diversity in the audience at the hearing, and within the city council itself. There are few places in America where immigrants speaking in broken English can be elected to office.
But the city council members don’t acquit themselves very well, being generally starstruck. The question about free wi-fi comes off as petty. The question about safety standards is superfluous. One councilman gets sentimental. Serious questions could have been asked of Jobs – what will an influx of 12,000 employees do to area traffic? – but they weren’t.
The implicit threat Jobs makes during his presentation – that Apple will leave Cupertino if this campus isn’t approved – is made explicit during the question period. That’s when it becomes clear why Jobs is there.

Steve Jobs has been on medical leave from Apple since the beginning of the year. But he’s still the public face of the company. When Apple needs him, he’s there. And as shown by this city council hearing, he’s damn effective.

Web-Based Slide Decks Done Right

2011-06-07T00:00:00-04:00

For software developers, the era of using desktop presentation software like Powerpoint and Keynote will soon be over. Slide decks can now be created using modern HTML5 technologies. Web-based slides make a lot of sense, especially for those comfortable writing code. They’re just web sites, so no special software is needed to view them. They’re built with code: HTML, CSS, and JS. Web-based slide decks can be programmed and interactive. And recently, tools have emerged to help developers build them.

Desktop-Style Web Apps

Using the Web to put together a slide deck is not a new idea. Back in 2007, Google integrated a presentations app into their larger Google Docs suite of tools. Much like the rest of Google Docs, the presentation app was largely inspired by productivity software offered by Microsoft, but this tool lived on the Web. Unsurprisingly, Microsoft soon followed with a Web version of Powerpoint in their Office Live suite.

In 2008, the Web framework Cappuccino was born, and with it came 280 Slides, built by former Apple engineers. Cappuccino, and the language it’s built on, Objective-J, are a complete abstraction on top of the modern Web stack. Try out 280 Slides – it’s impressive technology.

By early 2009, Web-based slide decks meant relying full-featured in-browser apps that attempted to replicate what was found in desktop clients. A paradigm shift had not yet occurred.

Slidedown and ShowOff

In the spring of 2009, Pat Nakajima released Slidedown, a novel approach to making a slide deck for the Web. The toolchain introduced was familiar for web developers: Create content using Markdown and code. Use the command line to generate the slideshow as a single HTML file. Post it on a web server. Slidedown relies heavily on jQuery for slide transitions and animation.

GitHub’s Scott Chacon made a splash in early 2010 with ShowOff. With a workflow similar to Slidedown, ShowOff generates a presentation as a self-contained Sinatra app. In the era of Heroku, hosting the Sinatra app is trivial. Today, many presentations at Ruby conferences use ShowOff.

Embracing HTML5 and CSS3

Starting in 2010, HTML5 and CSS3 were clearly becoming the Next Big Thing on the Web. Google’s Chrome team launched HTML5 Rocks*, an all-encompassing resource for HTML5 technologies. One of the most eye-catching parts of the site is a fully HTML5 slide deck about (what else?) HTML5. Breaking from the reliance on JavaScript in Slidedown and ShowOff, Google opted to use CSS3 to accomplish effects that were once cumbersome to implement in the browser. Maximizing the potential of modern browsers, this was clearly the way forward.

In May 2010, Adam Zaptletal released Landslide, an HTML5 slide deck generator that borrowed heavily from Google’s original HTML5 Rocks presentation. I used it earlier this year for a presentation to journalists on HTML5.

But Google wasn’t done. Last month at their annual developer conference, Google I/O, all presentations by the Chrome team used an I/O-themed HTML5 slide deck. While not yet formally announced, Google has actually released the slide template as an open source project, called htm5slides on Google Code. Googlers Eric Bidelman and Arne Roomann-Kurrik went a step further for their HTML5 Wow presentation, and released that code as its own open source project. HTML5 Wow features bleeding-edge stuff, like WebGL and the File API.

Mozilla has been getting in on the fun as well. Technical Evangelist Paul Rouget recently released DZSlides, with the accompanying slide show HTML5 in the Wild. DZSlides enforces almost no theming, and importantly, includes support for embedding slide decks. Presenters know that the ability to embed their slide deck through Slideshare is a major reason to stick with traditional formats like .ppt, .keynote, or .pdf. DZSlides is a pure HTML5 solution that replaces Slideshare’s Flash embed. It should be noted that as of this writing, DZSlides doesn’t render correctly in the latest Webkit-based browsers. The latest Firefox and Aurora builds do support the correct behavior.

Today, solid options exist for creating slide decks in HTML5. With the current tools, using code to craft a presentation is easy, and frankly, more fun than using Powerpoint or Keynote. And there are no barriers to entry for inventing a custom solution.

* I wrote an article about WebSockets and EventSource published on HTML5 Rocks.

Leveling Up for JSConf and NodeConf

2011-03-21T00:00:00-04:00

Both JSConf and NodeConf are taking place in Portland during the first week of May, a mere six weeks away. Having been to both previous JSConf US conferences, I have a good idea of what to expect: bleeding edge web technology presented by programmer pioneers. I’ve been to an uncountable number of developer conferences in the past four years, and I’ve picked up a lot of good stuff along the way. But JSConf has been the only one that really expanded my horizons.

Confession: I usually leave JSConf overwhelmed. There’s too much new material for me to absorb effectively. This year, throw NodeConf into the mix, and I fear even more goodness will just go over my head. As a Ruby on Rails developer, I don’t do enough day-to-day JavaScript to call myself an expert. I want to change that between today and May 1.

I’ll be going on a self-imposed JavaScript training regimen these next six weeks. I’ll mix in some comprehensive exercise, some good nutrition, and hopefully when game day approaches, Coach will tell me I’m ready. Here’s the plan:

Build a useful web app using Brunch on the frontend and Express on the backend. This exposes me to Node, CoffeeScript, and many more important JavaScript libraries.
Figure out how to use Socket.IO to keep it real(time).
Explore frontend libraries Knockout and SproutCore.
Truly understand what Brendan Eich talks about in his A Minute with Brendan podcast.
Catch up on other podcasts like JSConf Live and yayQuery.
Pour myself a glass of wine and watch Paul Irish’s two videos on the jQuery source code.

What have I missed? How do you plan to level up in preparation for JSConf and NodeConf?

This Day in History, as told by Gmail

2011-02-26T00:00:00-05:00

My colleague James Turk looked back at his recent life as told by Gmail. I’ve decided to do the same for this day, February 26.

2005

Doing the advanced search in Gmail, I noticed that when given a date to search on, Gmail will actually return email from that day along with email from the day before and the day after. This is a welcome feature, as I didn’t send or receive any email on this day in 2005. I received a mere seven emails during that three day time window.

It was my senior year of college at Georgia Tech. I received an email from the Georgia Tech College Democrats president, who became my girlfriend three weeks later (and still is). I registered for Dice.com and CyberCoders, two IT industry job search sites.

A recruiter from Microsoft scheduled a phone interview with me. Two months later I was invited to Redmond for on-campus interviews with the Office Online operations team. I was twenty-one years old, and Microsoft paid for my first ever rental car (I would not drive another rented car for five years). The Emerald City was sparkling and gorgeous on the April Sunday I arrived and did the tourist thing on Microsoft’s dime.

The next morning, I got ready in a Redmond hotel room watching the white smoke announcement of Joseph Ratzinger as the new Pope. I completely flubbed the marathon session of interviews. It was during my third interview of the day when I realized I was simply not qualified for the job, having no experience with IIS or .NET. I left Seattle knowing I wouldn’t be offered a job. That summer I took a developer position with Democracy for America(DFA) in Vermont, for a salary that was probably a third of what I would have earned at Microsoft.

2006

I was now braving a winter in Burlington, Vt., as the lone developer at a small PAC. I was testing out a new contribution system I was building for DFA, a mini-ActBlue, making small donations through the system to the first candidate we’d raise money for, Ciro Rodriguez. DFA later scrapped that system and just used ActBlue.

In my search for a Mac text editor, I purchased skEdit, which was my editor of choice until I purchased TextMate in May. I also signed up for a DreamHost account to host my personal website.

Nearly a year into our relationship, most of it spent hundreds of miles apart (she was still in Atlanta at Georgia Tech), a good percentage of the 36 email conversations and chat logs were with my girlfriend.

2007

I moved back to Atlanta in early January. I exchanged several emails with Jerome Armstrong concerning a new campaign platform I was to build in Rails. I ended up living in Atlanta for two years, working remotely for Jerome’s consultancy based just outside DC.

At this time, I was also talking to folks who wanted to draft Al Gore in the presidential race. I had registered ForGore.com and launched an ineffective website a few months later. Email traffic was still reasonable, at 47 conversations for the three day period.

2008

Fully into my consulting gig in an election year, I was inundated with client requests. I was doing Salesforce.com work for the U.S. Senate campaigns of John Kerry and Mark Warner, and building a social network for the SEIU. I was harried and overwhelmed with the workload, and the 117 conversations in this period reflected that. I also bought a first generation iPod Touch for my girlfriend, which was my third anniversary present to her.

2009

My girlfriend graduated from college after the fall semester, and we moved together to Northern Virginia. My remote work for the consulting agency was now on-site work and I was an actual employee. While there were 222 conversations in Gmail for this period, that was a function of automated email. Our team was using Pivotal Tracker and practicing continuous integration, and the system sent out emails after every commit to GitHub with the build results. I named our CI server Phyllis Schafly so that broken test suite notifications came from her.

I was subscribed to the Sunlight Labs mailing list, which was seeing loads of activity around the first Apps for America contest. My app entry was a precursor to Poligraft, a tool I would build a year and half later as an employee at Sunlight. Also, I received an Eventbrite notification about the first ever TransparencyCamp.

GMAT prep books I had checked out of the library were due. I took the GMAT a few weeks later, did horribly average on the math section, and decided I didn’t want to get an MBA after all. I purchased the Beta of The RSpec Book, which wouldn’t be published until December 2010.

2010

Now working for Sunlight, email load was a bit lighter at 138. The Sunlight staff was abuzz with the positive reaction to our first ever production of Sunlight Live. Today in 2011, I’m working on a follow-up project to Sunlight Live: a generalized platform for reporting on real-time events called Datajam. I was working on the website for Politiwidgets, and bought an ebook from Manning called iPhone and iPad in Action. This was a new title for the book. It was first called iPhone in Action, and today it’s called iOS 4 in Action.

The Strange Populist Outrage Against Google

2011-02-02T00:00:00-05:00

The debate over the great 2011 clickstream controversy has played out in phases that seem hilariously obvious in retrospect. First, outrage combined with confusion over what exactly happened. Then, a rebuttal replete with generalities that only hints at an explanation. Third, a less outraged rebuttal to that rebuttal, that strangely waits to the last paragraph to make its point. And now, a populist revolt against the original Google claim, pointing out perceived hypocrisy in Google’s position.

Let’s look back at the facts, ignoring the rhetoric for now:

Google suspected Bing of stealing its search results, and conducted some experiments. In a trial of 100 search terms, they were able to prove that Bing did somehow manage to copy Google results in seven (out of 100) cases.
Google posited that clickstream data from IE or Bing Toolbar was causing this to happen. That is, users who opted in to share their data anonymously with Microsoft were sending back data on what they typed, which sites they visited, and what they clicked on. One website is often Google, and so Google results were effectively brought into the Bing search algorithm.
Harry Shum of Bing more or less verifies that this is what happened in his blog post and appearance yesterday morning.

So Bing is effectively using some Google results in its search engine. It’s not direct copying, but the effect is the same. This is certainly wrong, though it’s not outright “cheating” as was originally posited. Bing is using the fruits of the core offering of its direct competitor to make its own core offering better. Bing should immediately blacklist any clickstream data coming from Google. It’s just that simple.

But it’s not that simple, because the current populist revolt against Google seems to believe that Google has no right to be upset. The thinking goes: Google uses the the Web’s shared work, so they have no right to cry foul when Bing uses their search results. This is all part of the game!

That position is even more indefensible than Bing’s. Search engines provide a service to users: their own unique take on what links should be presented to a user based on a search phrase. Providing value on top of existing work is a legitimate way to do business. That’s what Google, and all search engines, do. It is not legitimate, however, to take existing work and simply pass it off as your own. That’s what Bing is doing, even though it was unintentional.

Journalists translate events into words. If Journalist A takes Journalist B’s reporting on a certain event and passes it off as his own without citation, we’d all agree that would be wrong. Even if Journalist A didn’t know he was copying work, it would still be wrong. And we don’t fault journalists for observing the world as it is and profiting off the reporting of that world.

Bing is using work done by Google’s core offering for its own core offering. That’s wrong and needs to stop.

Cr-48 Chrome Notebook with Decals

2011-01-24T00:00:00-05:00

Today I received a present from Google in the form of the Cr-48 Chrome Notebook. Shipments of the Cr-48 stopped just before Christmas, according to various reports on Twitter and The Chrome Source. It appears they’ve started back up, and now included in the box are a set of decals to decorate the drab black top of the notebook. There’s one large decal meant to cover most of the top cover, and a set of small stickers for a touch of personalization:

The hardware seems the same from previous reports. The touchpad is still a bit flaky, especially when attempting a two-finger scroll. I’m a fan of relying totally on the keyboard anyway, so Vimium becomes even more indispensable.

Protect Yourself with Password Recipes

2010-12-14T00:00:00-05:00

Are you now more wary about your passwords after the Gawker incident? Good. The lesson is simple: if you supply a password to a website, expect it to get stolen. Malicious hackers succeed at getting user passwords all the time. And user data breaches are almost never as public as with what happened to Gawker. But if you follow two simple rules, you’ll be able to protect yourself from greater damage when the inevitable FAIL happens.

Rule #1: Don’t supply a password if you don’t have to. Many sites today offer registration via pre-existing accounts with Facebook, Google, or Twitter. Take advantage of those. UPDATE: In February 2013, Twitter itself was compromised, but the method described below would have protected you against that attack.

Rule #2: When you do have to supply a password, you should always create a unique password for every single site. That way, if your password gets stolen from one site (and it will), only your account with that site will be compromised. Furthermore, your unique password should not contain any word in any language. It should be a nonsensical series of letters, numbers, and punctuation. Otherwise, attackers can guess your password through dictionary or brute-force attacks.

The lazy way to accomplish Rule #2 is through a password manager like 1Password or my personal favorite, LastPass. I prefer LastPass because it’s a simple browser extension and it’s free, while 1Password is a $40 desktop app. Both will generate unique, one-time passwords for your accounts, and store them securely. Of course, you still need a “master password” for those apps.

But I prefer managing passwords the old-fashioned way: using my mind. I don’t like completely relying on an app I may not have access to on the various devices I want to use.

So how do you generate a unique yet memorable password that’s unguessable by man and machine? Come up with your own password recipe for constructing a password based on the website’s domain name. To make this concrete, walk through this example with google.com:

There are 6 letters in “google” (the secondary-level domain), and 3 letters in “com” (the top-level domain). Multiply 6 by 3 to get 18. 18 is the first part of the password.
Now come up with a phrase you’ll remember, like “All’s well that ends well”. Take the first letter of each word to build the second part of your password, awtew. Note that this step has nothing to do with the website’s domain name, which is completely fine.
The next part of the password will be based on the “google” part of the domain name again. This time, we’ll take all the consonants and capitalize them: GGL.
Last, let’s subtract the 3 letters in “com” from the 6 letters in “google”: 3

So our password for Google would be 18awtewGGL3. Our password for Yahoo.com would be 15awtewYH2, while LifeHacker.com would have the password of 30awtewLFHCKR7. But of course, come up with your own recipe, don’t use the one above.

The idea is that you memorize four or five steps to build a unique but reconstructable password. Here are more ideas for individual steps for your password recipe:

Choose different punctuation for different top-level domain names. % for .com, @ for .org, ! for .net, ^ for everything else.
Pick out the first or second letter/consonant/vowel from the secondary-level domain name, and capitalize it or repeat it a couple times.
Apply math in different ways. Multiply the number of vowels by your lucky number, then add 13. Or use division to come up with a remainder.
Come up with a nonsense word to always use, but make sure it’s not in the dictionary or discoverable via Google. This is effectively the same as taking the first letter of each word from a longer phrase.
Determine some universal rules that will make guessing your password even less likely. If the letter “a” is to be used, always use a capital “V” instead.

So that’s all there is to it: Come up with your own recipe for constructing passwords based on the domain name of the site. Make sure it’s repeatable and consistent – your formula should always produce the same password for the same domain.

I’ve been using this technique since 2005, when Steve Gibson recommended it on episode #5 of Security Now. Steve called it the “Personal Password Policy”, but I like the phrase “password recipe” better. Speaking of Steve, watch him review LastPass earlier this year. I personally use LastPass to remember my passwords, but I don’t let it generate passwords for me – that’s what my password recipe is for.

Reading Code is Good, Writing Documentation is Better

2010-10-15T00:00:00-04:00

Yesterday, John Nunemaker wrote a widely disseminated blog post on reading code. He argues that instead of searching Google or asking a mailing list when trying to figure out how to use a software library, developers should open up its codebase and just start reading. John is 100% right. In fact, I too once opened up the Typhoeus library to figure out which options were available to me. Having a high level of code literacy – when it’s not your code – is essential to our craft.

But considering that point alone ignores what should be an important social contract when it comes to libraries. Software libraries, whether they be called gems or packages or jars, are a unique kind of software: their users are software developers, not actual end-users.

And we software developers are busy. We have work to do. We hate wasting time (when we don’t choose to, that is). If I have to open up your codebase while working with your software library, then something went wrong while using your software library.

As a library author, you should not expect your users to read your code to understand how to use your code. Instead, you should write thorough and clear documentation. A single blog post isn’t enough, and neither is relying on sparsely written inline documentation.

In this age of GitHub, your library should be fully documented in a README. Tom Preston-Werner advocates for Readme Driven Development: write your Readme first, before a single line of code. Every object, method, and parameter that is meant to be available to the users of your library needs to be described. Critically, there should be no ambiguity. If you’re writing inline documentation, feel free to repeat most of it in the Readme. If you practice test-driven development, you can usually copy and paste bits of your test cases into your Readme.

Beyond the Readme, consider setting up a dedicated website. If your library is sufficiently complex, breaking the documentation down into distinct sections and pages may make more sense. Some projects that practice this are Mongoid, DataMapper, and of course, Rails. Though they’re technically command-line tools, both Bundler and RVM have terrific documentation too. Yehuda Katz likes to keep a completely separate repository from the codebase for a documentation site.

Reading code is sometimes necessary. Maybe a bug or error has occurred, and the stack trace tells you it’s happening from inside the library you’re using. There are at least two possible causes here. You may not be using the library correctly, or there may be an actual bug with the library. Good documentation, along with helpful error messages, would help you avoid the first case. The second case–a genuine bug–is actually the more forgivable mistake by the library author. One never intends to introduce bugs, but one does make the decision to not write documentation.

To be clear: reading code is an important skill to have. Developers should not be afraid to do it. But if I were asked to choose between two programmers, one who writes good documentation but is awful at reading other people’s code, and another who never writes documentation but is great at reading code, I would favor the first programmer. Being able to read code is good, but writing documentation is better.

A Rubyist's Guide to the Twitter OAuth Dance

2010-09-01T00:00:00-04:00

Twitter’s switchover to OAuth is complete, and developers are realizing what a huge pain in the ass it now is to authorize with the Twitter API. No doubt about it: OAuth is complicated. For developers who just want to quickly work with the Twitter API, the OAuth dance is tedious and complicated. Simple HTTP authentication with a username and password has been replaced with four steps:

Register a Twitter application, which is an OAuth consumer
Use the consumer key and secret to generate a request token and secret
Use the request token and secret to generate an access token and secret
Use the access token and secret to access the API

The problem with OAuth 1.x (OAuth 2.0 will solve these shortcomings) is that it’s geared towards the use case of external, user-centric applications. For a developer who wants to write a quick script to access the API, or integrate the API into an existing app, OAuth 1.x is overkill. There’s documentation from Twitter on all of this, but when I tried it out for myself the other night, it was an exercise in frustration, and I found myself following documentation from various places to get it working.

Registering the App

First, register an app, called a consumer in OAuth parlance, at dev.twitter.com/apps. I’m using a single “Utwilities” app for all my one-off uses of the API.

I’m not sure if creating one app for multiple uses violates the API terms of service, but I don’t want to go through the ceremony of registering a new consumer every time I want to use the Twitter API.

Skipping the Dance

Steps 2 to 4 outlined above are commonly referred to as the OAuth dance, where the consumer app generates a one-time request token that’s then used to generate the user-specific access token. The user-specific access token (and secret) is used in largely the same way as the username/password combination of that now bygone era. The great news is that Twitter lets us skip these steps by automatically generating an access token for the same user that registered the app. You can view your access token and secret in the sidebar of the app page:

Things get really easy once you have the access token, especially with the Ruby Twitter gem. As documented, you’ll need to instantiate and authorize an OAuth client, then use that to instantiate a Twitter client.

require 'twitter'
oauth = Twitter::OAuth.new('consumer token', 'consumer secret')
oauth.authorize_from_access('access token', 'access secret')

client = Twitter::Base.new(oauth)
client.update('Tweeting with OAuth access!')

Joining the Dance

There will be times, however, when you need to authorize access to the consumer app for a Twitter user that isn’t the app owner. In that case, you’ll need to dance, and this is where things get hairy.

The important thing to know is that there are currently two ways a user can grant OAuth access. The first, and most popular, method is based on webpage redirects. You’ve gone through this process if you’ve ever authorized a third-party app that uses the Twitter API on your behalf. It’s great for end-users, but I won’t be covering that here.

The second method is better for developers: granting OAuth access when a web redirect isn’t possible or practical. This is necessary when generating a key as a developer – when the code will only be needing one access token and doesn’t need to support Twitter authorization for end-users. Authorization in this manner requires, believe it or not, a PIN number.

Using your app’s consumer key and secret, instantiate an OAuth client. Then ask for a request token.

oauth = Twitter::OAuth.new('consumer key', 'consumer secret')
request = oauth.request_token
request_token = request.token

Use the request_token to visit a URL that gives you back a PIN number:

https://twitter.com/oauth/authorize?oauth_token=REQUEST_TOKEN_HERE&oauth_callback=oob

When you open that URL in your browser, substituting the request_token appropriately, you’ll be asked to authorize and then will be presented with a PIN. You can now ask for an access token:

oauth.authorize_from_request(request.token, request.secret, 'THE_PIN_HERE')

# Save these to your database or keep in a config file
access_token = oauth.access_token.token
access_secret = oauth.access_token.secret

# Now you can use the client
client = Twitter::Base.new(oauth)
client.update('Tweeting after generating an OAuth PIN!')

OAuth 2.0

OAuth 2.0 promises a much easier workflow, but Twitter doesn’t support it yet. See Michael Bleigh’s slides from RailsConf for details. For now though, we have to dance.

The Present Future of OAuth

The Computer for the Room

2010-08-23T00:00:00-04:00

The computer on your desk. The computer in your hand. And the computer for the room.

The definition of the personal computer is expanding. Ever since the first PCs appeared on the market in the 1970’s, computers have been things we put on our desks. Even the laptop, renamed the notebook after too many lawsuits, works best on a desk. We interact with these computers with a keyboard and some sort of pointing device – either a mouse or a trackpad.

Beyond technical and physical considerations, the definition of a personal computer encompasses several important social constructs. Personal computers are general purpose, not single purpose. They run software that can be installed by the user – though the manufacturer will always pre-install some software. Most importantly, there are few barriers to entry to writing software for that computer. Manufacturers of personal computers actively promote the development of new software for their machines. Recently, having a personal computer has meant the ability to connect to the Internet – specifically the Web.

And until about 2007, that’s what a computer was: the machine on your desk that ran software, most importantly a web browser. Phones, video game consoles, set-top boxes, and graphing calculators were electronic devices, not computers.

These days, Apple is routinely excoriated for their App Store policies. But they don’t get enough credit for how much they changed the ecosystem of personal computing in 2007 and 2008 with the iPhone. Apple recognized that the Web democratized software development. Anyone who could afford $10 a month for Dreamhost and learn how to develop on the LAMP stack could create software that ran on every personal computer that was connected to the Internet.

Through Mobile Safari on that first iPhone in 2007, Apple made the full Web (sans Flash) available on a machine that wasn’t a desktop or notebook computer. Anyone who could program a LAMP application could build software for the iPhone. However, it became clear that the Web in 2007 was not quite there yet in terms of standards and browser technology. So native iPhone development was introduced in 2008. Once again, Apple chose a drastically different path than their predecessors – a path in favor of an ecosystem of software development. For $99 a year, anyone could write software for the iPhone.

When compared to the wide open Web, or the no-approval-needed Android Market, Apple’s App Store is correctly regarded as a walled-garden. But it’s an easy walled-garden to get into – especially when compared to software development for RIM or Microsoft phones, or for video game consoles or set-top boxes.

Because of Apple’s moves, the mobile phone industry is no longer about phones. It’s about personal computers that can make phone calls and receive text messages, among many other things. Similarly, the television industry will no longer be about TV shows and movies. It will be about personal computers that deliver television content, among many other things.

Android and iOS, operating systems designed for the computer in your hand, are making their way to the computer for the room. Google has already announced their Google TV initiative: Android computers running a full version of the Google Chrome web browser. Watch the announcement video from Google I/O for all you need to know about Google TV. Similarly, Apple will likely rebrand their closed AppleTV device as iTV, with full support for iOS and the Web. Kevin Rose recently blogged about how disruptive the iTV will be.

User interface is a point of contrast when considering the three kinds of computers. The computer on your desk uses a keyboard and mouse (or trackpad), something unlikely to change. As a workstation, nothing beats the one-hundred physical buttons on the keyboard. The computer in your hand uses touch, specifically multitouch gestures like pinch and swipe. Touch has been a revelation because it’s a natural interface. We already interact with the world around us through touch, but we need high school classes dedicated to learning how to type on a keyboard.

Disappointingly, Google seems to think a wireless keyboard with a trackpad is the way to go for Google TV. The rumor mill is predicting that iPhones and iPads will act as a remote control for the iTV. But there are far more appealing alternatives for the computer for the room. The first is voice. Like touch, it’s a natural interface. Recently, Google launched sophisticated Android voice actions that work surprisingly well. The problem with voice commands is that no one wants to use them in public. It looks silly. But in the privacy of a room, voice commands make a lot of sense. Hopefully voice is available on the Google TV platform.

A third natural interface is gesture and facial recognition: talking with our hands and facial expressions. Here, Microsoft has a huge head start with Kinect for XBox, and we have yet to see if Google and Apple will develop similar technologies. Moving away from the keyboard and mouse combination and towards natural user interfaces will be key for the widespread adoption of computers for the room.

Interestingly, the iPad already works both as a computer in your hand and a computer for the room, just as long you can touch it:

Admittedly, the phrase “computer for the room” is a bit awkward. It doesn’t have the same flow as “mobile computing” or “desktop computing”. So how about “console computing”?

The term console is already overloaded, but it may be appropriate in this case. Nintendo and Playstation systems are already referred to as consoles. The stereo and climate system in a car is controlled by the center console. While computer programmers and system administrators think of a command-line interface when they hear the word console, most people think of a console as a device that’s not quite a computer. It won’t be much of a stretch for people to realize that yes, that thing sitting under their TV, is in fact a computer. And having the Apple and Google brand on those consoles will help change our mindset.

Personal computing for this decade will look very different than what we’ve been used to so far: The computer on your desk. The computer in your hand. And the computer for the room. While the first computers for the room will be geared towards home entertainment, they will evolve into shared computers embedded throughout our lives. They will be in our bathrooms and kitchens, in our office conference rooms and doctors’ waiting rooms. In ten years, we’ll wonder how we ever lived without them.

Ten Ruby Conferences (and Three Camps) in Eight Weeks

2010-07-19T00:00:00-04:00

In a fifty-two day stretch, beginning on Thursday, August 19, and ending on Saturday, October 9, the Ruby community will self-organize in a way that no other technical community has ever done. Ten conferences and three camps will take place, mainly in the US.

Here’s the list:

Aug. 19-21: CapeCo.de - Brewster, MA
Aug. 21: RS on Rails - Porto Alegre, Brazil
Aug. 26-28 Lone Star Ruby Conference - Austin, TX
Aug. 27-29: RubyKaigi - Tsukuba, Japan
Sep. 3-4: RubyHoedown - Nashville, TN
Sep. 9-10: [Ruby Web](http://rubywebconf.org/) - Snowbird Resort, UT
Sep. 11: Windy City Rails - Chicago, IL
Sep. 17-18: Golden Gate Ruby Conference - San Francisco, CA
Sep. 17-19: Ruby DCamp - Northern Virginia
Sep. 25: SunnyConf - Phoenix, AZ
September: RailsCamp New England - Vermont (dates and location TBD)
Oct. 1-3: JRubyConf - Columbus, OH (replaces erubycon this year)
Oct. 6-9: Mountain.rb - Boulder, CO

Credit goes to the organizers of this past weekend’s excellent Ruby Midwest for compiling most of the list above.

And that’s not all for the year. Later in October, there’s Acts As Conference in Orlando, and of course, RubyConf in New Orleans in November. Also, there will be a RailsCamp Perth in Australia from November 12-15. Add to that a smattering of DevNation events across the country geared towards Rubyists and like-minded developers. For nearly everyone in the US, there will be an event within a drivable distance.

The Importance of Meetup

So what does all this unprecedented activity mean? Yes, this proves that the Ruby community is stronger than ever, and continues to grow. In most major cities, and in many smaller ones, evening Ruby Meetups are a common occurrence at least once a month. Rubyists meet, watch a couple of presentations, and have dinner or drinks together. In some places, these monthly meetings have been augmented with lunches, hack nights, or other gatherings. There’s nothing particularly unique to Rubyists about any of this. Meetup.com has been used for this purpose for nearly a decade – providing a service that helps witches, knitters, Tea Party activists, social media consultants, web designers, and software developers to self-organize.

But what’s striking about the list above is that no other technical community today could pull off thirteen organized events in the span of two months. Ruby developers today are a small subset of software developers. There are many, many more programmers who write Java and C# for a living. There are probably more Pythonistas than Rubyists. But there currently isn’t, nor has there been, a culture of self-organized regional conferences and camps for developers in other languages like there is for Ruby.

The Big Shows

Why has this phenomenon happened in the Ruby community, and nowhere else? Developer conferences are not at all unique to Ruby. Apple’s WWDC, Microsoft’s PDC and Tech:Ed, and Sun’s JavaOne (now Oracle’s OpenWorld) are well-established and heavily-attended. But huge international conferences need huge budgets and big name sponsors, which forces them to be more like trade shows than developer events. It should be noted that there’s only one Ruby conference with a “show floor”, and that’s RailsConf.

Also, Ruby has never had a large corporation to champion it. No Sun, Microsoft, or Apple to bankroll events. So the build it quickly, think small mindset that attracts many to Ruby is carried over to its community’s conferences. To be fair to Java developers, a group of them did create the No Fluff, Just Stuff series of smaller, regional conferences as a direct response to the trade show nature of the big Java shows. But even those events are driven by central planning, not grassroots enthusiasm.

How about Python?

Python, like the other languages, does have its big conferences. There’s PyCon and there’s DjangoCon. Similarly, the big conferences in the Ruby world are RubyConf and RailsConf. The Python and Ruby communities share many of the same values and attract similar kinds of developers. Many cities have Python meetups. The biggest tech company out there, Google, loves Python. It’s widely used in academia. But there is no regional Python conference ecosystem.

While the conception and production of regional Ruby events are organic, they are not without some coordination. Some time in 2006, when Ruby was exploding in popularity thanks to Rails, the wise old sages who had been organizing RubyConf since 2002 made a concerted effort to encourage local Meetup leaders to organize their own conferences. A few took the bait, others then sought to emulate them, and every year since more and more regional conferences have sprung up. The RubyConf organizers, a non-profit called RubyCentral, even have a grant program and have set up a mailing list for would-be regional conference organizers.

Tying it Together

Ten conferences and three camps in the span of eight weeks. Credit for that falls on the Ruby community as a whole, and especially to those brave few who’ve decided that organizing a conference was a good way to spend their free time. But credit also goes to Ruby Central for planting the seeds for self-organized regional conferences.

Other developer communities should be taking notes. There’s no reason in particular, it seems, for this to not be possible with Javascript, Python, Clojure, Scala, or any other “progressive” language. But it hasn’t happened. The thriving open-source platforms like Ubuntu and Android are hampered by having central authorities in Canonical and Google. Maybe the regional conference phenomenon will always be unique to Ruby.

How the Ruby community got to this moment is utterly fascinating, at least to me. But beyond the reasons above, there must have been other contributing factors. What have I missed?

Mouseless Monday #3 - Divvy

2010-06-14T00:00:00-04:00

Window resizing is one of the more inefficient actions in modern desktop operating systems. It’s particularly bad on Mac OS X. That little green button at the top left of every window doesn’t really do anything useful, and dragging to resize requires the user to hone in on that little handle at the window’s bottom right corner.

Enter Divvy for Mac OS X, the best window resizer ever created. It’s a mouseless dream. Bringing it up is easy with a quick key combination. Once shortcut keys are set up, resizing a window is a simple keypress. It works in concert with the operating system in an elegant way.

Go to the Divvy site and watch the See Divvy in Action video right now. It explains everything you need to know.

Shortcut Heaven

By default, the user can resize windows by dragging the desired shape on Divvy’s mock up of the desktop. But keyboard shortcuts are Divvy’s real strength, and they’re completely customizable by the user. To bring up Divvy, I’ve set the key combination to be Ctl+Cmd+Space, a close analog to the Ctl+Space combo I use for Launchbar. My shortcuts for window sizes are:

There’s the fullscreen shortcut and then three complementary pairs: left/right, top/bottom, dominant/sidebar. I use the left/right and top/bottom combinations for coding, and the sidebar shortcut works well for chat and Twitter apps.

Bryan Liles on Mouselessness

I participated in Ignite RailsConf last week, and it was a great time. Bryan Liles gave a talk on going mouseless on the Mac. Video hasn’t been posted yet, but his slides share some great resources.

A few months back, Bryan recorded a detailed screencast and wrote a blog post about his mouseless techniques:

Bryan’s presentation style is certainly one of a kind. To catch everything, I suggest referring back to his blog post and taking notes on points to explore as you watch the video.

How We Use MongoDB at Sunlight

2010-05-28T00:00:00-04:00

This post originally appeared on the Sunlight Labs blog.

Last week, David and I attended MongoNYC, a one-day conference focused on MongoDB. We like Mongo here at Sunlight. We like it a lot.

Working with Mongo, it’s become clear that it’s a more natural way to store data. We primarily use Python and Ruby, and because Mongo allows us to think in JSON, everything tends to just click. JSON documents are close enough to objects in Python and Ruby that mapping between application and database becomes almost effortless. Mongo has really shined in two specific use cases: as a datastore for a resource oriented web service, and as a datastore for results from scraping a web site.

Powering a Resource Oriented JSON API

We currently have two APIs powered by Mongo, and we’ve blogged about both recently: the National Data Catalog API and Drumbone.

A great thing about Mongo is that it allows you to store data as you’d naturally want work with it, particularly through the use of Embedded Documents. Looking at an example entry on the National Data Catalog, it’s easy to determine the Mongo document schema just by looking at the page design. We have a single document representing the earthquake data source, and it has fields that represent the title, homepage, and documentation URL. Embedded in the document are collections representing the downloads, ratings, and comments made for this entry. There’s no need to think about tables and joins – a catalog entry can be thought of as a JSON document.

It’s easy to create a resource oriented web service with a framework like Sinatra, but for the National Data Catalog, David wrote a framework called sinatra_resource, that provides a nice DSL for exposing Mongo documents as resources.

For the Drumbone API, Eric exposed Mongo’s ability to reach deep into embedded documents with dot notation through the query string. So, let’s say you want to grab only two of the fields from the earmarks sub-object:

// ?sections=last_name,first_name,state,earmarks

{legislator: {
  last_name: "Lee",
  state: "CA",
  first_name: "Barbara",
  earmarks: {
    average_number: 20,
    total_amount: 10000000,
    average_amount: 22994535,
    total_number: 28,
    last_updated: "2010-03-18",
    fiscal_year: 2010,
  }
}

You would modify the parameter string to use dot notation like so:

// ?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number

{legislator: {
  last_name: "Lee",
  state: "CA",
  first_name: "Barbara",
  earmarks: {
    total_amount: 10000000,
    total_number: 28
  }
}

The ability to ask for partial responses from an API is a huge win when it comes to speed and efficiency, and Mongo makes supporting it dead simple.

Storing Results from Parsing and Scraping

Not coincidentally, both the National Data Catalog and Drumbone aggregate lots of data from disparate sources. For the National Data Catalog, we scrape other catalogs like data.gov and the DC Data Catalog to create our central catalog. Drumbone uses API data from GovTrack, TransparencyData, and USASpending.gov, among others.

Using a relational database to store data from different sources with divergent schemas usually means creating a lot of small, single-purpose tables, or creating one generic key/value table. With Mongo, schema isn’t enforced on the data level, making parsing and scraping much less tedious when it comes to data storage.

We’re converting the Fifty State Project to MongoDB. We’re scraping the legislative web sites for all fifty states, and while we have a baseline schema for legislators, bills, and votes, we also want to preserve all valuable but unique data that a given state may provide.

Lightning Talk

Here’s video of a lightning talk I gave at MongoNYC, with the slide deck below it. I go over what we do at Sunlight, and how we use MongoDB in our projects. Please excuse the sound quality, as I was not wearing a microphone and the walls to the room were thin, so at some points you can hear the session next door in the background.

</embed>

Civic Hacking @ MongoNYC

Stop Using the text-indent:-9999px CSS Trick

2010-05-25T00:00:00-04:00

The usage of text-indent:-9999px to display a site logo or other image while hiding text should be avoided. It’s a technique used by web spammers trying to game search engines like Google, and is considered spammy behavior. Instead, use an <image> tag and put the text inside its alt attribute.

At TransparencyCamp two months ago, I attended an excellent session on Searchability by Vanessa Fox. This spurred me to dive deeper into the topic. I concluded that Googleability matters, and it matters a lot. But when you hear the term SEO, your BS detector probably goes off, and justifiably so. It’s unfortunate that the so-called SEO experts, modern day snake oil salesmen, have ruined the field’s reputation. But for web developers, being aware and proactive about searchability is incredibly important. Building sites with searchability and accessibility in mind is just as important as practicing test-driven development and keeping code DRY.

One thing that stuck out in my mind during Vanessa’s talk was her recommendation to avoid using the text-indent:-9999px CSS trick when implementing things like logos as a page header. So the HTML is semantic:

<body>
  <h1><a href="/">My Site Title</a></h1>
  ...

While the CSS pushes the text far off the page and replaces it with a logo:

h1 a {
  background: url(logo.png) no-repeat left top;
  text-indent: -9999px;
}

This is widely considered a best practice, because users will see your site logo, while search engines see the text contained in the <h1> tags. But the reality is that this technique is also used by spammers to game search engines.

Bottom line: Don’t try to trick Google. You can’t.

This particular issue was discussed further on a podcast called Webmaster Radio Office Hours, with Google Senior Developer Programs Engineer Maile Ohye. In the discussion, which starts about 4 minutes into the podcast, Vanessa and Maile discuss Google’s view on the technique. In summary, here’s what Maile said:

When you make things accessible for screen readers and people you also do it for search engines. It’s the fundamental first step in site development.
There’s a lot you can do that don’t involve what are traditionally spammy techniques like text-indent:-9999px, which has been used to hide text and links in the past by spammers, because they thought only search engines would read it and users wouldn’t know. So they used that to boost their rankings. So it’s kind of gotten a bad rep from our standpoint, and we see it more as a spam technique than we do an accessibility technique.
Instead, use the alt attribute. Or, put text around the image and have descriptive headings and captions for it.
text-indent:-9999px is really a band-aid solution. You kind of cross into a black-hat/spam technique with it.
Don’t take the risk, you don’t want to be accidentally classified as spam.

So, this is what good markup would look like, without any need for CSS tricks:

<body>
  <h1><a href="/"><img src="logo.png" alt="My Site Title" /></a></h1>
  ...

Google understands the alt attribute, and will not punish you for using it.

If searchability is something that interests you, subscribe to the Google Webmaster Central Blog, where Maile regularly posts. Vanessa has also penned a new book, Marketing in the Age of Google, that contains a great chapter on searchability principles and practices for web developers.

ADDENDUM: Incredibly, Maile has published a blog post on this exact topic right as I’m set to publish it. So read her take, and become a true believer.

Grassroots Fundraising for Open Source Software

2010-05-21T00:00:00-04:00

This originally appeared on the Sunlight Labs blog.

By now, you’ve heard of Diaspora, the Kickstarter-funded effort to build an open, privacy-minded alternative to Facebook. In recent weeks, helped by a widely-circulated New York Times article, the project has raised $180,000 from 5,500 backers. Considering that the project, while well-thought out by four undergrads at NYU, has not produced a single line of code, these figures are surprising to say the least. But hoping for the best, assuming that this project does deliver something tangible and useful at the end of the summer, it would inform a workable funding model for open source software projects.

Righting a Wrong

Since 2003, Internet-based grassroots fundraising has had a major effect on presidential campaigns. Think of Howard Dean with his bat, Obama’s record-breaking numbers, and the moneybombs of Ron Paul. Surely, donors to those campaigns knew that their contributions would provide a direct benefit to their preferred candidate. But beyond that, donors felt their money would help the nation as a whole. They were donating for a greater purpose – something beyond their own self-interests and the interests of the candidate trying to win an election. They were convinced that the country was off course, and giving money to their favored candidate would right the ship.

In that light, maybe the wave of money donated to Diaspora isn’t so surprising. The recent negative attention Facebook has received around privacy and information sharing have left many disenchanted. Some have even gone so far as to delete their accounts. So, when Diaspora comes onto the scene as a direct antidote for those who feel icky about Facebook, people respond. In what may a bit of unintentional brilliance, Diaspora’s Kickstarter Pitch tosses around what are effectively code words in “GPG” and “AGPL”; dog-whistle politicking at its finest.

In the Ruby world, a similar situation occurred in March, after it was announced that Ruby on Rails would not be a Google Summer of Code project. In fact, no Ruby projects were chosen, in stark contrast with previous years. Many felt that Ruby was being slighted. Within a week, the community rallied around it’s own project, Ruby Summer of Code(RubySoC), and raised $100,000 to fund 20 distinct open source Ruby projects over the summer.

What’s interesting about RubySoC is how the money was raised, because it mirrors political fundraising to an extent. A dozen companies donated the “maximum” of $5,000 to fund a full project, while eleven more chipped in $2,500 to fund half a project. Together, twenty-three companies accounted for $87,500 of the $100,000 the project needed. About 150 more individual contributors (yours truly included) donated the rest, at varying amounts.

The Diaspora contribution page is a bit more opaque when it comes to exact amounts, but does give a good sense of donation levels. About 4,000 of the 5,000 donors so far have given less than $50, with four donors giving $2,000 or more. The average donation amount to Diaspora is $35, which is the same average donation amount for a presidential campaign. Stretching the campaign analogy further, the Diaspora funding was that of a grassroots model, while RubySoC employed “high-dollar” fundraising from two dozen companies to achieve its goal.

So a fundraising model can look like this: Identify a serious problem within a community, propose a solution to rectify it, and then target the right audience to fund it. Ruby open source developers, a community numbering in the thousands, could not raise $100,000 from individuals alone. Ruby-centric companies needed to chip in a large share. On the other hand, Diaspora’s community of disenchanted Facebook users is much higher, potentially in the millions. Thus, a large number of donations in relatively small amounts of $35 can be expected.

It’s All About Execution

Jaw-dropping fundraising, of course, does not guarantee ultimate success. Just ask Howard Dean and Ron Paul. For half a decade now, Google Summer of Code has employed a model that’s been effective: Provide funding for students to write code for well-established open source projects. Ruby Summer of Code follows the same path, and should be expected to yield similarly fruitful results. For example, it’s expected that by the end of the summer, Ruby on Android will be a viable development platform. All individuals and companies who funded RubySoC knew exactly what they were getting.

But Diaspora is a big fat question mark when it comes to execution. Can four college students really build a viable alternative to Facebook? Will the wishes and desires of Diaspora’s donors be fulfilled by the end-product? It seems just as easy to be overly pessimistic about the project as it is to be overly optimistic about it.

Funding Strategies

Running fundraising campaigns isn’t the only way to fund open source software. Up to this point in Sunlight’s history, fundraising has been done primarily through very large grants. Alternatively, figuring how to make the software pay for itself is another option. The Mozilla Foundation best exemplifies this, receiving over $50 million a year from its search agreement with Google.

What are other means of funding open source software? If Sunlight Labs were to seriously launch a fundraising campaign, what should that look like?

Fourteen Android Phones in the US since the iPhone 3GS

2010-05-10T00:00:00-04:00

On June 19, 2009, the iPhone 3GS went on sale. On that day, nearly 11 months ago, the Android platform was full of potential, but light on delivery. Only one Android phone was available on the US market – the T-Mobile G1. But on the one-year anniversary of the iPhone 3GS release, we’ll have seen fourteen Android phones hit the US market. The effect of this steady drumbeat of Android phone releases, averaging just over one per month, cannot be overstated. And next month’s announcement of the new iPhone 4 (or maybe “HD”), will do little to stop the Android platform’s momentum.

Here’s a detailed looked at the US phones released since the iPhone 3GS, courtesy of Wikipedia:

Released	Carrier	Manufacturer	Name	UI	Screen	OS	OS Upgrade?
July 2009	T-Mobile	HTC	myTouch 3G	Google Experience	3.2"	1.6	Rumored to 2.1
July 2009	Sprint	Samsung	Galaxy	Google Experience	3.2"	1.5	Despite a petition, no.
October 2009	Sprint	HTC	Hero	Sense	3.2"	1.5	2.1 by May 2010
October 2009	Sprint	Samsung	Moment	Google Experience	3.2"	1.5	2.1 by May 2010
November 2009	T-Mobile	Motorola	CLIQ	MotoBlur	3.2"	1.5	2.1 by June 2010
November 2009	Verizon	HTC	DROID Eris	Sense	3.2"	1.5	2.1 by May 2010
November 2009	Verizon	Motorola	DROID	Google Experience	3.7"	2.0	2.1 in April 2010
November 2009	T-Mobile	Samsung	Behold II	TouchWiz	3.2"	1.5	None planned.
January 2010	T-Mobile	Google/HTC	Nexus One	Google Experience	3.7"	2.1	--
March 2010	AT&T	Motorola	Backflip	MotoBlur	3.2"	1.5	2.1 by Q3 2010
March 2010	AT&T	Google/HTC	Nexus One	Google Experience	3.7"	2.1	--
March 2010	T-Mobile	Motorola	CLIQ XT	MotoBlur	3.2"	1.5	2.1 by June 2010
March 2010	Verizon	Motorola	Devour	MotoBlur	3.2"	1.5	Under evaluation
April 2010	Verizon	HTC	DROID Incredible	Sense	3.7"	2.1	--
June 2010	T-Mobile	HTC	myTouch 3G Slide	Sense	3.4"	2.1	--

Some observations:

Not double-counting the Nexus One on T-Mobile and AT&T, that makes for thirteen phones that have been released since the iPhone 3GS launch, and one, the T-Mobile myTouch 3G Slide, that has been given a June release date.
There’s an even mix of mid-level and high-end offerings.
AT&T has only released one Android phone, the Motorola Backflip, and it’s not very good. The AT&T Nexus One, sold exclusively by Google, isn’t even available subsidized with a contract.
The Motorola Devour is the only Verizon Android phone not using the DROID branding.
The vast majority of phones either shipped with or are scheduled to be upgraded to 2.1, but only the Verizon DROID has received the 2.1 upgrade as of today.
The 2.1 upgrade is important because a plurality of Android phones are still on 1.5. Those 1.5 phones are likely to be either a Sprint Hero or a DROID Eris, which will both see an upgrade in the coming days. When that happens, up to two-thirds of Android phones could be at 2.1.
It looks like the early adopters who got a T-Mobile G1 and those few souls who got the lesser Samsung models are SOL when it comes to Android 2.1.
The iPhone 3GS sports a 3.5” display. It might be argued that only the Android phones with larger (and higher resolution) displays truly compete with the iPhone 3GS. Most have a smaller 3.2” display.
The table above ignores all the international phones and other devices that have been released, as well as upcoming efforts like the Sprint EVO and Dell phones.
The Sprint EVO in particular has the potential to be an iPhone 4 killer, even before the iPhone 4 has been officially announced.

Mouseless Monday #2 - Keyboard-Friendly Web Apps

2010-04-26T00:00:00-04:00

Last time for Mouseless Monday, I reviewed the wonders of Google Chrome and Vimium. Vimium is great for bringing keyboard shortcuts to the Web, but it’s a general tool. What if, like on the desktop, we could have site-specific keyboard support? There’s no extension that supports this yet (it would require quite a bit of custom scripting), but the good news is that several popular web apps do natively support keyboard shortcuts.

For all of these apps, hit the question mark (? a.k.a. shift+/) to bring up an explanatory overlay, much like Vimium.

Gmail

Gmail was probably the first widely used web app to support keyboard shortcuts, and set the standard for others to follow. Like Vimium, the key bindings are Vim-esque.

For this and other Google products, keyboard shortcuts need to be enabled in the Settings.

Google Reader

Google’s feed reader closely followed suit, with many of the same bindings, but added some subtlety with the Shift key:

Google Calendar

Google Calendar has a very succinct set of shortcuts, but navigation works surprisingly well. However, I have not been able to figure out how to navigate to and edit a single event.

Pivotal Tracker

Moving away from the Google products, I use Pivotal Tracker to manage agile software projects. The shortcuts are a bit simple, like Google Calendar, so I still find myself using the mouse quite often.

GitHub Issue Tracker

GitHub’s issue tracker is also great for managing software projects. Anyone familiar with Gmail’s shortcuts will find the interface familiar.

Excluding URLs in Vimium

To take advantage of these keyboard-friendly web apps, you’ll need to exclude their URLs in Vimium Settings from chrome://extensions.

Rolling Your Own

If you’re a web developer looking to incorporate keyboard shortcuts in your own apps, consider using the jQuery Hotkeys plugin.

Mouseless Monday #1 - Vimium on Google Chrome

2010-04-12T00:00:00-04:00

Over the last few months, I’ve made a concerted effort to wean myself off of the mouse (and my MacBook’s trackpad). Why? With a keyboard, we have about a hundred buttons at our disposal, giving us incredible power and opportunity for efficiency. Those buttons are there; we should be using them as much as possible. Learning key combinations and navigating around in modes does require significant practice, but it’s no more difficult than learning to touch type. Touch screens may be the new hotness, but keyboards win when it comes to effectiveness.

I no longer work with a mouse, but I do find myself reaching down for the trackpad too often. True, the MacBook trackpad, with its multi-touch gestures, can be quite efficient, but the second or two it takes to move my hand down and back up is time wasted, and is almost always unnecessary, as a keyboard analog exists for nearly every action.

So over the next few months, on Mondays, I’m going to consciously try not to use any pointing device unless it’s absolutely necessary. If your job primarily consists of entering text – you’re a programmer, systems administrator, or writer – I think you should join me.

When you work in applications that were designed with text entry in mind, like text editors or word processors, they’re usually filled with keyboard-friendly features. But how about the web browser? Websites are filled with links, and to get to those links, they need to be clicked on. Right?

Well, no. There’s a better way…

Vimium

Vimium is an extension for Google Chrome that makes the point and click nature of web browsing nearly obsolete. As its name implies, Vimium is modeled off the text editor Vim, and the extension itself was greatly inspired by Vimperator for Firefox. The key concept behind Vimium is that it adds a keyboard navigation mode to Chrome. So when you’re using Chrome, and you’re not currently focused on a form field, you’re in navigation mode (also called normal mode in Vim-speak).

After installing Vimium, type a question mark, and a help menu pops up:

The help menu is pretty straightforward. Use “j” and “k” to scroll down and up. The most important keystroke for going mouseless is the “f”. Hit it, and all links in the browser get their own key combination:

It looks overwhelming at first, but your eye is trained on one spot, so in practice it works out. So if I were to type “f”, then “js”, I’d visit my page of Mentions. Lowercase “f” will open the link on the same page, while capital “F” will open it in a new tab. If I wanted to do a search, I’d hit “jj” and then just start typing my query, pressing Enter to submit. Hitting “jj” would put me in Insert mode and I’d type as usual. To get back to Navigation (or Normal) mode from the search box, I’d hit the Esc key.

With Vimium, I can use the Twitter website more efficiently than any of the desktop Twitter apps. Vimium will change your web browsing forever. Once you get the hang of it, you’ll only be using the mouse for interacting with Flash embeds.

Google Chrome

In addition to the goodness that Vimium gives you, I encourage you to learn the key combinations built into Chrome, many of which are shared with Firefox and Safari. Some of my favorites (on the Mac):

  Ctl-Tab         Toggle between open tabs
  Cmd-L           Jump to the Location bar
  Cmd-F           Search (nicer than Vimium's search)
  Cmd-D           Bookmark

One annoyance I have is that there’s no easy way to jump from the Location bar back to Navigation mode in Vimium. I’m finding myself hitting Cmd-F for search, typing a dummy string, and then deleting it, all to just get out of the Location bar.

Up Next

In the coming weeks, I’ll be posting more tips for going mouseless. Subscribe to my feed if you’d like to keep up.

Drafting Guidelines for Government Data Catalogs

2010-03-29T00:00:00-04:00

Originally posted on the Sunlight Labs blog. Written with David James.

A major focus of the Sunlight Labs is to push government to publish its data online. In recent months, we’ve gained in-depth familiarity with government data catalogs through our work on the National Data Catalog. The most prominent example of a data catalog is data.gov. Since its launch last year, a handful of states and cities have followed suit with their own efforts. As more data catalogs come online, we want to make sure their contents are open and exchangeable. We want to determine how to best structure the data catalog itself, and we want to ensure that the metadata it contains – the data about the data – exists in the most accessible way possible.

Last week, Clay posted three challenges for the community to tackle, and this is challenge #3. We’re looking to start this conversation now and move towards consensus within a few months. I was at Transparency Camp, digging deeper into this topic, putting us on the path to make recommendations that governments can adopt quickly.

Resource-Oriented Architecture

A data catalog lives on the Web, so it makes sense that it embrace the architecture of the Web. A simple Web site with a few pages is good for citizen use, but doesn’t lend itself to being interoperable. As such, we strongly recommend following the Resource Oriented Architecture (ROA) guidelines. When applied to the concept of a government data catalog, ROA means:

Resources that represent the data sources, agencies, and jurisdictions (if there are more than one) should exist.
Each resource should have a unique URI, so that each one can be addressed individually.
Resources should be made available in both human-friendly (HTML) and machine-readable formats (such as XML or JSON).
Content negotiation should be used to serve the resource correctly depending on the user agent.

Defining a Vocabulary

Let’s turn to the metadata that describes a particular government data source. Existing government data catalogs like data.gov have already established some best practices. But going back through the Internet’s history, it’s helpful to contemplate the work of the Dublin Core, which over a decade ago published a set of fifteen metadata elements for describing online resources. The Dublin Core spec applies to a wide range of things published online, from videos to academic papers to datasets. The fifteen elements forming the spec give us a good starting point, and remind us how similar online government data is to any other kind of resource published online. Some elements are not fully applicable (Language and Contributor, for example). Others can be broken up into several elements. Coverage can mean a physical geographic area, a political jurisdiction, or a period of time.

Going from theoretical to practical, let’s look at some sample data pages for existing data catalogs:

Looking at those four examples, we begin to see a lot of similarities among the catalogs and with Dublin Core. From these, we propose a preliminary vocabulary:

Element Name	Description
Title	Title of data source.
Description	Short description of data source.
URL	Permanent, unique URL that contains this metadata. Can be self-referential.
Type	Is the data source a dataset, API, or online database?
Downloads	File format/URL pairs that point to data files.
Created	Creation date of the data source.
Released	Release date of the data source.
Last Updated	Update date of the data source.
Update Frequency	How often the data is updated
Creator	Entity (agency, department, or organization) that created the data.
Publisher	Entity that published the data.
Maintainer	Entity that maintains the data.
Jurisdiction	Political jurisdiction of the data.
Time Period	The time period the data refers to.
Grouping	Can the data be grouped with a larger set of similar data? Recommended for data sets scoped to a time period or jurisdiction.
License	The license under which the data set is released.
Documentation	Any documentation, such as a data dictionary, or a reference (URL) to that documentation.

This vocabulary forms a base-level set of metadata. Individual data catalogs can and should publish more elements as appropriate, but serious efforts should be put into ensuring that the fifteen elements above are present.

The Formats

With a vocabulary defined, we can move to the actual data formats to represent an entry in a government data catalog. We don’t need to just choose one. Instead, we can map our general vocabulary to several existing data format standards:

XML: Maps easily to the vocabulary defined above. Loved by the enterprise.
JSON: XML’s lighter-weight alternative, loved by modern Web developers (and the Sunlight Labs).
CSV: A good compromise between machine-friendliness and human-readability.
Microformats: Can act as an easy-to-implement solution since these can be placed on plain ol’ Web sites with only a little bit of effort.

Lastly, and possibly most importantly, we need a data format to represent updates to entries in a data catalog. For that, we recommend the Atom Syndication Format. An individual entry in the Atom feed would contain the unique URL identifier and any updated elements. The entry should use any one of the four formats above, with a bias towards XML, since Atom itself is XML.

So in summary, the preliminary proposal is:

Ensure that you have enough metadata to closely follow the vocabulary.
If publishing the catalog as a Web site, use the Microformat on the data detail pages.
If publishing an API, use XML or JSON in conjunction with Resource-Oriented principles.
If publishing the data catalog as a bulk download, use XML, JSON, or CSV.
To publish updates of entries in the catalog, use Atom.

We’ll be fleshing out examples of these data formats in the Sunlight Labs Wiki over the coming weeks. I’ll be working with the good people of Socrata, who announced their SODA API at TransparencyCamp on Saturday, and any other folks interested.

Interview on the Ruby on Rails Podcast

2010-03-08T00:00:00-05:00

Geoffrey Grosenbach of Peepcode fame interviewed me for the Ruby on Rails Podcast. We talked about civic hacking opportunities for Ruby developers.

Sunlight Labs on The Changelog

2010-02-10T00:00:00-05:00

Originally posted on the Sunlight Labs blog.

Jeremy and I are on the newest episode of The Changelog, a terrific podcast about open source development. We talked about the work we do here from our offices in D.C. as well as the great work the entire Sunlight Labs community does across the country.

You may remember Wynn Netherland, one of the show’s hosts, from TweetCongress and the big splash they made last year. Along with co-host Adam Stacoviak, Wynn also interviewed Apps for America winner Jeremy Ashkenas back in December.

Enjoy the show!

Real Time Disclosure, Technically Speaking

2010-01-26T00:00:00-05:00

Originally posted on the Sunlight Labs blog.

Last week’s Citizens United v. FEC Supreme Court decision was a game-changer in terms of corporate money in politics. In short, corporations will be allowed to freely spend unlimited amounts of money to support or oppose a candidate, just as long as there is no direct coordination with that candidate’s campaign. Unprecedented amounts of corporate money will now flow into our political process. But here at Sunlight, we’re focusing on the disclosure aspects of the decision. The majority opinion stated:

With the advent of the Internet, prompt disclosure of expenditures can provide shareholders and citizens with the information needed to hold corporations and elected officials accountable for their positions and supporters.

While the decision doesn’t mandate “prompt disclosure”, it does strongly recommend it. So what does disclosure at Internet speed look like?

Examples of Real Time Disclosure

Say you contribute $100 to your congressional campaign of choice. The instant your credit card is approved, or your check is processed, the campaign will send a notification to the FEC of the contribution. The FEC’s servers will verify and record it, and instantly broadcast the details of the contribution to any third party who wants to receive it. It should all take place in a matter of seconds.

Campaign expenditures (and now political expenditures by corporations) would work similarly. For example, when a campaign or corporation decides to spend $25,000 on an ad buy, they would need to disclose that to the FEC within a 24 hour period. Unlike with contributions, disclosing expenditures needs to have a staffer manually enter data, so it would be impractical to demand instant disclosure in this case. But once the FEC receives the data about the expenditure within the 24 hour timeframe, the FEC should instantly broadcast that data to all interested third parties, just like with contributions.

Proposed Solutions

Real time web technologies have come into their own this past year. Heck, there was even a conference about it. And importantly, real time solutions can be created with open web standards and open source software, two things we strongly want to see government embrace.

HTTP

One proposed solution would be based on HTTP, the protocol of the Web: An FEC-powered RESTful API with strong querying capabilities, speaking XML and/or JSON, and made available for read and write access. The past few years have proven that well-documented RESTful APIs are the way to go. Twitter, Google, and Sunlight Labs have all embraced this convention. Contributions and expenditures would be POSTed by registered campaigns and corporations, and the public could read all the data via GET.

One newer real time technology, based on HTTP and RESTful principles, is PubSubHubbub, or PuSH for short. PuSH excels in real-time, event-driven notifications. Usually, when you want to know if a given API has been updated, you need to ask it repeatedly if there’s any new data. This technique, called polling, inefficiently uses up bandwidth because there are a lot of requests made where nothing new is to be found. Instead, PuSH mandates that interested parties subscribe to receive notifications, giving the publishing server a URL to send a notification to, and it’s up to the interested party to take action when they receive that notification at the URL they control. Registering your URL endpoint to receive a notification is known as a Webhook.

XMPP

Unlike HTTP, XMPP (eXtensible Messaging and Presence Protocol) was designed for real time applications and asynchronous communication. You’ve used XMPP if you’ve used Google Talk for chatting, or other Jabber IM services. At the simplest level, XMPP is a way to send instant messages around the Internet, with support for extending the definition of those messages to suit your needs. XMPP also defines a publish-subscribe extension, which would allow third parties to subscribe to a particular set of disclosure data and receive messages from the server instantly.

Note: For simplicity’s sake, the diagram above ignores XMPP’s decentralization features.

Either solution is well-supported by solid open source software. HTTP servers like Apache and nginx are ubiquitous, and frameworks like Django and Rails make creating RESTful APIs a snap. For PuSH, Google has released a reference implementation running on App Engine. The server of choice for XMPP is ejabberd, an Erlang-based project.

Real time disclosure is possible today. The technology is here, and it’s open. Now, we need to muster the political will to see it through.

Your Input Wanted on Recovery.gov Data

2009-09-24T00:00:00-04:00

Originally posted on the Sunlight Labs blog.

Here at the Sunlight Labs, we’ve focused a lot on the recent bid on version 2.0 of Recovery.gov. This morning on the Labs mailing list, Rusty Talbot of Synteractive, one of the winning contractors, asked for input on the best way for Recovery.gov to publish its data.

Rusty wrote:

“The Recovery, Accountability, & Transparency Board wishes to have an open discussion with all interested developers about how data should be made available via Recovery.gov.

As you are all aware, a new version of Recovery.gov will be released soon. From a data standpoint, the initial release of the new site will replicate existing functionality. However, the Board aims to set a new standard of transparency with this site and would therefore like to make the data available in the most convenient and straightforward way (or ways) possible so you can use and analyze official, up-to-date Recovery Act data. We need your input to achieve this goal.

Please let us know how the site could best meet your needs in terms of machine-readable data format(s) and standards, APIs, guidance, training, etc.”

This is a great opportunity for all of us who work hard to make government data more open and accessible. Back in April, we at the Sunlight Foundation, as a member of the Coalition for an Accountable Recovery, published a report to the OMB giving recommendations on data feed publishing (starts on page 8 of the document).

As one of the project leads for the upcoming National Data Catalog, I’ve been thinking a lot about what makes government data truly useful for software developers, researchers, and journalists. Here’s my more technical take:

First, publish as much machine-readable raw data as possible. The data should be in a widely-used format, XML or JSON, and should be accessible from predictable, permanent URLs. Documentation clearly describing the schema and data types is essential.
As the Recovery Act programs are ongoing, document when, where, and how new data will be published. Set up a regular schedule for updates. Communicating updates could be as simple as setting up a mailing list or feed.
After the raw data is made accessible and proves to be useful, consider building an API that complements, rather than duplicates, the raw data sets. APIs are useful for developers who wish to get smaller, discrete sets of data without having to crunch through the entire raw data set. An emphasis should be placed on querying functionality that can be integrated into other apps.
The API should also speak XML and/or JSON, while following REST principles. Libraries to consume such APIs exist in all modern programming languages. Again, clear documentation is vitally important.
When it comes to data, the RAT Board and Recovery.gov should eat its own dog food. That is, the data used internally by the RAT Board for analysis, tracking, and powering the website should be exactly the same data published to the public. This may not be technically possible right now, but it should be a goal because doing so would be the single best way to ensure that data being published is serving real-world needs.

So those are my thoughts. What are yours?

Recapping the Ruby Hoedown

2009-09-03T00:00:00-04:00

Originally posted on the Sunlight Labs blog.

I was in Nashville this past weekend giving a talk at the Ruby Hoedown, a completely free conference held for the South’s Ruby developers. My talk was titled “Civic Hacking”, and the slides are on Slideshare:

The conference was one of the best regional Ruby conferences I’ve been to, and a lot of that had to do with the venue, the Gaylord Opryland Hotel. Essentially a self-contained small town, reminiscent of the Biodome, the hotel provided ample opportunities for running into folks and picking each others brains over a meal or drinks. Visit it if you want to experience how we’ll live on Mars.

Photo credit to Stephen Yeargin on Flickr.

Some solid talks were given as well. Corey Donohue of Engine Yard described the importance of breaking up large apps into smaller, RESTful services. Leon Gersing demoed the features of Appcelerator Titanium. While showing off Titanium’s incredible new iPhone and Android support, he was asked “Are you a witch?”, easily the best question of the conference. RSpec lead David Chelimsky spoke on the art of mocking, and a kazoo-enhanced lightning talk by Clinton R. Nixon on MongoDB was the most entertaining. Lyle Johnson has posted a thorough wrap-up of the weekend as well.

One day in the Labs, a question was raised as to why one of our developers, David, always seems to be wearing Ruby-related shirts. The answer is because we Rubyists put on a lot (and I mean a lot) of regional Ruby conferences. The Python crowd, it seems, only have PyCon and DjangoCon, the latter of which Labs member James is headed to next week.

So we Rubyists love to organize ourselves and get together, that’s certain. It’s high time that we organize more around making our government more open and more transparent. We’re already off to a good start. Apps for America 1 winner Filibusted is a Rails app, as is Know Thy Congressman , which placed third. This We Know and GovPulse, two of the three finalists in Apps for America 2 are also Rails apps.

But I think we Rubyists can do a lot better. Using the new tools we have here at SunlightLabs.com, let’s go over some Ruby-based projects and ideas:

Congrelate - A project at the Labs headed by Eric, powered by Sinatra and jQuery, which focuses on bringing together disparate data sources in powerful ways. We already pull in data from Congress, the Census, OpenSecrets, and GovTrack, but we’d like to bring in more data sources. Check out the code base for instructions on how to build in support for new data sources.
Transparency Corps - Much like Amazon’s Mechanical Turk, TCorps allows anyone anywhere to have a positive impact on making our government more transparent by aggregating small actions that require human intelligence but not specialized political knowledge. If you have an idea that can benefit from this tool, it’s easy to write your own Task that we can plug into the system.
National Data Catalog - Led by David and I, we’re aiming to build a comprehensive catalog of all data sets and APIs published by all levels of government. We’ll soon need help developing parsers that scrape existing data catalogs out there and load that information into our catalog.
API wrappers for open government web services - While we have a Ruby gem for the Sunlight Labs API, the other APIs out there have been neglected. Let’s get cracking on Ruby wrappers for OpenSecrets.org, GovTrack, MAPLight, and LittleSis.

So let’s get going, Rubyists! Feel free to contact me with any questions. And speaking of regional Ruby conferences, I’ll be at Ruby DCamp in two weeks.

Google Wave: First Thoughts

2009-07-19T00:00:00-04:00

I received a developer preview invitation for Google Wave this weekend. I’m luigi@wavesandbox.com if you want to add me to your contacts list.

My initial thoughts can be summed up thusly:

How bad is it? After about an incredibly frustrating hour or two of trying to wrap my head around things, here are some thoughts…

The Wave Discuss Group

Apparently in my excitement of signing up, I joined the wave-discuss group, which displays a collective intelligence that can only rival the comment threads to Avril Lavigne videos on YouTube. Click through to see the gory details:

I have no idea how to opt myself out of this group. I’m subject to a mountain of crap in my inbox that I simply don’t want. I tried the Settings option, which appears to be a wave, albeit a broken one speaking Japanese:

The Waves Themselves

The Archive and Mute buttons don’t seem to work. Spammers have already struck:

The WYSIWYG interface promotes some truly heinous typography decisions:

Threading seems to be broken. These answers don’t make a lick of sense:

The Good

The live type feature is indeed awesome. But watching the slowness of typing reminds you how much more efficient talking can be. Also, there appear to be live Googlers trying to answer questions as fast as they can, so that’s nice.

Hopefully, when some people I know get accounts, we can participate is civilized, well-structured waves. But for now, it’s all just too much for me. Using Google Wave results in a sensory overload of low-quality content. Maybe when I figure out how to get off the wave-discuss group, things will be more clear and I can focus on the technology instead of all the chatter.

UPDATE

I figured out how to opt-out of the annoying wave-discuss group. It involved adding my name to a list (in a wave, naturally) of people who want to be opted-out. Apparently a Googler goes through the list every few days and manually processes the names. There’s no automated system yet. A lose-lose for everyone involved…

On Being a Civic Coder

2009-06-12T00:00:00-04:00

I started working at the Sunlight Labs this week, and I’m giving a talk on Being a Civic Coder at RubyNation today:

Be A Civic Coder

View more PDF documents from luigimontanez.

Some resource links:

Three Videos from RailsConf

2009-05-29T00:00:00-04:00

I attended RailsConf 2009 earlier this month. Overall, it was a good conference, and any commentary on the city to be made has already been made by others.

I brought my Flip Mino HD with me. While great for shooting outdoor (or very well-lit) scenes, I find the Flip doesn’t perform as admirably in lower light situations, which is unfortunately the environment of a conference hall. Also, the Flip is optimized to record voice at close distance, so picking up mic audio from a speaker system wasn’t ideal either.

That being said, here are three videos I recorded at RailsConf. The TweetCongress video has the best audio quality, while the Gov 2.0 Panel, I’m sad to say, has the worst.

Build an App, Start a Movement from Luigi Montanez on Vimeo.

The Russian Doll Pattern from Luigi Montanez on Vimeo.

Government 2.0 Panel at RailsConf from Luigi Montanez on Vimeo.

What's with the Wolfram|Alpha Control Room?

2009-05-18T00:00:00-04:00

There’s no doubt about it, the launch of Wolfram|Alpha has been truly strange.

There’s the technology itself. All answers get rendered as images, even when they don’t need to be. It doesn’t handle obvious errors very well.

They oddly chose to launch on Friday night, a time when even the majority of Battlestar Galactica fans opted to use their DVRs and go out. And they’ve fed a media hype machine to levels that they cannot possibly live up to.

But the weirdest thing about Wolfram|Alpha is that they have a control room, like NASA, or like the villains from 24. Nerds hunch over their laptops in stadium style seating, with huge monitors and projected images on a big wall in front of them. Apparently, launching a web app is serious business. They even have a live web feed, which primarily broadcasts video of some guy standing at the back of the room scratching himself.

So, if you think this is all incredibly surreal like I do, I invite you to take any of the above images, download Comic Life, and have some fun. Here’s my stab at it:

Ruby on Rails != Its Core Team

2009-04-30T00:00:00-04:00

While observing the hullabaloo surrounding the Rails community in recent days, one meme kept popping up that struck me as simply not true: The notion that the Rails community is centered around a group of self-aggrandizing rock star programmers, namely DHH and his core team. This cannot be further from reality. The strength of Rails lies in the vast, diverse community of developers who have sought time and time again to help each other out, to teach others, and to share code, primarily because it gave them joy to do so.

As a complete nobody in the Rails community, I’ve done much more observation and learning than I have done sharing these past few years. As such, I’d like to share with you a list of people in the Rails world who have established themselves not through their status or through the force of their personality, but through their output.

Dave Thomas – The author of the Programming Ruby/PickAxe book, and the main author of the first few editions of Agile Web Development with Rails, Dave was probably the guy who taught you Ruby and Rails.
David Black – The author of the now-classic Ruby for Rails (refreshed edition coming out soon), David provided some essential reading for the Rails developer looking to harness the power of Ruby. Also director of Ruby Central, the organization behind RubyConf and RailsConf.
Jamis Buck – A guy who whittles wood in his spare time while living in Idaho, Jamis created the invaluable Capistrano deployment tool to help us automate away tedium. He used to be on the Rails Core team, so putting him here may be fudging the list criteria, but his contributions have been so huge (both his projects and his blog posts), that he deserves a mention.
Ezra Zygmuntowicz – Literally writing the book on Rails deployment, Ezra did the hard work of figuring out what exactly constituted a solid Rails stack when the existing options left plenty to be desired. He has since worked on a myriad of projects, including Merb.
Yehuda Katz – @wycats may be the man who saves Rails from itself. As the lead driver for Rails 3.0, he’s already begun sharing the gory details of the Merb/Rails merge to the eagerly waiting masses. He’s also contributed greatly to jQuery, Merb, and Datamapper.
Leah Silber – Whenever you go to a regional Ruby conference, you’ll always see Engine Yard as the top sponsor. While EY may send Ezra, Yehuda, and others to present at these conferences, it’s the behind-the-scenes help provided by Leah and company that really make a conference successful.
David Chelimsky – Behavior Driven Development is a complete win that we in the Rails community get to enjoy, and as lead developer of RSpec, David has shown a solid and steadfast commitment to continuously improving the tool so that it in turn improves our software.
The Thoughtbot team – This company’s work speaks for itself: Shoulda, Factory Girl, Paperclip, and Hoptoad.
Ryan Bates – Putting out one or two RailsCasts a week for over two years, Ryan has been amazingly consistent both in the quality of his work and in his dedication to helping us become better Rails programmers.
John Nunemaker – RailsTips may be the most consistently helpful of all the Rails blogs.
The GitHub team – Even without considering the contribution the site itself has made, the developers behind GitHub have contributed some fantastic projects to the community, namely Cheat, cache_fu, Jekyll, and god, and they’ve done plenty of helpful blogging along the way.

The people above have written essential libraries, authored books, produced other educational material, and/or helped organize conferences. In other words, they’ve taken the great technical work of the Rails Core Team, and created a community around it.

The above list is by no means comprehensive. So if you believe I’m missing someone, please leave a comment below. Also, in addition to Jamis and Yehuda, there are many of members of the Rails Core team who do it for the music, so to speak, and nothing else. But hopefully the list above conveys the point that Ruby on Rails is so much more than DHH and its small group of core committers.

Respect the Browser - Lessons from JSConf 2009

2009-04-29T00:00:00-04:00

“This is the real Future of Web Apps conference,” I thought to myself during the first day of JSConf 2009, the first Javascript-focused conference. Upon further reflection of the weekend, I realize that my idea of what a web application can be, and more importantly, how it should be created, has been fundamentally altered. The web browser, usually thought of as a dumb, thin client, is actually full of unrealized potential. It’s time that we, as web developers, tap that well and create a tidal wave of innovation. (This idea was championed by conference organizer Chris Williams in an impromptu lighting talk during the conference.)

In short, we should treat the web browser as a first-class development platform, and Javascript as a Serious Language™. No longer just for presentation, a browser can process business logic, and it can talk directly to our data stores. In offline mode, it can talk to its own data store and sync up later. A well-written application for the browser can give our users a markedly better, smoother experience. For developers, browser-centric development can help us deliver better applications faster, continuing on the promise of agile web frameworks like Django and Rails.

JSConf opened my eyes to the work already under way to accomplishing those goals.

The Frameworks

The two “killer” frameworks out there today are SproutCore and Cappuccino. Both aim to give users a Mac OS X-like experience on the Web. SproutCore is what powers Apple’s MobileMe and OtherInbox, while Cappuccino powers the Apply Keynote-like 280 Slides.

Francisco Tolmasky of 280 North kicked off the conference with an overview of Cappuccino and Objective-J (an Objective-C-like superset of Javascript) and a drawdropping demo of the nib2cib utility, which allows a Mac developer to use XCode’s Interface Builder to construct Cappuccino interfaces. Overall, it appears that Cappuccino is targeted at Cocoa developers.

SproutCore, on the other hand, is a Ruby gem, uses ERB for templating, and supports Rubyish concepts like Mixins. Mike Subelsky, co-founder of OtherInbox and upcoming speaker at RailsConf, presented on SproutCore on day 2. He stressed the importance of Key-Value Coding and Key-Value Observing in his talk, concepts which I admit I don’t yet fully grasp. Interestingly, it looks like SproutCore is in the early stages of iPhone support.

Speaking of the iPhone, a mobile framework called PhoneGap was presented as well. Currently targeting iPhone, Android, and Blackberry (with Palm Pre, Windows Mobile, and Nokia on the way), PhoneGap lets you write your web app (in HTML+Javascript) once, and have it available for all supported platforms (along with proper skinning for the target platform). PhoneGap also comes built in with offline storage support, and does other nice things for you like geolocation and harnessing the accelerometer.

One final framework worth mentioning is Titanium. This is the easiest to explain: it’s Adobe Air or Microsoft Silverlight, but using HTML/CSS and Javascript instead of proprietary languages. A Tweetie-like app was demoed by Appcelerator CEO Jeff Haynie, and it was certainly impressive.

Something that wasn’t presented, but I was given a demo of by a Mozilla employee, is Bespin, which is a web-based code editor harnessing the CANVAS tag in HTML5.

Dealing with Data and Processes

One of the negatives associated with Javascript has been a lack of options when it comes to data persistence. In recent years, we’ve seen support for Gears and HTML5’s WebStorage implemented in the browser, and these suffice for most use cases. There’s also been talk of CouchDB in the browser.

Gears also supports the notion of a worker pool, and Firefox will soon have Web WWorkers, both of which allow arbitrary Javascript to be executed in the background. For real-time messaging, Jack Moffitt presented on Strophe.js, a library for writing XMPP clients, which improves on the Comet concept that popped up a few years ago.

Browsers Almost There

With the pending releases of Safari 4, Firefox 3.1, and Google Chrome 2, there’s an arms race among browser camps to see who can come up with the best and fastest Javascript engine while building in the most comprehensive support for HTML5. That’s definitely a Good Thing. Who wouldn’t want to write code for a development platform simultaneously supported by Apple, Mozilla, and Google?

So, what about Internet Explorer? The latest version (IE8) does support some, but not all, of HTML5, and Gears is available for storage and threading. But I think this is an issue where we developers need to draw a line in the sand and send a clear message: Either Microsoft catches up with the others, or it gets left behind. If we treat Internet Explorer as a dying platform, it will become one. I’m talking about all versions of IE, not just the oft-maligned IE6.

There’s More

For a more comprehensive blow-by-blow, see Michael Harrison’s recap. Conference organizer Chris Williams penned a recap as well.

Plucking Out Article Content from an Arbitrary Web Page

2009-03-22T00:00:00-04:00

For Defogger, the user should be able to paste in a URL (or click on a bookmarklet when on another web page) and get back a defogged, enhanced view of the content. Using OpenCalais, Defogger can figure out the important people, organizations, events, and relationships contained in a news article. While OpenCalais does accept HTML as input, it assumes that everything in the HTML is fair game. For example, putting this AP article hosted on Google through OpenCalais will return “Google” as a company mentioned in the article, when in fact Google is only detected because its name is contained in all the extra clutter around the content of the article.

So I needed a way to remove that extra clutter and pass on exactly what mattered to OpenCalais. Interestingly, I had seen this problem already solved with a bookmarklet called Readability, “a simple tool that makes reading on the Web more enjoyable by removing the clutter around what you’re reading”. Try out the bookmarklet on some websites to see for yourself. It seems almost magical, and comes in handy all the time, especially for long articles, or for reading on the iPhone.

Luckily for me, Readability is open source, with the code hosted on Google Code. It was just up to me to translate the Javascript algorithm into Ruby. So here’s my version of the code:

require 'open-uri'
require 'nokogiri'

def pluck_article(url)
    
  # get the raw HTML
  doc = Nokogiri::HTML(open(url))  
  
  # get the paragraphs
  paragraphs = doc.search('p')
  
  # assign points to the parent nodes for each paragraph
  parents = {}
  paragraphs.each do |paragraph|
    points = calculate_points(paragraph)
    if parents.has_key?(paragraph.parent)
      parents[paragraph.parent] += points
    else
      parents[paragraph.parent] = points
    end
  end
  
  # get the parent node with the highest point total
  winner = parents.sort{ |a,b| a[1] <=> b[1] }.last[0]
  
  # return the plucked HTML content
  "<h1>" + doc.search('title').inner_html + "</h1>" + winner.inner_html
end


def calculate_points(paragraph, starting_points = 0)

  # reward for being a new paragraph
  points = starting_points + 20

  # look at the id and class of paragraph and parent
  classes_and_ids = (paragraph.get_attribute('class') || '') + ' ' + 
                    (paragraph.get_attribute('id') || '')  + ' ' + 
                    (paragraph.parent.get_attribute('class') || '')  + ' ' +
                    (paragraph.parent.get_attribute('id') || '') 

  # deduct severely and return if clearly not content
  if classes_and_ids =~ /comment|meta|footer|footnote/
    points -= 3000
    return points
  end
  
  # reward if probably content
  if classes_and_ids =~ /post|hentry|entry|article/
    points += 50
  end  
  
  # look at the actual text of the paragraph
  content = paragraph.content
  
  # deduct if very short
  if content.length < 20
    points -= 50
  end
  
  # reward if long
  if content.length > 100
    points += 50
  end
    
  # deduct if no periods, question marks, or exclamation points
  unless (content.include?('.') or content.include?('?') or content.include?('!'))
    points -= 100
  end
  
  # reward for periods and commas
  points += content.count('.') * 10
  points += content.count(',') * 20  
  
  points

end

I’ve commented the algorithms thoroughly, and Ruby is quite readable, so it should be easy to follow. Essentially, the code takes advantage of several common conventions used on modern websites:

An article (or blog post or press release) is coded in HTML as a lot of text enclosed in <p> tags.
Those <p> tags are grouped together, usually under a containing <div> element.
Written English contains a lot of commas.

Using those observations above, along with a few other tricks, the code assigns points to all elements that contain paragraphs on the page, and the containing element with the highest point value “wins”. To use this, it’s a simple pluck_article('http://www.path.to/some/article').

I’ve tried the method above out on the sites of the New York Times, Google AP News, Wall Street Journal, Huffington Post, Athletics Nation, and even a press release on Nancy Pelosi’s site. All worked pretty well. The one platform where the algorithm breaks (both for my implementation above and for Readability) is on Scoop, the blogging platform behind Daily Kos and MyDD. Specifically, the parsing doesn’t work on blog posts that have a long extended entry (usually diaries). This is because Scoop generates a blog post’s main entry and extended entry as two separate <div> elements, and the extended entry <div> ends up “winning”. That’s not at all semantically correct, but I don’t expect it to ever be fixed. Subsequently, SoapBlox sites (like OpenLeft) also have the main entry problem, as SoapBlox is a direct port of Scoop.

I’ll sleep on the problem with Scoop/SoapBlox, but I’m pretty happy with the solution for now. Next up for Defogger: building the page templates using Haml, Sass, and Compass.

Liveblogging the Development of Defogger

2009-03-18T00:00:00-04:00

The Apps for America contest put on by the Sunlight Foundation is happening right now, and submissions are due on March 31st. Like any respectable software developer, I’ve put off working on my idea for far too long. With two weeks left to go, I’m finally getting around to working on it, most recently coming up with some mocks of the home page:

I’ll likely go with the second mock, as the Capitol background is a bit too busy and won’t work well for the interior, information-heavy pages. In the spirit of transparency, I’ve decided to “liveblog” the entire process of creating the app during these next two weeks. I hope to share design decisions I make, problems I run across, and things I learn. Knowing that I have an audience will hopefully spur me on to create something really good, and maybe will encourage others to attempt an app of their own before the contest is over.

So what’s the idea behind Defogger? It’s a simple app that uses OpenCalais to semantically mark up a news article (or a blog post or a press release), and enhances the marked up document with transparency-related APIs. So imagine giving Defogger an article about some member of Congress presiding over the groundbreaking of some multi-million dollar project funded with stimulus money. Defogger would answer some of these questions:

The Politician: Which district does he represent, and for how long? Who are his biggest contributors? Are any of them mentioned in the article? Did this member of Congress put in an earmark or otherwise grease the legislative wheels for this project?
The Construction Company: Does this company employ lobbyists? How much money have they contributed to members of Congress, and are any of those mentioned in this article?

Like any good side project, I’m using Defogger as a means of learning some new tools and techniques:

jQuery: I don’t know this Javascript framework nearly well enough, and usually need to defer to other teammates for their expertise. I want to become a jQuery ninja when this is all said and done.
Shoulda, Factory Girl, and Mocha for Behavior-Driven Development: I’m comfortable with RSpec, but I know that Shoulda is what the “cool kids” use in the Rails world, and I definitely could do BDD and TDD much better. I hope to build the entire app this way. I employed full BDD with the Sunlight API Gem to great success.
New Rails Features With Rails 2.3 just released, it’s been far too long (since maybe Rails 2.0) since I did a $ rails my_new_app on the command line. While Rails has been pretty stable these past two years, there are a lot of goodies that I’ve never learned and tidbits I have yet to pick up. I hope to fill those gaps with this project.
Haml + Sass, and Compass: While I don’t have the best web design chops, I like to think I can hold my own. However, the process of skinning a site is always painful, and I’m constantly looking for ways to produce better, more maintainable, front-end code.

37signals likes to tell us to embrace constraints. Hopefully, I’ll learn that lesson and more throughout these next two weeks. Please consider subscribing to my feed if you’d like to come along for the ride.

Ruby Sunlight API Gem 0.9 Released

2009-03-15T00:00:00-04:00

This release includes support for two new fields on Legislator, a refactoring of the base functionality so that the gem is more compatible within Rails and Merb apps, and support for Lobbyists, Filings, and Issues. There is a new way to set up your Sunlight API Key, so this release is not backwards-compatible with releases 0.1. and 0.2.

This gem is now considered a feature-complete wrapper around the Sunlight API, and will be bumped up to 1.0 after being used for a bit in the wild.

GitHub | RubyForge

Changelog:

Warning: This release is not backwards-compatible!
Change loading behavior of base functionality, works better with Rails and Merb
Sunlight::SunlightObject is now Sunlight::Base
For set up, Sunlight.api_key= is now Sunlight::Base.api_key=
For set up, using “include ‘Sunlight’” is no longer recommended
Correct usage is Sunlight::Legislator.all_for(…) instead of just Legislator.all_for(…)
Credit to Rue the Ghetto (rughetto on GitHub) and Eric Mill for inspiring the improvements above
Add support for senate_class (“I”, “II”, or “III”) and in_office (0 or 1) on Legislator
Add support for Lobbyists, Filings, and Issues
Huge credit to mindleak on GitHub for Lobbyist-related functionality

Ruby Gem for the Sunlight Labs API Updated to 0.2

2009-03-02T00:00:00-05:00

Back in August, I released an initial version of a Ruby gem for the Sunlight Labs API. The gem has since been used in many projects, most famously on Tweet Congress.

This weekend at Transparency Camp, I set out to improve several aspects of the gem, including adding a twitter_id field, which wasn’t there back in August when I made the initial release. Changes include:

Add support for twitter_id and youtube_url on Legislator
Add Legislator#search_by_name for fuzzy name searching
Add Legislator#all_in_zipcode for better lookups on a five-digit zip code
Raise exception when API Key isn’t set
Credit for various fixes goes to GitHub users pengwynn, hoverbird, and wilson

Check it out on GitHub. Ruby developers participating in Apps for America will hopefully find it indispensable.

Starting Back Up

2009-02-28T00:00:00-05:00

Over the last eight months or so, my blog about software development at Salesforce on Rails, has been criminally neglected. The cause has been twofold. First, I’ve just been incredibly busy, but that’s a tired and lame excuse. More significantly, I’ve largely stopped working with Salesforce.com and the Force.com development framework, for reasons I’ll get into later. So Salesforce on Rails is now retired, and I’ll instead be blogging about software development here at LuigiMontanez.com. Old entries from Salesforce on Rails will be migrated below this one.

So LuigiMontanez.com is now my place to blog about software development technologies, and also about becoming a better programmer. I’ve been heavily into the idea of software craftsmanship lately, facing the harsh reality that I have enormous room for improvement as a software developer. I’ll still blog about the intersection of potitics and technology at Leftmost Bit, so please do grab that feed if you’re interested.

What’s next for LuigiMontanez.com? Well, in addition to sharing my efforts on becoming a better software developer and project lead, I’ll be blogging about these technologies that are currently piquing my interest:

Ruby and Rails
jQuery
CouchDB

I’m also planning to participate in the Apps for America contest put on by Sunlight Labs. My submission will harness Open Calais and the various government transparency APIs, creating a tool to allow users to clear the fog about some of the non-obvious connections between our elected officials, special interests, and the laws and policies coming out of government. It’ll be called Defogger.org.

Compelled by the contest’s spirit of transparency, I’ll be blogging the entire app development process right here, from start to finish. With the contest deadline set for March 31, I certainly need to get started on the project sooner than later, and what better way than to force myself to blog about it on a regular basis.

Stay tuned!

Accessing Passenger sites from a Vista instance in VMware Fusion

2009-02-12T00:00:00-05:00

This article by Robby Russell reminded me of a hopefully helpful post I’ve been meaning to write for a while. The problem is straightforward: You’ve got Phusion Passenger set up on your Mac OS X Leopard (see Robby’s post for details). Your client called, and they say the site you just launched for them looks funky in Internet Explorer. Fun. So now, you want to access your local Passenger site living at http://myapp.local from the copy of Windows Vista living in your VMware Fusion.

There are two things you need to do in order to access your Passenger sites from Windows Vista via VMware Fusion:

Figure out what Vista thinks is your Mac’s IP address
Create entries in Vista’s /etc/hosts file (yes, Vista actually has one)

So fire up VMware and Vista, and let’s begin:

Determine Vista’s Gateway IP

When running VMware Fusion, the virtualized OS (in our case Vista) refers to the host system (Mac OS X Leopard) as its IP Gateway and its DNS. So you want to figure out what that IP address is, and you’ll be able to access the Apache server running in Leopard. Determine the IP with the following steps:

Open the Control Panel, switch to Classic View, open Network and Sharing Center.
Click on Manage Network Connections in the sidebar.
Double click on the Local Area Connection, and click on the Details button.
Make note of the IPv4 Default Gateway – that’s your ticket to accessing your Passenger sites.

Edit Vista’s /etc/hosts file

I discovered this handy link, which tells you all you want to know about Vista’s hosts file. In short, it works pretty much the same as any Unix /etc/hosts file. I followed the instructions to set up a shortcut on my Vista Desktop for quick access, and the screenshot of the Properties dialog for that shortcut is below:

Now, just edit the file, creating an entry for each Passenger site:

172.16.160.2 site1.local
172.16.160.2 site2.local
172.16.160.2 site3.local

With the file saved, you should be able to access your Passenger sites from a browser in Vista (no restart of anything needed).

Multiple IEs

Now for a bit of a bonus, I highly recommend installing IETester in Vista. IE8 beta 2, IE7, IE6 and IE5.5 will be at your fingertips.

Find this helpful? Please consider recommending me on Working with Rails.

Custom Fields on Standard Objects in the Force.com IDE

2008-06-28T00:00:00-04:00

Browsing through the Force.com Discussion Boards while playing with the new version of the Force.com IDE (released June 19), the ever helpful JonP explained exactly how to generate a .object file on standard objects.

Fire up Eclipse, and in your Force.com project open up src/unpackaged/package.xml. Find the XML element with the name of “CustomObject” and add in some new subelements with the value of the standard object. It should look like this:

<types>
    <members>*</members>
    <name>CustomObject</name>
    <members>Contact</members>
    <members>Account</members>
</types>

Then, run the “Refresh from Server” functionality by right-clicking on src/objects and accessing the Force.com menu. Your standard objects should then appear.

Alternatively, if you’re paranoid about too many custom fields being editable (and quickly erasable) in the IDE, you can stick in an entirely new types element named CustomField that looks like this:

<types>
    <members>Opportunity.SomeCustomField__c</members>
    <name>CustomField</name>
</types>

Now, you’re able to use the power and speed of the IDE to create, edit, and deploy custom fields on your standard objects. It’s a great productivity win. A big thanks to JonP for explaining all this.

Engine Yard Express: A Production Slice on Your MacBook Pro

2008-06-20T00:00:00-04:00

While computing moves more and more to the cloud, it’s interesting that Ruby web development best practices tend to go in the opposite direction, toward the workstation right in front of you. Running a local web server via ‘script/server’ (or my preferred method of ‘thin start’), hosting gem documentation locally, and using local git branches are all win/win practices because they’re far more efficient, and frankly, let you do some bonehead moves without your co-workers ever having to know.

So maybe deployments should be practiced locally as well. Ezra Z. has announced Engine Yard Express, a VMware image that’s essentially an Engine Yard production slice. The idea is that you run the image via virtualization software on your workstation, so that you can get all the benefits of a virtual server without having to actually connect to one through the Tubes.

Yes, this really rocks. You’ll be able practice deployments on the plane!

So first, you’ll need to buy VMware Fusion for your Mac. I first purchased Parallels over a year ago, but VMware clearly kicks its ass. Clearly. It actually runs Windows without making the rest of your system unusable.

Next, download the image and double-click on it. Before you even realize what happened, you’ll be presented with a ready, fully-functional slice just waiting for you like an obedient dog:

You’ve been pleasantly provided with randomly generated passwords for the root and express users, and you’re even given the IP address it’s chosen.

Let’s create an alias for that address so we don’t have to memorize it. Open up /etc/hosts in your favorite text editor (requires sudo permission) and stick something like this in there:

192.168.233.128 express.local

Now, you should probably just leave your VMware window alone. Instead you’ll want to SSH into the slice as if you were logging into a remote server:

$ ssh express@express.local

Nice! A quick visit to http://express.local:81 gives you your Merb start page.

Now, in your Capistrano scripts, you can use ‘express.local’ as the domain of your servers, and you’ll be able to practice deployments without ever connecting to a remote server. Bom chicka wa waa!

Running a Local Gem Documentation Server on Mac OS X

2008-06-10T00:00:00-04:00

When I first started playing around with Ruby on Rails, one of the things that struck me was how development was done completely locally. Instead of uploading or synchronizing your source code to a remote web server, you simply fired up a Ruby web server process on your local machine, and the development process seemed to flow more smoothly.

In that same vein, documentation for Ruby gems can be annoying to hunt down. You have to go to the gem’s web site, and hope that RDoc exists somewhere. Or, you can navigate to your gem’s local install directory and poke around in there. Too much work, and it could break your flow. Not anymore:

$ gem server

Now, just visit http://localhost:8808, and you’re good to go. That easy!

Also, you may find yourself missing RDoc on updated gems, so make sure to do this when it bothers you:

$ gem rdoc [--all|gem_name]

Now, it would be great if the server just started automatically on boot up, with no need for a command line directive. In Mac OS X Leopard, the preferred way is to use launchd/launchctl, which is about as obvious to the uninitiated as the rules of Cricket. Luckily, there’s Lingon, which wraps all that ugly XML in a nice GUI. Make a new entry that looks like this:

Note the full path to the gem command, which I needed for the launch command to work. And that’s it. Full documentation to all your locally installed gems, available whenever you need them.

Special thanks to Daniel Fischer for the ‘gem server’ tip, and Brandon Beacher for the tip on Lingon.

Essential Force.com Resources

2008-04-26T00:00:00-04:00

Because the Force.com developer community is still very small, it’s important to make the most out of what Salesforce.com (the company) provides for us on the Web. The best site for all things Force.com is of course developer.force.com, with its invaluable Wiki and Discussion Boards. But sometimes dead-tree versions of knowledge can be helpful too.

Ebooks

SFDC has made available two free books, one is an introduction to the platfom and the other is a recipe-style book. If you’ve been to a Force.com event recently, chances are you received these as actual, bound books. Here are the PDF ebooks:

Introduction to the Force.com Platform – Really a beginner’s guide, this book takes you on a feature tour of the Force.com platform, and goes over both administrative/declarative features as well as the more programmatic Apex and Visualforce capabilities.
Force.com Cookbook – I find myself flipping through this problem/solution recipe book often. It’s very helpful at producing those “Ah! So that’s how I’m supposed to do it!” moments.

The APIs

The Force.com Application Programmer Interfaces (APIs) allow us developers to integrate our Salesforce instances with external web services, as well build in some powerful customization into those instances. I don’t print these out, but rather just keep the PDFs quickly accessible for fast keyword searching:

Apex Web Services API – Covers the SOAP API in all its glory. I personally wish this was REST, but SOAP is better than everything but REST.
Apex Metadata API – A newer API, the Metadata API allows us to define the structure (fields and relationships) of our custom objects via XML rather than via the declarative point-and-click interface. Unfortunately there’s no Metadata API available for standard objects (Contacts, Accounts, Opportunities) yet, but I expect that to change this year.
Apex AJAX Toolkit API – The AJAX Toolkit is primarily used with S-Controls, which are being phased out in favor of Visualforce. To be honest, the AJAX Toolkit has always seemed like a workaround hack to me, and hopefully the combination of Visualforce and Apex Code will render it obsolete.

The Developer Guides

These are must-prints. As more thorough extensions to the ebooks above, these guides are the definitive resources for developing with Apex and Visualforce.

Frequent Updates

Both the APIs and Developer Guides are updated with every release (Spring, Summer, Winter), so make sure to grab the newest versions when you see your org being updated, and please do recycle the old versions if you print them out.

I suggest bookmarking this post, so you can quickly access these PDFs instead of browsing through the developer.force.com Wiki. Enjoy!

Opening Gems in Textmate

2008-04-12T00:00:00-04:00

Here’s a problem: you need to look at a gem’s source code to figure out something, but you need to figure out the path to your gems directory, find the gem in question, and then open it from there. What a pain. Luckily, Dr. Nic has saved the day:

$ sudo gem install find_gem
$ find_gem activerecord

This also comes with the wonderful command ‘edit_gem’. Make sure you have your environment EDITOR variable set in your .bash_login|.bash_profile|.bashrc:

export EDITOR="mate"

Save the file and source it:

$ source .bash_profile # or .bashrc

Now, “edit_gem activerecord” does what you think it should do. Nice little pieces of functionality like these really make the term “gem” so appropriate. Hat-tip to Graham Ashton for sparking my interest to find a solution.

Presentation on Thin

2008-04-09T00:00:00-04:00

Tonight, I’ll be doing a Lightning Talk for the Atlanta Ruby Users Group on Thin, a Ruby web server. Below are my slides, accompanied by several resources:

| View | Upload your own

Thin is in!

Salesforce Tools of the Trade

2008-04-07T00:00:00-04:00

Salesforce.com is billed as Software-as-a-Service: a complete software experience contained in the web browser. While regular users will never have to leave the warm comfort of their browsers, the reality for administrators and developers is much different. We depend on several essential tools to wrangle our data. Luckily, all the following tools are downloadable for free, and versions are available for both Windows and Mac OS X.

Apex Data Loader

Windows Installer | Mac Installer

Originally called LexiLoader (the Mac application is still called this), the Data Loader is the most essential tool for getting data in and out of Salesforce. The Data Loader connects via the Force.com SOAP API and understands the ubiquitous format of CSV (comma-separated values). When exporting data out of Salesforce, you’ll need to use SOQL, Salesforce.com’s SQL-like language for querying your data. SOQL doesn’t work exactly like SQL, and at times can be maddeningly confusing. But have no fear, as we have help.

Force.com Explorer

Windows Installer | Mac Installer

The Explorer explores your data. With it, you can build and execute SOQL queries, view the results, and edit the data (one record at a time).

The Windows version includes a SOSL Tester, which let’s you play with results from SOSL, Salesforce.com’s search language.

The Mac OS X version, SOQLXplorer, doesn’t include SOSL support, but does include a nifty graphical view of your data schema:

Let’s say that a city changes it’s name from Raritan to Edison (it’s happened before). So your workflow for updating contact data using the Data Loader and Explorer looks something like this:

Use the Explorer to figure out the correct SOQL query for all contacts in Raritan
Using the Data Loader and your cool query, export the data to a CSV file
Open the CSV file in Excel, and replace the City column with “Edison”
Fire up the Data Loader again and update the fields using with the new values in the CSV

Three different applications, four steps. Whew! If only there was a tool that did all of that in one neat little package…

Mass Update Anything

Windows/Mac Installation Instructions

Mass Update Anything, as the name implies, updates fields in bulk, based on a SOQL query. In addition, you can choose to delete fields in bulk as well. It’s a great little tool that does one job very well, and will save you from CSV hell.

Force.com IDE

Windows/Mac Installation Instructions

Last but not least, the Force.com IDE, based on Eclipse, is essential if you’re going to write any custom code for Salesforce. It has more features than I could ever describe in one post, so your best bet is to just visit the page. The IDE also includes an integrated Schema Browser (like the Force.com Explorer), and I’ve found myself using that rather than the stand-alone app.

Summary

So there you have it. The Data Loader loads data. The Explorer explores data. Mass Update Anything does what it says. And the IDE provides you with an integrated development environment for Apex Triggers and Classes, S-Controls, Visualforce, and Metadata. These are the tools you need to kick ass, so get ’em and get started!

Setting a Default Account Using Triggers

2008-03-28T00:00:00-04:00

In Salesforce, every Contact record must be matched to an Account in order to be viewed by your entire organization. Unfortunately, users uploading in new Contacts via data import, or entering a new Contact manually, usually forget to enter in an Account field. Later, when other users search for that Contact, they won’t be able to find it, as Salesforce considers a Contact to be private (and thus not searchable), if it’s not linked to an Account.

Since the Account field on Contacts is standard, we can’t just modify it to make it a Required Field. Oddly, creating a Workflow to specify an Account when one doesn’t exist doesn’t work, for reasons that are a bit of a mystery to me. Enter Triggers: a programmatic way to take some action before or after a record is created, updated, or deleted.

First, create a generic Account that can function as a catch-all. I suggest “Individuals”. Then, use either the Apex Deployment Tool or the Force.com Toolkit for Eclipse to create a new trigger (using those tools will be the subject of future posts).

trigger SetDefaultAccount on Contact (before insert) {
	For (Contact nc:Trigger.new) {
		If (nc.AccountId == NULL)
			nc.AccountId = [select Id from Account where Name = 'Individuals'].Id;
	}
}

The code basically says: "Before a new Contact is inserted and if it doesn’t have an Account Id, set the Id to the Account named ‘Individuals’.

In Eclipse, you’ll be prompted to provide some test coverage for the trigger before deploying:

public class testcontact {
	static testmethod void mytest1() {
		Contact c = new Contact(LastName='test lname');
		insert c;
	}
}

Awesome! Now you’ll never have to worry about anyone in your organization forgetting to link a new Contact to an Account.

Hat-tip to Mike Rosa @ Salesforce.com for helping me out with this.