• NYT “Desk”
• Organization
• Publication month
• Publication year

There are other facets available too, but the ones up there now are the most user friendly. (Perhaps I should provide an “advanced” mode which would show you all of them, even the unfriendly ones.)

To choose which facets you see, hover over the “Refine your search” area—a “Customize” link will appear in the upper right hand corner.

Enjoy! Any feedback is appreciated.

Some might say I’m reinventing the wheel, but I don’t see it that way. I realize there are some great pagination libraries for Rails/ActiveRecord (will_paginate), and I’m not trying to compete with or replace them. Rather, I see simple_pagination as a compliment to these projects. It is useful when you aren’t using ActiveRecord but you have data to page through… such as when you are consuming search results from a remote API.

(In fact, if mislav wanted to, he could refactor will_paginate to use simple_pagination. I doubt he wants to though. I know I wouldn’t want to.)

I hope other people find this useful—I know I’m tired of writing pagination logic when dealing with non-ActiveRecord data, which I am doing more and more of. Anyway, enjoy!

These people are getting a lesson in making personal connections.

1. Put yourself out there. Success is still (and always will be) about personal connections. Actively seek out new connections. Go to meet ups and conferences. Don’t be shy. People are the world’s greatest resource: they are your sounding board, your marketing firm, your user base. Embrace them.
2. Keep It Simple, Stupid (aka, “The KISS Protocol”). Your next great idea is probably too big—simplify it, get it out there, generate feedback (see #1), and iterate. Getting something out there and starting a conversation will always be more valuable than weeks of brainstorming all holed up in your apartment, wild-eyed and crazy-haired.
3. Trampoline off the efforts of others: seek out and make use of the wealth of data- and service-related APIs on the web. (This also gets back to #2 and helps you keep in simple.)
4. Finally, follow your nose and have fun! Make something you think is cool. For one thing, this will help you when it comes to promotion: genuine enthusiasm on your part will probably get other people interested too. For another, this will enable you to create a better product: when it’s your baby and you love it, you are going to do your best to make it look and feel great.

(See also: Getting Real, by 37signals.)

• added spelling suggestions using the YAHOO! BOSS spelling suggestion API
• added thumbnails where available
• added a feedback forum; if you have any questions, comments, or ideas, I’d love to hear them
• modified layout slightly to be more flexible; I think it’s more attractive now too

Enjoy!

Keynote: Tim O’Reilly

For me, someone who read his first O’Reilly book (Programming Perl) 12 or so years ago, it was a thrill to finally see a Tim O’Reilly keynote (you can find my attempt at a liveblog in the previous post). In essence, O’Reilly threw down a challenge for The Times and every newspaper: to reinvent itself in the age of Web 2.0, and to discover a new business model while doing so.

The talk summarized the basic tenets of success on the web, with particular attention paid to the concerns of a newspaper. O’Reilly criticized the Times bluntly for current shortcomings, and suggested ways they could improve. Still, not all problems have easy answers. The biggest challenge for The Times is to balance the openness that characterizes success on the web with the propriety and tradition of a newspaper—and not just any newspaper, but a centuries old one: The Old Gray Lady herself.

I’m not going to cover this talk in detail because I think the liveblog below does a pretty good job. Suffice to say that the gauntlet has been cast.

API Review

At this point each developer (or development team) ran through their API (or APIs). It was a brief run through, and most of the information is available on The Times’ developer site, but I highlight the parts that interested me the most.

Newswire API

To be released next week, this will be The New York Times firehose. It will provide a real time feed to track content as it is published. At first this will feature a trimmed down data set (just articles to begin with, I think), but it will eventually include all online Times content (including blog posts, for example).

The feed will be cached for up to a second, and they are happy to have people poll it as often as they like (they kept saying that we should poll it every second, but this is flawed for a couple of reasons: 1. it will only feature about 300 items a day, so polling it every second it serious overkill; 2. we are limited to 5,000 requests per day per API key, so polling every second would put us over the limit in a little over an hour).

Article Search API

I was most intersted in Derek’s future plans for the API:

• Open Calais integration (which is a great idea and a thought I had myself)
• Related document search
• More fields
• More article data
• OpenSearch RSS/JSON

After the API reviews we broke for lunch. I got to meet Steve Myers from The Poynter Institute, Gilad Lotan of Global Voices Online, and a handful of others (if you were one of the others, please leave a comment and identify yourself!).

R&D Presentation

The next session was an overview of R&D at The Times, including a presentation of shifd, which was incubated there. I actually missed this presentation because I got to go on an unofficial (clandestine?) tour of the R&D departnment, which I would happily take over a presentation any day. I’m not sure how much I’m supposed to talk about this, but they are working on some very cool stuff, including a fully wired living room that uses your cell phone to identify you and show you localized content tailored to your preferences. They also have a killer view from the 28th floor of The Times building.

Interactive News

Jacob Harris’s presentation was one of the most intruiguing of the day (and one of the most difficult to put together, I imagine). He started with a premise that interactive news can be described as a triangle whose points are:

• Data is the raw material - that from which news is forged
• Story story gives data structure; in traditional media, this is what turns data into news
• Interaction gives news dimension, makes it compelling, and gives users a reason to use it

Often, interaction has thus far come down to navigation (navigating a data set, for example). Jacob thinks we can do more, and hopes to see some really interesting resources for Interactive News grow out of the developer community.

The Alcohol API

When the day was over, we gathered over food and what the organizers insisted on calling cocktails but what I call beer and wine. (Brian Kennish had termed this the Alcohol API.) I finally got a chance to meet Derek G at this point, which was nice because he had some nice things to say about my nytexplorer app, and I had some nice things to say about his API.

We ended up going to a bar after that, and things devolved and the group waned. In the end it was just me and David Billingham. Unfortunately I had to run to catch a bus, because I wished I had more time to spend with him.

Conclusion

The New York Times recognizes the need for a news platform, and wants to fill that void. Their developers understand that Web 2.0 is about openness, and are committed to opening up The Times data to outsiders (whether or not the suits—and they do wear ridiculously nice suits—agree remains to be seen). They want to create a community of hackers around their APIs, and invite community feedback to drive future innovation. The Times executives should be supremely grateful for having employees with such vision at their organization.

My most sincere thanks go out to Derek and the whole Times dev team for hosting this event. I am thrilled to have been there. I don’t care what people say—this event was worth every minute. (I was lucky to get in at all—I was literally the last person on the list, and only got on there because I sent nytexplorer to Derek on Thursday afternoon).

Until next year?

I’ll update this as much as possible throughout the talk. Enjoy!

11:18AM - Clay from Sunlight Foundation - “why are you headed in government 2.0 direction?”; O’Reilly wants to see creative innovators working on “problems that matter”; wants to “bring silicon valley to washington”

11:12AM - rep from yahoo news - how do we (the news orgs) work together to make something better, rather than competing? example: one interface for news search APIs, rather than a bunch of divergent ones. O’Reilly picks right up on it and totally agrees that this is the way; parallels to social networks; it’s all about interoperability; the faster we adopt open standards, the sooner we can create the really interesting solutions

11:09AM - great question: “how long do we have” as people working at news organizations?; O’Reilly: “48 hours” (haha); “it starts now” - if you’re not “doing it”, start; this will be a pruning - lots of orgs will go out of business, those that don’t will get stronger; again: news isn’t going anywhere - if you look at what news is being passed around in social circles, its largely from big publishers (techcrunch as example)

11:05AM - Q&A; Gottfrid appreciates the blunt criticism of NYT

11:04AM - News and the need for it isn’t going anywhere. Just need to harness that need into a business model. Alan Kay quote: “The best way to predict the future is to invent it”

11:03AM - search for a business model as a road trip; you have to keep gas in the car, but this is not a tour of gas stations; focus should be on building an interesting future consistent with the value that we give to our users (not, “where is the money going to come from?”)

11:01AM - search for a business models starts with a question: “What Job Does a Newspaper Do?”

10:58AM - Theme #5 - “Asymmetric Competition”; Craiglsist 7th top site on internet in 2005, with only 18 employees! The rest on that list have thousands, often tens of thousands; the idea is to demolish old business models; compete.com: nytimes.com vs washingtonpost.com vs huffingtonpost.com; then add digg.com, whoa!

10:57AM - lessons from twitter: do one thing and do it well; let others build on what you do, even if it appears to compete (this is what makes it a platform); when users innovate, support their behaviors in your platform (twitter: @, #); “insert and extend”

10:55AM - twitter a great example of this - not only client apps but user generated stuff (hash tags for example) as well as fully feature third party apps - http://bit.ly is analytics for twitter

10:52AM - Google as a great platform because of integration with external data; geotagging on google maps via iphone was driven by hackers, then google supported it; if you are building a platform, customers/partners may innovate features before you do

10:48AM - “We are building the global electronic brain”; NYT platform is one piece of that; one of O’Reilly’s “beefs” with NYT: they don’t do a good job of linking; linking as critical to enriching the web; apture.com is a cool tool related to this

10:46AM - Web 2.0 theme #4: “The Internet as Platform”; there are two types of platform: “One Ring to Rule THem All” and “Small Pieces Loosely Joined” (O’Reilly asks: “how many of you have read Lord of the Rings?”, about half the people here raise their hand); showing a cool “internet routing graph” titled “The Whole Internet”

10:45AM - Google mobile app - speak “pizza” and it finds you restaurant - so many sensors! accelerometer turns on mic when you lift handset; mic captures voice; cell signal captures location

10:43AM - “Instrumenting the world” - using sensors beyond the keyboard/mouse to power collective intelligence - see http://code.flickr.com/blog/2008/10/30/the-shape-of-alpha

10:42AM - programmers can contribute to journalism by creating a context where people can connect and share

10:41AM - Search blog.wired.com for “Group Spots Giant Hacks by Combing Small Newspapers”

10:38AM - Web 2.0 theme #3 - “programming as journalism” - usaspending.gov, which was purchased/cloned from fedspending.org, which is basically programmer driven investigative journalism

10:37AM - markmail.org - searches tech newsgroups and shows you trends; also shows the most influential people in that community

10:33AM - personalities are critical - don’t just show me aggregate data, show me who is a part of it; gives users an incentive to participate (increase in status); example: NYT shows “most blogged”, but doesn’t show who’s blogging; cites techmeme as getting it right

10:31AM - TimesPeople as “one more social network” - O’Reilly’s TimesPeople page is a ghost town - wants to see outside content there (twitter)

10:30AM - “In social networks, you gain and bestow status through those you associate with”; “bestowal of status by what you pay attention to”; “if you only pay attention to yourself, you aren’t as valuable to your readers”

10:27AM - Twitter more compelling that Facebook (for Tim O’Reilly and anyone who has a large public persona - i.e. lots of followers - 30,000+ in his case)

10:26AM - Web 2.0 theme #2: Social networking; cool graph from mediaczar.com of politically oriented tweets

10:25AM - Question for newspapers is: “what assets do you have that increase value through participation?”

10:23AM - “The network as platform means that competitive advantage goes to systems that harness network effects to get better as more people use them”

10:22AM - Real Time is important - twitter as real time news; mybarackobama.com and project houdini (real time updates of get out the vote do-not-call lists as voters went to polls, the idea was “don’t call people who have already voted”)

10:16AM - Web 2.0 theme #1: “Harnessing collective intelligence”; best exemplified by Google and PageRank - implied social network - extracting hidden information that everyone else was throwing away; find meaning in user-generated data and create useful services from it; “harvesting meaning from user activities”

10:14AM - “You guys are out there inventing the future, and one day everyone is going to be living in it” (Twitter a good example)

10:12AM - Tim O’Reilly is on stage - how many people here have O’Reilly books? (Everyone raises hand)

nytexplorer is built on top of the Times’ new Article Search API. This is a really interesting API. There is a ton of data behind it, and the data is super clean and easy to work with. Lots of possibilities here.

This has been a really fun project to work on. If you have any feedback, please post it here for now. I will setup a proper feedback forum (I’m thinking UserVoice) soon.

Don’t try this at home

Well, try, but don’t get your hopes up. I happen to work literally blocks from MIT, which is where I grabbed this from.

• Cost we are saving $152/month, a 44% savings
• Uptime this is based on word of mouth, but I’ve heard that Slicehost’s uptime is nearly impeccable. In our few months with Rails Machine, our VMs have gone down two or three times
• Speed I was able to configure my slices so that I get a far larger CPU share than I did with Rails Machine
• Confidence I started out with Rails Machine because I wanted to be able to fall back on their support team if I needed it (I am no expert when it comes to Rails deployment). It turns out that I have only used their support staff for routine stuff - restarting VMs that have gone down, for example - and not for anything specific to Rails. Plus, I am switching to Phusion Passenger, which makes deployment so simple a monkey could just about do it.

These are just my main reasons for switching. There are other benefits, too, like a simple automated backup system, and a great web interface for managing my slices.

Now, onto my review (admittedly, it may be a little early for a review, but at least I can review the signup and install process).

Signup

The signup form is streamlined and simple. You give them your billing information and tell them what kind of slice you want and which OS. From there, a slice is automatically provisioned. In my case, it took less that a minute to receive confirmation that my slice had been created. Ten seconds later I was logged in as root via ssh. This is by far the fastest and simplest signup process I have ever seen.

Lock Down

Since Slicehost gives you a VM with root access, it’s really up to you to make sure it’s secure. That was pretty easy since they give you a simple, easy to use guide describing how to do this. Obviously there are no cookie-cutter security situations, but they give you a great set of defaults that result in a pretty secure box.

Rails Stack

From there I needed to install the Rails stack - in my case, Apache, mod_rails, Ruby Enterprise Edition, MySQL, and Memcached. Slicehost provides guides for everything but Ruby Enterprise Edition and Memcached (and the guides are tailored to your OS, which is nice). REE and Memcached are really easy to install, so I didn’t miss guides for these very much at all.

Staging Environment

After setting up my production-box-to-be, I wanted to go ahead and set up my staging server. Although the production setup was really easy, it did take some time (four to six hours, I would say). This has nothing to do with Slicehost and everything to do with (a) my perfectionism, and (b) my slow, calculated approach to sysadmin tasks (which results from the fact that I am a programmer, not a sysadmin). The second time around would have been quicker (by far), but wouldn’t it be nice if I could just clone the box I had just setup instead?

Luckily Slicehost makes this super easy. First, I turned on backups for my production box (an extra $30/month, but something I wanted anyway). Then I took a snapshot, which took about 5 minutes to complete. Then I created a new VM, and selected my snapshot as the image (rather than a bare bones OS install). The build out took under a minute again. I logged in, tweaked some stuff specific to staging (changed the database name, for example, as well as Passenger’s RailsEnv setting), and everything worked!

So it took about 20 minutes to setup the staging environment.

Conclusions

So far, I am loving Slicehost! I can’t comment on uptime or customer service just yet, but so far this is the best experience I’ve had with setting up a VPS. Slicehost appears to be a best-of-breed hosting service, and I’m glad I found it. Thanks, Slicehost!

For one thing, the terrain in Little Cottonwood Canyon is incredible. Alta and Snowbird combined offer either inbounds or “sidecountry” access to just about any type of terrain you can imagine, including extremely steep couloirs that look as if they belong in Alaska.

And then there is the snow. I will write a more detailed post later, but suffice to say that it has snowed over 50 inches since I arrived. Andy has been an excellent guide, and I am leaving tired and happy.

I came here in search of something I had never experienced before, and I found it. And for that, I am sincerely grateful.

(OK, I wrote the last three sentences just now, in Boston. But I wrote the rest Friday evening, in the moments before my cab arrived.)