Here’s the thing. The “old” Kindle 2 works great. For me, it’s the perfect size - it fits in my laptop bag, roughly the size of a paperback. It’s got paging buttons on both sides, which is great if you’re like me and you keep shifting around and switching hands when you read. For plain old text, like a novel, it’s just the ticket.

I also read the Wall Street Journal on it; it actually works better than I expected. It takes a little getting used to how the navigation works - but once you get it, it’s pretty easy to work through the paper pretty quickly.

Where it breaks down in my experience is reading technical books which have charts and figures that you want to refer to when reading the text. If the chart fills a half page, say, then there’s not much text left on the page…and if you need to continue to refer back to that chart when reading the text, it’s pretty inconvenient. I never noticed this until the past few weeks, when I’ve been working through a book on stock trading, and there are of course lots of charts the text refers to.

This would be the same situation for textbooks, I would imagine. Lots of pictures and charts, and lots of text referring to them. Keep it all on the same screen - just like a book tries to keep it all on the same page. If the Kindle DX can do that effectively (and I have no reason to doubt that it can), it will be a great device for these use cases.

Will I buy one? Well, not right away…I like the form factor of the Kindle 2. But when I’m reading technical books, I’ll undoubtedly quietly covet a DX.

Yesterday evening, I was meeting someone at Starbucks, and she was running about 15 minutes late. I was fiddling with my iPhone, and remembered that I had the new Kindle app on there, and had already downloaded the book I’m currently reading on the “big” Kindle.

I opened the app, which was already on the page I’m currently on. I read for a bit, and actually finished a whole chapter before my friend arrived. When I was done, I closed the app, and it did its magic sync back to the Amazon cloud with the page I had made it to.

When I got home later, I turned on my Kindle just to see what would happen. I clicked into the book from the home screen, and a message popped up saying something like “you’ve read up to location 2500 on gregr’s iPhone; would you like to move to that location now?” (those weren’t the exact words, but pretty close). I clicked yes, and that’s all there was to it - I was exactly at the point I left off earlier.

Pretty cool - better than I expected. I can totally imagine reading a bit of my book when waiting for an appointment, standing in a long line, or something like that.

So the good and the bad?

Good:

- Pretty much does exactly what you’d hope. You can read your book, and the rest of the GUI disappears.

- It’s free!

Bad:

- While the “swipe” is intuitive to change pages, it’s not very much fun after you’ve done it 50 times in a row. They should make it so if you tap somewhere, it skips to the next page.

All in all - it’s not the same as a Kindle, or similar to a book for that matter. The screen is small, and it’s backlit and less comfortable to read, at least for me. I wouldn’t want to read a whole book on this screen myself - but for short breaks, it works quite well. And somehow, I feel like my e-books are worth more now that I can read them in multiple places. :-)

I never had a Kindle 1. I looked at a friend’s once, and read some reviews and such, but that was the extent of my experience. But when the Kindle 2 started shipping last week, I ordered one. I’ve now read about one and a half books on it, and wanted to write about it.

First, why did I buy one? Well, a few reasons:

1. I like to read bestseller types of books, usually paperbacks; I don’t like hardcovers as much, as they’re too big to fit in my bag, and more expensive. But, this means I’m usually a year or so behind my favorite authors. With the Kindle, the brand new books (otherwise only available in hardcover) are only about $10.

2. I travel a lot, and I like to read on planes. But don’t you hate it when you only have 40 pages left in a book when you get on, so you have to take two books? And, to add to the problem, I’m a procrastinator, which means I’m always perusing the airport newsstands looking for a new book…and if there’s nothing there that sounds good, I have to settle for something that doesn’t.

3. I’ve got hundreds of old paperbacks piled up in my house, and have no idea what to do with them. Can’t really sell them, as they’re generally not worth anything. I could give them to friends, but I’ve got more books than I have friends who read the same genres. And based on past experience, with only a couple of exceptions, the likelihood of my re-reading one of these books later is approximately zero.

So…in came the Kindle.

My first impression? It’s thinner and lighter than I thought. I also got the leather cover for it, which snaps onto the Kindle with a couple of metal clips. I’ve heard a lot of complaints about this cover, but it seems to work fine for me.

Downloading books works as advertised - and it’s truly seamless. Seems to take about 20 seconds for a new book to download, which I can’t complain about.

The menu system seems painfully slow. I think it’s just the nature of the display, in that updates take a while. This doesn’t affect the reading experience at all, though - so it’s only when you’re in there screwing around do you notice it.

So on to the reading experience. I was a little nervous about this; I actually read an e-book on an iPaq way back in the day, and it was somewhat painful…enough so I only read one. So I was a little apprehensive about the Kindle. I asked Brad about his - he said it took him about 5 Kindle books, and then he was ambivalent as to whether he read a new book on his Kindle or on paper. That didn’t sound so bad.

So I sat down on Saturday, downloaded a new book, and started reading it. At first, everything seemed weird…but then, actually very quickly, the Kindle sort of disappeared and I became engrossed in the book, just like I do with a paper book. The screen contrast was fine, the paging controls were convenient enough, everything just worked, and got out of the way. By the time I was halfway through the book, I was noticing how it was actually more comfortable than a regular book in some ways…for example, you can hold the Kindle, and page through it, with one hand.

It makes buying new books a little too easy. Similar to the Apple TV (which I will write about one of these days), the Kindle is like a cash register that’s hooked up to Amazon.com. Genius in that way.

I do wish it had more screen and less keyboard…but I suppose you can’t have everything. It should have come with a case, and maybe a credit for a free book. But once you get past all that, get your case, order a book, it works. Really well.

So all in all, I’m liking the Kindle a lot. It’s small, light, and should be easy to travel with. I don’t notice it when I’m reading. It just works.

(Note - Amazon.com links in this post are affiliate links; if you use one to buy something, maybe I’ll make enough to buy a new book!)

First, let me get this out of the way - RSS use in the enterprise is definitely alive and well. But it’s not in the obvious places. No one is writing articles talking about how their desktop feed readers are revolutionizing the way they do business. No one is talking about how they’re retiring their Exchange servers because so much content is delivered via RSS instead of email (and in fact, email is alive and well). No one is saying “if I only had Google Reader behind my firewall, I could save millions of dollars.” Few companies even say their users are clamoring for some sort of enterprise RSS application.

So if not all of that, then what?

My team and I, collectively, have detailed conversations with at least 50 different large companies every week, talking about the real problems they do want to solve. Many of these include 10 or more people on their side, ranging from IT folks to business owners with line-of-business responsibility. And these conversations rarely start with any mention of enterprise RSS. These include:

• Portal enhancement. They want to pull additional content into their portal to make it more useful, or get content out of the portal to distribute in some other way. Or in many cases, they want to add personalized content capabilities to the portal. A while back, an analyst told me “this could be the technology that saves the corporate portal.” Indeed - that scenario is playing out today.
• Alerting. There are many use cases where folks want to alert certain groups or individuals of a specific event. For example, we have a large bank that uses our system to deliver fraud alerts to teller desktops, replacing a legacy system that worked via fax and paper. Incredible efficiency gains and cost savings.
• Competitive tracking. Or tracking any other external news for that matter - many companies wish to track media mentions and online conversations about themselves, their competitors, or anything else; once they have this information, then, they need to filter and distribute it appropriately. Increasingly, these systems are being used as a replacement for high-end premium content services. We have a large bank who reduced their premium content subscription expenses by over $1M per year by using our solutions - we’re talking about real savings here.
• Knowledge capture. In large companies, mountains of information are created every day, in many different systems. If there is some way to collect all of this information, and surface it in appropriate places, the content’s value is multiplied.
• Social networking. More and more, companies are investigating the idea of implementing social networking within the enterprise. The use cases for this are numerous; we have a large PR firm, for example, who uses this to manage their talent globally. A global advertising firm uses this technology to connect not only the people working on a specific account, but also others in the company who might have relevant experience or interests; it’s amazingly effective.
• Collaboration. This is sort of a catch-all, I suppose…but it’s a popular use for this technology. For example, one of largest banks in the world is using our system to unify groups of people who are on different communication systems (due to acquisitions and such); their analysis indicates rolling this out will be much faster and cheaper than attempting to unify their email systems in 2009, and provide other benefits such as social connections.

For any of these specific problems, RSS forms the underlying plumbing to transfer content around the organization. And if you can track the user interactions with the content, using enterprise RSS infrastructure, very rich data is collected - and the combination of the content plus this interaction data forms the underlying foundation for a social computing solution. One that can even form relationships based on implicit behavior, rather than explicit actions.

An interesting tidbit - of the content recently added to NewsGator Online, 40% of it is content related to social computing (e.g. Twitter updates, Facebook feeds, etc.). In the enterprise, this percentage can be even higher in active social computing environments. But the point is, RSS (and associated enterprise infrastructure) is increasingly used as the basis for social networking and collaboration.

From a business perspective, momentum for all of this is accelerating dramatically. As Brad mentioned, NewsGator added over 30 paying enterprise customers in Q4 2008 - most of whom were large Fortune 2000 companies (there is some breakdown by industry in this press release). These enterprise installations (plus ongoing maintenance) represent about 75% of NewsGator’s revenue, the rest coming from our enterprise SaaS offerings. Enterprise RSS-related infrastructure, including social computing, is growing quickly in demand.

RSS in the enterprise is not about “reading feeds.” It’s more about enterprise integration, search, feedback, etc. It’s about extending the value of existing systems, and leveraging efficient content distribution and tracking into new applications (e.g. social computing) as well as existing applications (e.g. portal enhancement). It’s not about installing FeedDemon on every desktop to enable folks to read content.

And, thus, it’s hard to see from the outside. But it’s there, forever an enabler, making new things possible. It’s enterprise infrastructure. And it’s in more places than you probably think.

Public feeds

So first, let’s discuss the general case where no security is involved. If user A and user B both subscribe to publicly accessible feed X, the feed content is the same for both - so there exists only one copy of the feed in the system, and everyone sees that same copy and the contents therein. This is highly efficient for all systems involved.

Secure feeds

Now on to the slightly more complicated case. When NGES encounters a secure feed (more specifically, a feed which requires user credentials to access), it treats the content from the feed as secure. When user A retrieves feed Y using his credentials, the contents will be stored specific to user A. When user B retrieves the same feed Y but using her own credentials, the contents will be stored for user B. So now we essentially have two copies of the feed.

Why, Greg, why would you do this? It’s the same feed, so you should only store it once. Right?

As it turns out, the answer it no. The issue is that NGES has no way of knowing if feed Y will return the same content for every user that has access to it. Just because the feed URL is the same, doesn’t mean user A and user B will see the same thing when they retrieve it using their own credentials. So, the system stores the content separately.

“Shared” secure feeds

Ah, but now we get to the heart of Peter’s issue. He has a case where he _knows_ the content for a given feed will be the same for every user that has access to it. The problem is, Peter knows, but NGES has no way of knowing.

The only part of the system that could possibly know this is the publisher of the feed - because it’s the application generating the feed, and it knows whether it would generate the same feed for everyone that has access. But it’s a valid case - there are indeed situations where the publishing system knows this is the case.

So, in NGES 3.x, it will actually look for a specific header in the feed that basically says “this feed will be the same for everyone.” If it sees that header, it will still individually authorize access for every user trying to access the feed, but it will only retrieve and store the content once. So essentially, authentication and authorization works the same way - we are just making the behavioral assertion that the content in the feed may be shared among the group of people who have access.

[note: I've slightly simplified here, but this is very close to what is actually required.]

This code was initially written to optimize some of the content-retrieval that goes on with our SharePoint-integrated Social Sites Enterprise product…but there are definitely use cases for this feature beyond just SharePoint.

As many of you know, NewsGator had TopStyle as a result of the acquisition of Bradbury Software a few years back. Since then, its development has been somewhat slow; the company’s focus was on RSS and social software, and TopStyle, while a very successful product, didn’t really fit directly into that mix.

So that’s why I’m so excited that Stefan will be taking this over, and putting some serious effort into TopStyle. To all the customers that stuck with the product, a big thank you! And now, sometime in the next few months, you’ll have a new release to look forward to…and I know that Stefan is eager to hear your feedback and feature requests, so don’t be shy. :-)

Announcement

Q&A with Stefan and more TopStyle information

Nick Bradbury’s take

The system can also automatically create topics pages, whereby the user can see a list of articles matching a specific topic of the currently viewed content.

The system uses our online database of over 3 billion articles, which sees over 8 million new articles per day. At the moment, it matches content based on tags/categories, including categories derived from analyzing the content. You can also specify “fallback” tags to use in the case where you have, say, multiple unrelated articles displayed on one page, and still want to have relevant content.

Two blogs you can look at that are using the widget are the NewsGator Widget Blog, and the NewsGator Daily blog.

Give it a try, and let us know what you think!

Update: here is a tutorial on creating a related content widget!

This data proved to be quite useful when Lightroom 2.0 was released, and I could test the 32-bit build against the 64-bit build. What I found was surprising.

Just like last time, the basic test was generating 1:1 previews for 211 raw images (compressed NEF format from a Nikon D200), about 1.75GB of files.

First, Lightroom 2.0 32-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization between 350-500% throughout
Total time 4:57, average 1.41 sec/image

And Lightroom 2.0 64-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization between 500-550% throughout
Total time 7:06, average 2.02 sec/image

So the 32-bit version on OS X 10.5.5 was actually a bit faster than my previous tests of Lightroom 1.3.1 on 10.5.2, which could be due to either the Lightroom 2.0 upgrade, or 10.5.5 changes.

But the 64-bit version of Lightroom 2.0 totally blew chunks, as you can see. Pretty disappointing, and totally reproducible for me, so I opened a ticket with Adobe. They got back to me saying they could reproduce the problem, and also had a bunch of mumbo-jumbo about how there is more overhead in accessing memory in 64-bit mode (which I’m not sure I believe, but I’m not up to speed enough on the Leopard 64-bit implementation to know for sure).

Well, Lightroom 2.1 was recently released, and here is the data:

Lightroom 2.1 32-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization between 350-500% throughout
Total time 4:55, average 1.40 sec/image

Lightroom 2.1 64-bit:

Mac Pro, 2 x 2.8 GHz Xeon, 8 cores, 12GB RAM, OS X 10.5.5
CPU utilization about 400% throughout
Total time 4:05, average 1.16 sec/image

Wow - obviously a big change; the 64-bit version now totally rocks. There was something in the 2.1 release notes about enabling SSE extensions in 64-bit mode - sounds like that might have been the bug!

I didn’t record the memory usage numbers, unfortunately…but I remember watching during the runs, and the 64-bit version seemed to be using about twice the memory that the 32-bit version did.

We have some other data as well that’s more difficult to explain, so we didn’t publish it…but generally, what we’re finding is users of the iPhone app are sticking with it and using it regularly. Said another way, the abandonment rate of iPhone users is quite a bit lower than our average rate (which is already fairly low, but the difference is significant). It seems the combination of NetNewsWire and the iPhone is a winner!

The Mechanics

All content is stored on NewsGator’s servers. When an application like NetNewsWire does a sync, it sends up some bookkeeping information up to the NewsGator system (a “sync token”), and the system returns a list of feeds that NNW needs to update. NNW then requests updates for each of those feeds (which is generally a subset of the list of subscribed feeds), again using a sync token, and the system returns the new (not yet seen) or updated articles for that particular feed.

This system is extremely efficient. For feeds that haven’t updated, NNW will not even have to request them. For feeds that only have a single new item, that’s the only data that will be returned to the client. For scenarios like mobile applications (like NewsGator Go! or NetNewsWire for iPhone), this is pretty close to the minimum theoretical bandwidth required to deliver the content.

It’s actually possible to reduce the number of calls even further, but at the cost of a potentially large (and expensive to process) response. Our APIs are instead optimized around the case I describe above.

[note: this is somewhat simplified; for example, metadata also travels both ways during a sync, but I'm leaving discussion of that out for purposes of this article.]

Details

NewsGator’s online platform processes about 3.5 million feeds, and stores about 9 million new articles per day, as of this writing. There are a total of about 3 billion articles in the system.

Suppose you subscribe to a feed from CNN.com, and further assume that that feed publishes 100 new articles per day (I have no idea how many it actually publishes - just go with me here). Now imagine you go on vacation for a month, and you come back, fire up FeedDemon, and sync. There are now 3000 articles you haven’t seen. Should we deliver them all to you?

Probably not.

Do we?

No, for several reasons. First and foremost, the user experience therein would totally suck; no one wants to wade through 3000 articles from a single feed. And second, it’s pretty inefficient to retrieve all of this content from the API - we could deliver it, but it’s going to take a lot of bandwidth to retrieve it, and a lot of work to process it. Using a mobile phone? This might well lock it up.

And the other reason is, our system allows you to mark individual articles as read, and that data is synchronized throughout the system and all of the clients you use. But we don’t store your read states for all time - we only store it for fairly recent data. Do you really care if you marked an article read 2 years ago? Probably not.

So what do we do? We have a limit of how many articles will have their metadata state synced through the system. Here’s the rule we currently use:

The number of articles in the current feed,

14 days, up to 200 articles.

Whichever of the two conditions above yields more articles is applied.

Here are three examples of applying this rule:

Imagine a blog that publishes 5 times per month, and its feed has the most recent 10 items in it. This feed would sync 10 articles.

Now imagine the hypothetical CNN feed above, which had 100 items per day, and imagine the feed held the 20 most recent items. In this case, we would sync 200 items.

And finally, if a feed published 10 times per day, and held the most recent 20 items in the feed, we would sync 140 items.

If you really want to go back and browse through all 3000 articles you missed, you still can - they’re all stored in NewsGator Online, and you can view them from the web site. In fact, you can go back all the way to when we first discovered the feed - over 4 years ago, in many cases.

The Gotcha

For most feeds, the algorithm described above makes things completely transparent, and articles and unread counts across all NewsGator-integrated applications will match up perfectly.

The gotcha is with feeds that have a lot of articles. For example, I have a smart feed for the term “NewsGator”, and I see probably 400 new articles there per day, 200 at a time. So in this case, only 200 articles have state synchronized.

What can happen is the following:

1. NetNewsWire downloads the feed, and shows 200 items, all unread.

2. 3 hours later, you sync from FeedDemon, and you see 200 items, all unread. You read them all, and mark them read.

3. An hour after that, you sync again from NetNewsWire…it syncs state from the old articles, and downloads say 25 new ones. You see 47 unread articles. You immediately sync again with FeedDemon, and it shows 25 unread. What gives?

The problem is the 200-article limit, and the fact that some of the articles fell off that ledge while still showing as unread in one application…and thus don’t have all of their state synchronized.

This problem used to be much more acute - it’s more rare now that we’ve raised the article-sync limit up to 200. But it’s still possible to run into this, specifically with very prolific feeds.

We’ve experimented with various different limits - the current 200 articles seems to be a good compromise, perfectly syncing the vast majority of feeds while maintaining the efficiency that our client applications demand. We’re also working on some things that will make this invisible to API clients, while still working within the constraints.