geek!daily

Live Blog: Comet & Other Browser Stuff

2009-10-15T13:27:25-07:00

It's a big component of what we call "real-time"

This is the kind of stuff that puts up a banner on a Twitter search ("# New Tweets Found") and makes things feel real time.

List of what's being done to move notifications from server to browser:
* Ajax polling: Anybody using the JS set-timeout function, then refetch. Used by GMail, Campfire, et al
* Long polling (typically Comet): Used by FriendFeed, keeps an http connection open as long as possible (keep alive), reopen when timed-out. Keep a local thread pool.
* Flash sockets: Same as long polling, but from Flash. Don't have the same-origin policy. No limitaions on number of connections. Does gzipping.
* Reverse HTTP: hosting a little webserver inside a JS connection. Long pushing?
* Silverlight ("MS's version of Flash")
* HTML5 websockets: part of spec, doesn't exist and not implemented. Expected to be like flash sockets.

(things discarded as too old/obscure/painful/absurd):
* Java applets (David Weekly is about to get himself kicked ;)
* Big ugly JS kludges (iframe tricks, etc)
* What's that Opera thing ... unite? ubiquity? Unite. Every browser is a webserver. Sounds like reverse HTTP.
* Using XMPP format, arbitrary JSON structure, Atom.

Libraries:
* Orbited (?) tunnels TCP thru HTTP, treats each end like a socket. JS front-end, Python backend. Often paired with Twisted. Specifically meant to be Comet.
* stropheJS: javascript, can use flash sockets
* Tornado: python
* Cometd: Java, made by Dojo to work with dojo
* Dojo: javascript, can use flash sockets
* APE project (?)

What formats are people using to send data:
* JSON
* XMPP

Apps and their libs/methodologies:
* Meebo: ??
* Google Wave/GTalk: GWT RPC, long-polling, their own JS
* FB Chat: ??
* FriendFeed: Tornado server with long-polling, their own JS
* Superfeedr: uses BOSH, stropheJS. BOSH is kinda long polling, bidirectional (two cs open all time). Very similar to comet, more friendly than strict proxies
* Collecta: uses BOSH
* Twingly: orbited
* PBWorks: long polling on network dashboard to see updates come in live; wrote their own libraries, also use stropheJS
* StatusNet - identi.ca and ??, orbited and cometd (you can do either)

Flex and Air apps? The most interesting stuff is what used to be called Flash Media Server (been renamed). They've got their own P2P protocol. All sorts of funky stuff you can do. OpenSource version of FMS is Red Five.

In what situations are these libraries breaking down?
* Transparent/Opaque proxies that give repeats/dupes/hangs. Keep buffer on server and check for these. Sometimes have to close connection to flush thru proxies.
* "Everyone focuses on the newest hottest event-based framework, but the hardest part is that HTTP wasn't designed for long-polling."

Have to hold open request/response pairs. Connection setup and teardown is expensive

Guy at UK telecom has only ~3K IP connections available in London area. Comet is going to force an upgrade of their hardware, which will be expensive. Real world constraints will always present.

Real-time at UI can be distracting at best, horribly annoying at worst. Charts work great, but text moving too quickly becomes hard. UX is challenging. Would be nice to add items while autoscrolling relative to the focussed item

When you want to add a pause button to your site, that's FAIL. But the hover-over-conversation to pause semantic is pretty good.

Go to jschat.com to see a bad resize.

FB pioneered notifications really well in UX. You get a "toaster popup" that doesn't disturb your screen and it fades after a few moments plus a bar at the bottom to persist the aggregated notifications/count. Really nice.

Growl is also mentioned as a good model. Is there interest in a notification aggregator with contexts in the browser? Meebo wants to go that way as a notification aggregator. Adobe wants to do this on your desktop. BrowserPlus also hits Growl. Prowl == Pushed Growl.

If everyone started using XMPP, it would increase the message load ("three bazillion individual messages"). No one is bundling XMPP updates; no reason you couldn't. ActivityStreams could also serve.

Why are people using Atom/JSON for this? They're so extensible, but when you extend them so far they're just as verbose as XML. "But we just don't like XML."

Would be nice if you could select things to pull out of the stream and hold onto.

Seems to be consensus that chat belongs at the bottom of the browser, append at end. Everything with a permalink (blogs, tweets, etc) gets added to the top (?). Seems to be related to height issues.

Infinite scroll vs. "More" button

Titlebar flash (Gmail and GTalk)

Some sites make a tiny flash widget to play a sound

Haptic: twitter dmesg to cellphone buzz

Notifications can become another form of info overload/noise

(We flailed at making a 1D/2D chart to represent frequency and value ... FAIL)

Phonetop apps are like desktop apps. Leah feels strongly that they're headed down-and-to-the-right and browser-ish apps are taking supremacy due to interoperability.

Apple's surprise: phonetop apps took off. (?) vs. John Gruber: "The most used app on my iPhone is Safari."

You can find the whiteboard pics in my Flickr stream.

Live Blog: Web Aggregation, What Works and What Doesn't

2009-10-15T12:17:59-07:00

[note: I originally scribbled this on paper thinking I could hand it off immediately, preventing the obligation of typing, posting, etc. Turns out I don't get off that lightly, so here's the spew in electrons.]

Scraping isn't a scalable model.

There are biz issues around aggregating data: many businesses don't want you to get their data, though many are becoming more open.

Doing aggregation right:
* minimize latency
* maximize engagement

When latency is high, it causes confusion and takes you out of real-time

Doing conditional gets can be somewhat useful.

Plaxo had to shard their crawlers, which lands you in the shared state/sync problem of any stateful system you want to scale horizontally.

Gnip integration has been good:
* Offload the long-running processes
* Gnip offers alerting or "fat ping" (ping includes update data)

Plaxo likes using the alert to escalate the priority of the crawler which fetches the rich data related to the update. This approach allows you to use a consistent model for content ingestion vs. get info from fat ping, then augment later.

Smarr: "Brad Fitzpatrick said, 'Make polling a special case of push.'" He attributed this to someone but I missed the attribution.

(Don't try to keep up with Joseph Smarr on paper. He's thinks too many cogent thoughts too quickly to preserve legibility)

Plaxo uses TripIt's RSS feed as alerting, grabs item ID, then uses their APIs to fetch rich data.

There's a move to homogenize the info from sites, which may not be a good idea. It suppresses the distinctive look and feel/experience of the publishing site. Allowing for these differences means more labor spent on making one-off shims, which increases maintenance. Still, right choice in order to provide value to the user.

Activity streams seek to provide more rich data in a somewhat normalized, extensible format.

Many/most sites aren't yet perfectly architected for real-time's push, ping, etc.

PubSubHubBub and Activity Streams are externally represented data shards

Plaxo's Pulse started with known architecture issues (in order to ship) and hit the wall sooner than expected. Threw hardware/software optimizations at the problem to move the wall far enough to give time for rearchitecture, sharding, and working out how to propagate changes throughout the system properly.

None of the NoSQL alternatives are quite ready for prime-time. Smarr: "It should be something that's just a primitive."

Conversation platforms are slightly different sorts of aggregation platforms. There are UI diffs (e.g. pause the stream when indicating interest). Handling the transition from slightly-latent/passive real-time to synchronous real-time/active not yet well-developed (think: when a comment inspires a conversation)

90-99% of the value of the real-time web is realized in not-real-time [unreal-time? ;] This is a big deal for discovery. Twitter and FB make this harder by obscuring history.

Ideal scalability/performance would be an index per user. This would be grossly inefficient due to the number of duplicate entries.

No one has nailed reader-controlled aggregation (Show me Joe's tweets and blogs but not his photos) quite yet.

Smarr: "If we're all kinda [sharing], we're all making each other smarter"

The firehose of info is a hard model to scale to. Ben Metcalfe proposes the garden hose -- a firehose filtered at the source according to your interests, which helps aggregators by allowing them to request the superset of all filters from a given publisher.

We really want to push contexts to the publishers and let them determine which content fits that context. Context shifts over time: Joe doesn't normally read my tweets (and why would he?) but when we're at a conference together, he's much more interested (thus the popularity of hashtags). This is a geographic and purpose-driven context (the conference) as well as Joe's context on me (Jim knows where the good bars are).

Folks like Twitter are so overloaded with info that they might not recognize non-immediate contexts that are interesting to me.

There's also the risk of exposing users to the amount of correlatable public data they have. Many don't want you to apply a transitive closure to identify them in all spaces even though doing so allows you to present a much more convenient UX around what they want you to aggregate.

Someone likened the real-time aggregation problem to a bar conversation: you get snippets here and there and follow your own thread of interestingness.

Three fundamental themes:
* How to specify contexts to data provider/publisher
* How to control access to private data (and carry ACLs with that data)
* How to do all this efficiently

Plaxo implemented polling-back-off (poll infrequently updated sources less frequently). Turns out this is a bad idea, as it introduces latency which makes it feel broken.

There's also the issue of aggregating conversation about web objecs (like blog posts) and how not to divert the conversation from the publisher's site. However, sometimes you want a private discussion of a public object (cf. LinkedIn company groups discussing an article)

Q: What's the state of open standards around this?
A: PubSubHubBub and Activity Streams are very exciting. OAuth as access delegation. There's still a lot of ground to cover.

Stepping Into BDD with Cucumber and WebRAT: Structuring the Work

2009-09-13T14:45:32-07:00

This is the first of a series on my adventures in going BDD with Cucumber and WebRAT with a greenfield project. Watch for the next article coming soon.

I'm working on a tiny project for the dads club at my son's school and using the opportunity to plunge fully into BDD with Cucumber. I've been tinkering with it a bit here and there, but this time I'm committing to fully specifying my acceptance criteria before I start writing specs for my implementation. I have to admit it feels a bit onerous right now, but I can see the value coming so I'm sticking to it.

My usual working methodology is:

Write a quick set of use cases
Infer features from the use cases
Divide the features into logical delivery phases
Start spec'ing the implementation
Code to the specs
Profit!

... so really I'm just adding a "Specify features using Cucumber" step between 3 and 4. Sounds small (but it's not).

The use cases for this project are pretty easy:


Use Cases:

NON-MEMBERS:
* Join

MEMBERS:
* See news postings
* See calendar events
* Subscribe to a calendar feed
* Read and send email to the group
* Maintain a member profile (email, phone, location, kid info, etc.)
* Browse and search a member directory (by grade, class, proximity/map, etc.)

PREZ FOR LIFE:
* Write news postings
* Post new events
* Send news/reminders to various publishers (group mailing list, school newsletter editor, etc.)

(Yeah, we have a President For Life. He's awesome.)

Time to break these into features. I try to scope each delivery to be a useful increase in features and something I can delivery quickly (I don't get much geek time these days ;). Some of this is old hat (it's mostly just a CMS), but some of it calls for me to learn about things I've never played with (Google Maps APIs and Geolocation, for example). Additionally, it was important to get something up fast last week so the main URL could be included in some printed materials for back to school week.

Here's the feature rough divided into phases:


Phase 0: Site
Non-Members can see front page

Phase 1: Members
Non-Members can join
Members can sign in with email and password or OpenID
PFL can manage news items
PFL can manage events
PFL can manage pages
PFL can suspend accounts of non-PFLs
Members can view news and events
News snippets and event titles show on top page

Phase 2: Admins
Members can be made admin by PFL
Admins can manage news items
Admins can manage events
Admins can manage pages

Phase 3: Feeds and Mailings
Members can subscribe to news and events feeds
PFL can set reminders to be sent by email
Members can recover passwords by email

Phase 4: Profiles
Members can add email, phone, address, availability, notes, profile pic
Members can add kids: name, age, picture, teacher
Members can mark info private, admin-only, or open

Phase 5: Directory
Members can browse directory info

Phase 6: Maps
Members can see people, events on map

Phase 0 was "get something up fast;" we made it a simple redirect to the existing Google group. Phase 1 is the biggest chunk of features, but they're very straightforward. Each phase after that introduces enough new functionality to be interesting for the members, but in small enough chunks that I should be able to knock 'em out in a weekend, plus-or-minus.

Okay, so now I'm ready to generate the app and cucumber-ify it. I'll cover that in the next article.

Generate the regex for a TLD hostname from Perl

2009-08-18T19:05:19-07:00

This was a quick, fun exercise to remind me that I can still write Perl. It fetches the list of TLDs from IANA, does a quick bit of munging, then renders a regex which should match any valid FQDN:


#!/usr/bin/env perl
use strict;
use warnings;

use LWP::Simple;

my $fqdn_regex;

if (my $content = get('http://data.iana.org/TLD/tlds-alpha-by-domain.txt')) {
  $fqdn_regex = '(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:';
  $fqdn_regex .= join('|', grep (!/^(#|xn)/i, (split /\n/, lc($content))));
  $fqdn_regex .= ')';
}

my $regex = $fqdn_regex . '(?:\s|\/|$)';
print "$regex\n";

Several caveats:

It doesn't match IPv4 dotted quad nor IPv6 ::-notation
It intentionally ignores Internationalized Domain Name in Applications (IDNA) domains
It borrows from my favorite reference for this, regular-expressions.info's page on email address regexes.

Maybe I'll extend it for completeness and/or rewrite it in Ruby someday. Until then, it'll always be ~/bin/tld_regex for me.

Gmail no longer beta? True dat.

2009-07-07T10:12:31-07:00

I guess it's no longer under construction, or so sayeth the GMail blog which also shows those who can't live without that ubiquitous beta tag how to bring back their old friend:

Michael Jackson's Obit via Wordle

2009-06-25T18:40:55-07:00

Story by Yahoo!, picture by Wordle.net:

LiveBlog: 10+ Deploys A Day: Dev and Ops at Flickr

2009-06-23T13:47:46-07:00

Update 1: Slides are now on SlideShare.

Update 2: Video now available on blip.tv

John Allspaw (Ops) & Paul Hammond (Eng), Twitter

“Actually work together and aren’t huge assholes to each other.”

(omitted: photo stats … that’s a lot of kittens)

Dev vs. Ops

It’s not my {machines,code} it’s your {code,machines}

Spock v. Scottie analogy

Ops as grumpy old man, says no all the time, cycle of “no all the time because no one tells them anything because they say no all the time”

CW: dev job to add features, ops job to keep site stable and fast

Flickr: Ops job is to enable the business (Dev’s, too)

Business requires change, otherwise you’ll be overtaken by the new guy … but change is the root cause of most outages.

Discourage change vs. Allow it to happen as often as it needs to (via tools and culture)

Lowering the risk of change via tools and culture.

Increase confidence in change goodness
Increase ability to react to those changes

You need {ops,devs} who think like {dev,ops}

1. Role and Config Mgmt

2. Shared Version Control: everyone looks in the same place for everything

Code and config in same place
Everyone has access—transparency
Everyone knows how to use it

3a. One-step build

Everything you need to do to convert svn co’d code into what goes to the site—one command
They have “Perform Staging” button at bottom of a page with stats on latest commit

3b. One-step deploy

Top of page is deploy log with notes: who, when, what (link to changes)
Bottom has “I’m feeling lucky!” button to deploy
Continuous deployment

You can’t pretend to deploy 10 times a day if you go down 10 times a day. That’s not being agile, that’s being retarded.

They use Hudson to generate packages which can be deployed by ops

Small frequent changes make it easier to see what went wrong and recover when needed

4. [missed that tag]

Branching is all about managing bugfixes
Always ship trunk
Branch in code instead: use conditionals to block out pre-release features and configure off/invis—provides an operational lever for adjustment
Makes these open for private beta in production on production hdwe, etc.
Allows dark launches, which allows you to size appropriately, fix major oversights, take the fear out of major new launches

They have a couple hundred “free contingency switches” to turn things off/throttle things down. Gives broad operational control to minimize effects on the site.

Tend to fail forward using these and fix the problem.

5. Shared metrics

You can see mine, I can see yours
Use ganglia as console
Devs know where dashboard is, and watch as obsessively as ops
Includes app-level metrics (which exposes them to Ops)
(helps drive accountability in both directions in the org—both can see and feel ownership)
This begins to create opportunity to gracefully collaborate to back off an oversub’d resource/degrade/throttle as needed
Show last site deploy info on every page/graph; you can corellate a change in the graph

6. IRC and IM bots

Heavy IRC users for ongoing dialog between dev/ops, remote/local
Last.fm wrote a tool to inject events into IRC (monitoring, events, deployments, builds)
Log it all and put it in a search engine

Culture

All the tools in the world won’t help if you have a contentious culture

1. Respect

No stereotypes: not all devs are lazy/cowboys, not all ops are obstructive
respect their expertise and opinions and responsibilities (they influence priorities)
Don’t just say “no”—it’s like saying “I don’t care about your problem/perspective”
Best solutions are collaborative. Memcache is an excellent example; written to solve the problem of DB overheat, which impacted both
Don’t hide things: share your solution even if you think they’ll say no; you deny their expertise and input

Talk about the impact of your code push

What metrics change
what are the risks
what are the symptoms of somethign going wrong
what are the contingencies (rollback)

2. Trust

When you walk up with all the above on hand, you demonstrate that you care enough about them and the business
“I don’t want to tell X …” == you’re a cowboy, and “cowboys are losers”
Have shared playbooks and contingency plans so all understand.
Provide as many knobs and levers as you can so Ops can tweak to match the env
Ops: be transparent, give devs insight and access to the systems. Playing telephone around shell commands is dumb.
It’s hard to help if you can’t directly see

3. Have a healthy attitude around failure—it’s going to happen.

Think about how you’ll respond more than you think about how you’ll prevent it
Would you rather be treated by a GP who deals with heart attacks infrequently or an EMT who handles them weekly?
Fire drills: when ops and sr engineers are fixing a problem, have others diagnose live in parallel (but make no changes!)

4. Avoiding blame

they have a rule of no-finger-pointing; it doesn’t need enforcement, folks step up
fixing blame wastes a ton of time, why not skip it? Feel guilty afterwards if you must.
They’ve got a bit of a potlatch culture as people try to assert responsibility in order to fix things.
Remember that when your code breaks, someone’s going to have to wake up to fix it. Own it and apologize, at least. Otherwise, you’re back to not respecting each other (“Screw you … aren’t you getting paid to do that?”)

Ops should provide constructive, continual feedback on how it’s going. Point out interesting things before they’re critical

LiveBlog: Q&A with Twitter's John Adams

2009-06-23T13:28:57-07:00

A short pre-lunch session to absorb a few moment:

John Adams (Twitter) Q&A

Q: How do you log all the info from your APIs?

A: syslog, looking at scribe, generally summarize and toss

Q: How do you control abusive clients?

A: Rate limiting, apply feature limits to abusers, etc.

Q: What would you do differently?

A: Implemented change controls much sooner. Process is much better now with more control, predictability

Q: How does your on-call team work?
A: More people reduces length in rotation. Nagios with alerts and aggregation of alerts. Make alerts actionable (db fails? see one page for db down, not 500 webservers). Also prevents burnout

Q: Carry a real pager?

A: Some, mostly SMS. There are escalations if you don’t answer. Always someone from Ops and Eng on the pager chain.

LiveBlog: PageSpeed

2009-06-23T13:19:27-07:00

Bryan McQuade (Google), Richard Rabbat (Google, Inc.)

What’s Page Speed

FF/Firebug addon
http://code.google.com/p/page-speed
Optimizes images, minifies JS, tells you what you should defer
100K downloads in 10 days, 1000s of tweets, 100s of blog postings

How’d it start

wanted to help stop relearning lessons in new apps
keep you from unintended consequeces (e.g. vary header, squid won’t cache anything with params, etc.) Google map tiles used to have ? in URL; removing it gave huge boost in perf, huge reduction in requests
Sourced from many smart people on the web

Prioritizes according to importance/savings, gives easy way to see detailed info about any given rule, what’s violating it, docs on why and how of each rule

Defer loading JS as much as possible

Rule looks to see what’s not been invoked before OnLoad completes
Not enabled by default, has perf hit
Load multiple times in multiple scenarios; some JS is triggered in different

Inefficient CSS Selectors: based on David Hyatt’s post on inefficient selectors

Activity Panel

Show where most time is spent, where you should focus optimization
Shows DNS, network, connection, latency, data available for use, JS parse/exec, cache hits
Coming: paint events, screen snapshotting

BBC “has lots of room for optimization” gmail waterfall is hugely vertical!

Just released FF3.5 compatible version

LiveBlog: 2 Years Later, Loving and Hating the Cloud

2009-06-23T13:16:21-07:00