Architectonic

Three Books

2011-04-07T23:51:00.001-07:00

Too long for Twitter: Three books the people who actually build high tech products should read:

Crossing the Chasm by Geoffrey Moore. Just read the whole thing cover-to-cover. Sure, it talks about ways to move from early to mainstream markets, but it also explains just what a market is, the difference between a cool technology and a "Whole Product" that's useful enough for somebody to actually pay for, and how to care for, feed and eventualy abandon early adopters. Moore didn't invent all (or even most of) the ideas in the book, but he took a bunch of esoteric information and turned it into a coherent, easy-to-read strategy for taking technology products to market.

The Four Steps to the Epiphany by Steven Gary Blank. Unlike Crossing, this one's a bit of a slog, but if you're selling to businesses rather than consumers resist the temptation to skim it or read one of the "lite" versions floating around. Blank takes a bunch of industry best practices, mixes in his own hard-won wisdom, and produces a plan for turning an idea into a workable business. Spend a day or two reading through Crossing to get some background, then spend as long as it takes to get through Four Steps. It's worth it.

Solution Selling by Michael T. Bosworth. There's "New Solution Selling" by Eades and "Spin Selling" by Rackham, but the takeaway should be the same no matter which you choose: selling is not a black art that only born salesmen can understand. Selling is a rational processing of matching customer needs to your solutions, and you (yes you, geek engineer) are fully capable of doing it. The key is that we're not talking hucksters selling vacuum cleaners door-to-door, we're talking selling expensive, complex, high-tech solutions in an ongoing relationship. And that means actually solving your customer's problem is important. And that firmly aligns the selling with the engineering. A good engineer is still going to need to put in some work to become a competent high-tech salesperson, but engineers have a huge advantage over non-engineers in this arena.

Four Steps has a great appendix with many more recommendations, but those three are the must-reads I'd want anyone I was working with (CEO to engineering intern) to have read and understood.

Dallas Big Data Micro Hackathon

2011-03-05T11:45:00.000-08:00

Tuesday, March 22nd at Cohabitat in Dallas, Texas a group of geeks is getting together to write code to process Big Data. There will likely be a group getting Hadoop up and running on laptops ("Hello, World" level), and possibly a group working with Cassandra and MapReduce on EC2 ("I actually have a real-life problem to solve" level) but anyone who's serious about Big Data is invited. You can sign up at:

http://dbdmh.eventbrite.com

"Big Data" isn't a formally defined term, but the general idea is that if your data set could fit on a single disk or be easily hosted inside a normal relational database it's not big data. Google calculating PageRank for every site they index? LinkedIn doing a complete social graph analysis for every user? Those are definitely Big Data problems. Technologies like MapReduce (see Hadoop) and certain kinds of highly scalable NoSQL databases (see Cassandra) fall under the heading, but it's a big tent and there are many other possibilities.

The event is free, but the idea is to keep the group small and focused, so spaces are limited. Sign up at: http://dbdmh.eventbrite.com

Frustrated with Dallas

2010-05-04T21:01:00.001-07:00

Union Square Ventures, the New York-based venture capital firm famous for its Web 2.0 investments (Foursquare, Meetup, tumblr, Twitter, delicious, Feedburner, etc) recently announced that they're hiring an investment analyst and a general manager. They invited anybody who was interested to send them links to their web presence, and the staff at USV would use that instead of a resume. Kinda cool.

In a followup post, they mapped the applicants. Nobody from Dallas applied for the general manager position, and only a couple could stir themselves to apply for the analyst spot.

Anybody who wanted to could apply. No old-boy network, no experience absolutely required (although they did have some "ideal candidate" desiderata). And Dallas just couldn't be bothered.

I'm very frustrated with Dallas.

Linked Data, Confidence Games and the Transitivity of Trust

2010-02-14T16:00:00.000-08:00

Over the Christmas holidays I took my family on a five thousand mile roadtrip around the American West. It took a couple of weeks and I expected to spend a lot of time on my favorite user-generated travel review site.

And I did spend a lot of time on the site, enough to eventually figure out that it had been comprehensively infiltrated by review spammers. Some of the spam reviews were obvious: "I loved this place! Five stars!" when all the rest of the reviews were negative. Some were more devious: "There were bedbugs! They spat in my soup! Zero stars!" when all the other reviews were stellar. In other cases it was much harder to tell, and in all cases the average rating was highly suspect.

Turns out there are companies that specialize in vandalizing review sites[1]. The companies employ actual humans who spend actual creative effort to craft misleading reviews. They even set up realistic user profiles, and on some sites they add each other as friends. In other words, it's considered worthwhile to spend real time and effort on this stuff.

It's been suggested that there's a technological solution: If the reviewers are part of a social network, it's possible to extract some useful statistics that might help determine if the user is real or fake.

If the reviewer is a friend, that's obviously useful information. But there's very little chance that some random reviewer is your friend.

But what if the reviewer is part of your extended social network? Surely the fact that somebody is a friend of a friend is some indication that they're trustworthy, or at least that they're a real person.

Nope.

First off, with a fan-out of 200 friends the 2nd-level extended social graph is around 40,000 people. Allowing for annoying people who friend everybody, an extended social graph could easily include a substantial portion of the entire population of the planet. All it takes is a couple of mistaken friend-adds to get you hooked up to a spammer-created sub-network. Even if you're careful, it's overwhelmingly likely that some friend-of-a-friend isn't.

So, trust is clearly not transitive and the idea of a "web of trust" cannot be taken literally[2].

In most cases, it's only possible to determine if the "shape" of the reviewer's social graph is reasonable. That is, are they friends with other plausible-looking people? Are many of their friends known fake profiles? Do they have a realistic number of friends? Etc.

But that's trivial to game. Even if there are obstacles to a totally automated approach, the application of ultra-cheap human labor makes it easy to set up a fake social network on any given site.

Linked data and distributed social graphs (ala FOAF + SSL) make things worse, because while before it took some amount of human effort to solve the captchas and create new accounts on a social-graph silo like Facebook, with a distributed "web of trust" approach it can all be completely automated.

That isn't to say FOAF + SSL isn't a neat replacement for the monstrosity that OpenID has become, but the "web of trust" part won't fly.

That said, in some sense it doesn't really matter. I'm certainly not arguing that we should slow in our rush towards a semantic web. The benefits are too great. But given the experience with email spammers and review fraudsters, it might be a good idea to be open about the fact that we're also introducing new hazards.

[1] So, honestly, I only have anecdotal evidence. But it doesn't seem like a very controversial assumption.

[2] "Trust" is a complicated word. It's not that knowing a review is by a friend-of-a-friend-of-a-friend isn't useful information, it's that using it to make a binary yes/no trust decision is misguided. There's been some interesting academic research in this area, Wikipedia has a rundown: http://en.wikipedia.org/wiki/Web_of_trust In what seems like a perfectly sensible approach, this paper: http://www.mindswap.org/papers/Trust.pdf suggests using social graph information as just one input into a full spam handling system.

303 Madness and the Giant Global Graph

2009-07-10T20:31:00.001-07:00

I had the opportunity to do a short talk at the latest Semantic Web Dallas meetup. I decided on an overview of the 303-redirect dance that differentiates a URI that points to a web page from a URI that names a concept in the Semantic Web. Yes, there's a difference. Yes, it's an important difference. Probably. In any case, it's a good topic for a 10-minute talk because having to listen to stuff like this for more than ten minutes at a time can lead to bleeding from the ears. It's a complex issue with an, uhh, unexpected? solution, best approached with a sense of humor. Well, maybe not best approached that way, but it seemed like a good idea at the time. And the list of references at the end is pretty good.

If you read the references you'll learn that you can also use URLs with fragment identifiers in your RDF. But doing it that way doesn't involve a fundamental redefinition of part of HTTP so it's a lot less entertaining.

Popstat on Google App Engine

2009-07-02T23:05:00.000-07:00

Popstat is the demo application from my Facebook Dev Garage Dallas presentation. It just posts a status message to Facebook and Twitter to demonstrate using both Facebook Connect and an external service. I developed it on my laptop and didn't have time to move it to a public host before the event. I wanted it out there live someplace and figured it was a good opportunity to try out Google App Engine's Java support (Popstat uses Grails with a mix of Groovy and Java)

I got it all working, but it was a pain.

I used the Grails AppEngine plugin. I liked it.
App Engine provides storage, but not in the form of a relational database. It's close enough that JPA and JDO both work (but not Hibernate, yet). I chose JPA, but either way you'll need to annotate your domain classes (I expected the GORM-JPA plugin to do that for me, but it didn't)
You'll need to put your domain classes into named packages. Things (silently) don't go well if you leave them in the default package.
If you're using JPA, domain classes will need to explicitly declare an id field. Make it a Long, and add the @Id and @GeneratedValue annotations. Use GenerationType.IDENTITY.
I was able to use the dynamic save() method provided by GORM-JPA, but I had to wrap up the calls in a withTransaction block, and the semantics are slightly different (use merge() instead of save() for updates)
Depending on your version of Spring, you may get a message along the lines of "org.springframework. context.annotation. internalPersistenceAnnotationProcessor': Initialization of bean failed" with something about "java.lang.NoClassDefFoundError: javax/naming/NamingException". The fix here worked for me.
Popstat uses the facebook-java-api library. Since App Engine forbids the use of JAXB, I had to switch to the JSON version of the client to avoid an error about JAXBContext.
To talk to Twitter, Popstat uses the oauth-signpost library. But Signpost depends on Apache HttpClient, and HttpClient uses low-level Socket calls forbidden by App Engine. I hacked Signpost to use URLConnection, but I wouldn't recommend that approach. If I had to do it again, I'd look around for an OAuth library that worked out of the box.
By default, the App Engine Java Development Server (a version of the App Engine environment you can run on your local machine) binds to localhost only. The command line client has a "--address" option, but the "grails app-engine run" command doesn't. I hacked the scripts/AppEngine.groovy plugin and harcoded the address parameter into startDevServer().

There was some other stuff that I didn't take notes on, but (other than registration being turned off) Popstat is doing what it did before.

Overall, though, it wasn't a great experience. Google turns off random bits of Java (for security and ease of management), which means that very few third-party libraries are going to work. You'll probably have to do some porting of your own code as well. That, combined with the admin service being down all morning, left a bad taste. The free hosting thing is great for demo apps but I think I'll stick to something like Amazon EC2 for real work. I'm very curious to see how Microsoft Azure stacks up (it's much more of a direct competitor to App Engine than the roll-it-all-yourself EC2)

The Semantic Web or The Generic at War with the Specific

2009-06-30T20:37:00.001-07:00

It's easy to imagine an application that takes advantage of Linked Data by extracting just what it needs and dumping it into a local relational database. But that's clearly cheating. It's equally easy to imagine a completely generic low-level Linked Data browser, but there's something less than completely satisfying about that, too. The basic problem is that a rich user experience requires specifics, while taking full advantage of the "anyone can say anything about anything" nature of the semantic web means that applications must be able to handle almost totally generic data[1]. At least that was the theme of my presentation to the Dallas chapter of the IxDA earlier tonight...

I'm especially proud of the way I failed to force people to sit through a detailed explanation of graph structures, subject-predicate-object triples, the use of URIs as identifiers, or any of the other traditional cruft that obscures the capabilities of semantic web technology under a morass of unnecessary detail. (Imagine introducing relational databases by first forcing people to understand index paging mechanisms, or learning to cook via an explanation of organic chemistry). The audience seemed to appreciate it.

[1] I struggled with this earlier over in /2009/03/linked-data-end-user-applications.html

[2] The translation from Keynote to Powerpoint to Google docs was not without problems. And you will definitely need to click through and get a larger version to read some of the screens.

Facebook, Intrigue, Betrayal, Murder

2009-06-28T22:03:00.001-07:00

A working understanding of authentication and authorization protocols is key to making use of modern web APIs. But protocols like the three-party delegated authc/authz[1] typical of modern web services can be difficult to follow. Role-playing protocol participants[2] is a fun way to make a very abstract process concrete, so I decided to write, produce and direct some geek theater at my recent Facebook Developer Garage Dallas presentation. When you get to the script pages, imagine Alice played by about the least feminine guy you've ever seen and you'll have the right atmosphere (you might need to click through and view the presentation full-sized to read the text on some pages)

I finished up with a quick review of some very traditional distributed programming topics. The questions "just how many test cases would you need to cover the possible states your program can be in?" and "what makes you think you can test these modules independently?" get people thinking along the right lines.

Oh. In the end Alice runs off with Bob and all of Dave's money, leaving him on the hook with the Mafia for four guns and several bribes. Such is life in the high-stakes world of distributed programming.

[1] Authc = authentication, or identifying a user, and authz = authorization, or determining what services a user is allowed to make use of once they're identified. Authentication says who you are, authorization says what you can do. In the presentation I talk specifically about delegated authc/authz, and ignore the more traditional single-process examples. People seem surprised to learn that OAuth, which is an authorization protocol, doesn't necessarily tell your application the userid of the user (although many implementations include the info along with the authorization tokens that are the primary purpose of the protocol) It doesn't help that the OAuth spec confuses the two.

[2] So, admittedly, the examples aren't usually acted out in front of an audience, but the role-playing does have a long and honored history. The script actually simplifies the real protocol considerably, but it should give the correct flavor: http://www.networkworld.com/news/2005/020705widernetaliceandbob.html

Talis Connected Commons

2009-03-29T14:41:00.001-07:00

I've got a Talis Platform developer instance that I've been using to host a tiny subset of the information from the National Register of Historic Places database. I spent some time this weekend at VoCampAustin trying to get the rest ontologized, and I was wondering what I was going to do when it came time to host the entire data set. Luckily for me Talis just announced the Talis Connected Commons:

...if you own, or are creating, a public domain dataset then you can store that data in the Platform as RDF, for free. We’re setting an initial cap of 50 million triples on each dataset, but thats should be plenty of space in which to collect some really interesting data.

While Amazon will "host" your public data set, that just means they provide some free disk space you can use from within your EC2 instance: you've still got to pay for bandwidth. With the Connected Commons, Talis takes care of those pesky bandwidth charges. I'm sure there are "within reason" disclaimers in there somewhere, but for anyone considering RDF-izing and publishing public data sets, the Talis offer is definitely worth checking out.

Facebook Redirects : The URL is not my son

2009-03-25T12:27:00.000-07:00

If you didn't get here via a very specific Google search[1] you probably want to stop reading now, otherwise... Those who eschew the standard Facebook PHP libraries are rewarded with a fuller, richer understanding of the intricacies of the API. I.e. they have to deal with obscure crud that normal people get to ignore. This particular cruddy bit recently managed to eat more time than it had any right to consume:

The URL http://www.new.facebook.com/login.php?v=1.0&canvas&next=http%3A%2F%2Fapps.facebook.com
%2Fhistoryradar%2F&api_key=643f14fgg7294eff is not valid.

That error message was in response to a redirect to force authorization if there was no session information on a canvas page request. I knew that the URL was in fact valid since not only was it straight out of the Facebook API documentation, I could cut and paste it into a browser and get the correct response. At first I thought it was an encoding issue (the "next" value needs to be encoded since it's a URL inside a parameter) but it took digging into the standard PHP library to find the real problem:

public function redirect($url) {
if ($this->in_fb_canvas()) {
echo '<fb:redirect url="' . $url . '">';
  } else if (preg_match('/^https?:\/\/([^\/]*\.)?facebook\.com(:\d+)?/i', $url)) {
// make sure facebook.com url's load in the full frame so that we don't
// get a frame within a frame.
echo "<script type="\"text/javascript\"">\ntop.location.href = \"$url\";\n";
} else {
header('Location: ' . $url);
}
exit;
}

Right. You don't need a "real" HTTP 302 redirect, you need to send a special snippet of FBML. It's easy to forget that Facebook is its own little Bizarro World until you step just a little outside the approved way of doing things.

[1] No, not "the url is not my son." I mean one with "login.php" and "url" and "not valid" in it somewhere.

Linked Data: End-User Applications?

2009-03-11T14:13:00.000-07:00

Linked Data is a common-sense set of rules on how to use big-S Semantic Web technology to publish data on the Web. More or less[1]:

Publish your data as RDF[2].
Use HTTP URIs to name resources in your RDF.
Make the URIs dereferencable and contain more RDF.
Use a standard schema language (probably OWL[3]).

It's also probably safe to assume that the RDF used in Linked Data is "informal" and that there's a lot of it[4].

Linked Data has been gaining momentum lately as sort of a down-to-earth version of the Semantic Web. The fact that it's relatively low-effort[5] means that there is quite of bit of data being published, despite the fact that very few applications outside the laboratory make use of it.

Linked Data has obvious uses inside the enterprise. The BBC has done some great work using web development techniques based on a combination of hard-core enterprise integration and brand-spanking-new semantic web technology. Linked Data behind (and just in front of) the firewall is an exciting area, and I believe that is where it will find its first widespread commercial acceptance. But that's another blog post.

This post is about is real end-user applications that take advantage of Linked Data. By "real" I mean something that sits on your desktop (or in your phone, or maybe even web browser) with a rich user interface tailored for a particular task. I don't mean generic browsers or infrastructure components.

Since I had no clue what such an app would look like, I decided to take a generate-and-test approach: list the defining properties of Linked Data, come up with some consequences of those properties, then take the Cartesian product. Somewhere in there there must be a pony or two.

After some trial and error, I picked the following:

Is the application meant to handle more than a fixed set of data? For example, a browser like Marbles[5] is meant to navigate across all possible Linked Data, while a other systems limit themselves to, say, social network graphs as expressed by FOAF, or even a fixed set of in-house data sources.
Is the linking visible to the user? For example, in a generic browser the user sees the links and chooses which ones to navigate across. The links are central. On the other hand, a social network browser might automatically choose how to spider a FOAF network, and present the user with a summarized view containing data from many sources.
Is reasoning important? That is, is the raw data presented to the user, or will new triples be generated (or filtered) using formal (or informal) reasoning?

For reasoning, it's pretty much yes or no:

Synthesizes new triples? { Yes, No }

The same goes for linking: either the program makes the underlying low-level links visible and primary, or it covers them up somehow:

Navigation? { User-visible Links, Invisible Links }

The extensibility question turns out to be a little more complicated than a simple Yes/No. While there are clearly some programs that are totally generic and others that are totally fixed, there are interesting cases in the middle. I added another possibility: applications that try to do something with data they don't totally understand (maybe by understanding ontology fragments like parts of FOAF or geodata)

Extensible? { Yes, Somewhat, No }

There's an interesting tradeoff, where the more specific the knowledge an application has about the data it works with, the better crafted the user experience can be.

I am not happy with the list above. It leaves out some important characteristics, conflates others and is generally unsatisfactory. I originally had closer to a dozen characteristics, but the resulting combinatorial explosion made things awkward. But the list is just for inspiration, and so I'm willing to live with it.

And the results:

Linked Data applications like browsers and analysis tools show up with their own categories. That's not surprising since that's how the categories were chosen.

I especially like the look of the "hybrid" applications: the ones that combine hardcoded knowledge of the data with the ability to process new data discovered through following links. If there's a pony to be found, I suspect it's in one of those rows.

I'm currently working on an iPhone application called "National Register Radar" that uses geolocation and a Linked Data version of the U.S. National Register of Historic Places database to help users maintain "situational awareness" of the history of the places around them. Right now it would be a first-row application: it has hardcoded knowledge of specific kinds of Linked Data, it hides the low-level linking and provides a summary view, and it presents the data as it finds it, with no logical reasoning.

Although it's a relatively immature use of Linked Data, hardcoding makes developing the initial version of the application much easier. It means that few external libraries are required (an important consideration on a mobile device wiht an non-x86 processor and no Java). It also means that traditional application development techniques apply: I don't need to mess around with an on-phone triple store and SPARQL queries.

I think, though, that I'd like to turn it into a row 6 "enhanced special-purpose browser" that's not restricted to just a few hardcoded data sets. It's unclear how all the technology would fit together (how can the application make use of DBpedia data without hardcoding? Can Fresnel fit on a phone? Can Fresnel be adapted for voice output?) but it's worth a try (and probably a follow-up blog post).

Ultimately, I suspect Linked Data will be more at home deep inside enterprise infrastructure than at the end-user level, but I'm going to give National Register Radar another couple iteration and see where it ends up.

I'll be demoing an alpha version at BarCampAustin4, if you're interested in Linked Data (or speech synthesis) on the iPhone and are going to be in Austin, ping me and maybe we can get a BOF together...

[1] The "official" list is here: http://www.w3.org/DesignIssues/LinkedData.html, or even better, here: http://www4.wiwiss.fu-berlin.de/bizer/pub/LinkedDataTutorial/

[2] RDF is the data model for the semantic web. Think of it as an ultra-hyper-normalized database where everything has been reduced to three columns: <subject> <predicate> <object>. Database keys are in the form or URIs. Don't let incredibly over-elaborate explanations fool you: it really is that simple.

[3] OWL is a popular schema language for RDF. Since reducing everything to triples removes pretty much all the type data, you need a schema language to add it back in. It's a little like a much more powerful version of XML Schema that lets you bake in the kind of semantics that would normally go into the comments. The documentation for it is uniformly awful.

[4] There's this really great article on just this topic that I can't seem to find a reference to right now. I'll edit it in later. Hey, it's a blog post, not a journal article, whadaya expect?

[5] It's relatively low effort because some pioneers have put in tons of effort developing some slick tools to help out.

[6] If it's up and running. Few of the commonly referenced Linked Data browsers worked reliably for me.

Speech Synthesis on the iPhone with Flite

2009-02-22T08:30:00.000-08:00

I was planning to use the Cocoa NSSpeechSynthesizer class to do text-to-speech (TTS) in a Linked Data based iPhone application, but Apple evidently isn't interested in my pitiful wants: the very slick OS X speech synthesis capabilities don't appear to be supported in iPhone OS. Not to be deterred, I managed to get the Flite speech synthesizer up and running on the phone. It had already been ported (multiple times) to the ARM architecture[1], so all I had to do was import the source files into Xcode.

Well, pretty much. Other than the tedium of importing all the files, there were a couple of things that needed figuring out:

I added the User-Defined Setting CST_AUDIO_NONE to turn off Flite's ability to play audio directly[2]. Alternatively, you could probably add a new implementation of the sound playing functions that used the iPhone audio framework, but that's too much work for a quick hack.

Since I just wanted a simple proof-of-concept I called the main routine[3] of the command-line "flite" utility directly from my application code:

int play_message( char* msg_file, char* msg ) {
 char* argv[5];
 argv[0] = "flite";
 argv[1] = "-t";
 argv[2] = msg;
 argv[3] = "-o";
 argv[4] = msg_file;
 return flite_main( 5, argv );
}

I was torn over whether to set up an in-memory buffer for the sound data or write it to a file then play the file back immediately. Since flite_main() was already set up to write to a file I took the easy way out. The only mildly tricky thing was finding someplace where I could write the temporary file. Some cut and paste from the SpeakHere[4] sample application, and:

NSArray *filePaths = NSSearchPathForDirectoriesInDomains (
 NSDocumentDirectory,
 NSUserDomainMask,
 YES);
NSString *recordingDirectory = [filePaths objectAtIndex: 0];    
NSString *tempFilePath = [NSString
 stringWithFormat: @"%@/%s",
 recordingDirectory, "recording.wav"];
char *path = [ tempFilePath UTF8String]; 
play_message( path,
 "mister watson, come here, i need you." );

That got the sound file written out. To play it back I pointed AVAudioPlayer[5] at the newly generated sound file:

NSError *err;
AVAudioPlayer* audioPlayer =  [[AVAudioPlayer alloc]
 initWithContentsOfURL:[NSURL fileURLWithPath:tempFilePath]
 error:&err];
[audioPlayer setDelegate:self];
[audioPlayer prepareToPlay];
BOOL plays = [audioPlayer play];

I couldn't get AVAudioPlayer working without manually copying over the AVFoundation framework from /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS2.2.1.sdk/System/Library/Frameworks/AVFoundation.framework. I'm not sure that's the right thing to do, but it worked, and my iPhone now says "mister watson, come here, i need you" with an old-school Cylon accent.

[1] And to at least one framework useful on jailbroken phones. This thread seemed useful if you want to go that direction. I suspect other people must have gotten Flite working in Xcode as above, but I wasn't seeing anything on the first dozen pages of search results.
[2] CST_AUDIO_NONE is a standard Flite build flag. In Xcode, defining CST_AUDIO_NONE as a User-Defined Setting for a Target adds -DCST_AUDIO_NONE to the command line of the compiler. I prefer a good textual Makefile to the annoying graphical interface, but you take the bad with the good.
[3] After re-naming it to "flite_main()" to avoid a clash with the main routine of the iPhone application. There were actually a bunch of little fiddly bits along those lines, but I didn't document them as I went along and you'll have to figure them out on your own.
[4] SpeakHere is at http://developer.apple.com/iphone/library/samplecode/SpeakHere/index.html, but you probably need to be logged in to the iPhone Dev Center to see it.
[5] Again with heavy cut and paste from the SpeakHere sample code.

OpenSocial: Behind the Corporate Firewall

2009-02-03T17:29:00.000-08:00

OpenSocial is an API for consumer social network sites like LinkedIn, Orkut and Hi5, but its greatest value may come from use behind the corporate firewall.

OpenSocial is really two separate things: a portal framework that describes how embedded content can interact with a page, and a glued-on social network API that provides a way to access things like profile and friend data.

The portal framework is just Google's iGoogle framework rebranded, and the social-network API is a least-common denominator mashup of the APIs from a selection of big commercial social networks combined a dash of influence from the Facebook API.

Both the portal framework and the social API are ok, but neither one is really best-of-breed. There are many widely deployed, battle tested portal APIs available (JSR 168 and Sharepoint spring to mind), and on the social side Facebook's API is (at least for the moment) superior.

But in this case, I'd argue worse is better.

The OpenSocial gadget API was designed to be used by web developers rather than corporate IT drones. Authoring a simple OpenSocial gadget is no harder than writing a web page, and the technology is nearly identical. It is much faster to get started with OpenSocial than to learn to program for a traditional corporate portal[1].

The social API is important, too. ERP systems (to optimize the use of your company's stuff) and CRM systems (to optimize the use of your company's customers) are important, but most companies claim their people are their most important resource[2]. Anybody who's spent time on Facebook or Twitter probably buys that there are advantages to including social network features in enterprise applications[3]. OpenSocial, as a well-documented, free-to-implement standard is an obvious choice.

I've been spouting some of this stuff whenever I got the chance, so it was encouraging to see Atlassian (darling of the developer tools world) say something similar at their annual conference[4].

I have a selfish reason for wanting OpenSocial behind the corporate firewall: I'd like Praxis Bridge to have a way to keep in touch with students after the course is over, and I'd like the students to have a way to keep in touch with each other. A silo'ed social network for a limited-time event is useful, but a way to hook into the user's everyday working social network is much more valuable. And since the OpenSocial architecture allows container administrators control over the information leaked to external gadgets, I think it would have a shot at getting past (justifiably) paranoid corporate gatekeepers. But that's a whole 'nother blog post.

[1] Or, well, it seems that way, which is the same. Isn't the whole "worse is better" thing annoying?

[2] On a recruiting brochure you can probably assume a line like "people are our most important asset" is just the standard BS, but when a company's 10-Q says it you can assume they're serious.

[3] ERP systems have human resources modules, but that's not quite the same thing as being "social network enabled". On the other hand, the ERP vendors are going to start buying up "behind the firewall social network" vendors just as soon as the downturn ends, and at that point OpenSocial becomes just another standard ERP module.

[4] OpenSocial comes in in the keynote at about 9:45. There's a whole session about their new plugin architecture at http://blogs.atlassian.com/news/2009/01/atlascamp_video_1.html. Thanks to Tracy Snell for the pointer.

CoHabitat in 90 Seconds

2009-01-23T20:55:00.000-08:00

Quick video from today's Jelly at CoHabitat. I'm posting it as much to play around with Vimeo embedding as anything else, but it definitely gives the flavor of the place:

OSDGD: Introduction to OpenSocial Presentation

2009-01-17T15:05:00.000-08:00

OpenSocial Dev Garage Dallas is over, the mess has been cleaned up, and pretty much everyone has gone home. The pizza Traxo sponsored was delicious, the coffee Praxis Bridge provided helped get people going, and the CoHabitat venue rocked. There were a couple new faces, and I'm pretty sure we've added to the group of regulars who show up at this kind of stuff in Dallas.

For reference, here's a copy of the presentation slides from the morning session:

The idea was to get people up and running with an OpenSocial application as quickly as possible, so I skipped over the normally obligatory history, motivation and background in favor of getting to the good stuff right away. I think it worked out well, although I suspect that for a couple of people in the audience the information was too basic. It's always hard to decide on a target level, and I'm happy enough overall. I think getting a more detailed agenda with a presentation outline posted in advance would have helped people judge better.

OpenSocial Dev Garage Dallas

2008-12-26T12:02:00.000-08:00

=== UPDATE ===

We've got a page on the opensocial.org wiki. This page is the canonical reference for all things OpenSocial Dev Garage Dallas (except the signups, which are still on Upcoming)

If you're developing Facebook apps, it's inevitable that you'll eventually also be developing for the "everybody but Facebook" OpenSocial API. Over the past couple of months, the API has stabilized substantially, and it's probably about time to check it out.

OpenSocial Dev Garage Dallas is your chance to write your first "Hello Friends" application in a supportive group environment. If you're more into writing containers than the gadgets that populate them, we're also planning to have a breakout session on implementing the API so your application can be an OpenSocial gadget host.

The event is planned for Sat, January 17th at the amazing new Cohabitat coworking space in Dallas's historic State-Thomas neighborhood. In the interest of keeping things focused we're capping attendance at 25 people:

http://upcoming.yahoo.com/event/1460416/

If there's enough demand, we may be able to open up some more space, but I'd recommend signing up now if you want to guarantee a spot!

See you there!

Google Friend Connect

2008-12-20T18:53:00.001-08:00

I'm experimenting with Google Friend Connect as you might notice from the new gadget over on the right. Feel free to join up, but as it's just an experiment there's a good chance I'll be removing the feature in a bit. I'm thinking that with the addition of a couple of OpenSocial event management gadgets a Blogger blog might be a good, non-Facebook alternative for organizing an event, but we'll see.

OAuth: Not a Silver Bullet

2008-11-23T12:29:00.001-08:00

OAuth[1] provides a standard protocol for delegated authentication. That is, it lets a site like Twitterank[2] get controlled access to your twitter account without your having to give Twitterank your username and password. Instead, there's a multi-step authentication process involving a "token" that grants temporary, limited access to a third party service.

This sort of authentication system has been in use for many years in commercial and academic systems. OAuth just brings a cut-down version to the web 2.0 world.

You may ask yourself: If it's so great and it's been around forever, why doesn't everybody use it? The short answer is that it's a major pain in the rear-end both for developers and users, and most of the time it's just not worth the bother.

As a developer I actually like OAuth. I think it's wonderful for the times when the added complexity of delegated authentication makes sense. But the complexity is not without cost. For trivial, single-purpose services it is, at best, a very small win, and possibly even a net loss.

Ignoring the implementation difficulties for now, let's take a quick look at the user experience via Twitter and the (hypothetical) Twitterevil service. Twitterevil operates by promising to calculate an amusing ranking number for your tweet history. It does in fact do that, but it also keeps your username and password and uses them to posts spam tweets. We'll look at scenarios both with and without OAuth.

The first scenario, sans-OAuth, goes someething like this:

I surf to twitterevil.com and enter my Twitter username and password.
Twitterevil logs into Twitter as me, grabs some data, and uses it to calculate some statistics, which it tweets under my name. It also (contrary to what it says on the Twitterevil home page) keeps my username and password.
Later, Twitterevil starts using my Twitter account to post spam links.
I notice the spam links and use Twitter's password recovery feature to change my password.
Twitterevil can no longer use my account.

The OAuth scenario goes like this:

I surf to Twitterevil. Twitterevil has a login button button that redirects me to a special Twitter OAuth login page.
I enter my username and password on the Twitter OAuth login page, and hit the "OK, allow Twitterevil access" button.
I get redirected back to Twitterevil, which calculates my score, tweets it, and then starts posting spam links.
I notice the spam links, log into my twitter account, go to my OAuth delegated permissions page, find the twitterevil.com entry, and remove Twitterevil's permissions.
Twitterevil can no longer use my account.

So, conservatively, the OAuth scenario has more steps and involves the new concept of "delegated authentication". This concept existed implicitly before but is now (very) visible to the user in the form of a multi-step login process and a new "OAuth delegated credentials" page. All to get the same end result.

Actually, as some of you may have noticed, I've lied by omission. There's an alternate OAuth scenario that I'm leaving out. It's a change to step 2, and it goes like this:

...
I enter my username and password on the Twitter OAuth login page, and then read through a paragraph of explanatory text about just what Twitterevil is asking permission to do: "twitterevil.com is asking permission to [Read through your posting history] and [Post tweets on your behalf]. Do you want to give twitterevil.com permission to do this (a) Once (b) For the next little while, or (c) Permanently. Note that you can always go back and revoke permission later."

No, I'm not kidding.

Anybody who's looked at the problem of user security knows that nobody, even security experts, ever reads that crap. They just hit the default button automatically. So 2b is "cover your behind" security theater, it provides no real security in the vast majority of cases.

Flickr, which uses a pre-oauth delegated authentication scheme, does it like 2a. They don't expose out the permissions set during authentication, instead allowing the user to go in later and see just what they approved (if users ever cared to look, which they don't). I suspect this is the result of some user experience testing. It's certainly easier.

A service like Twitter has a very limited number of possible actions, so the model presented in 2b is unusually simple. A more complex application could have dozens of possible permissions. Developers who care about the experience will probably layer a "user level" set of permissions on top of the real underlying implementation permissions, but most won't. And that's just nasty.

So, depending on how exactly the service implements the OAuth user experience, the number of user actions is either slightly or significantly more complicated for the OAuth scenario. The conceptual complexity (how many different ideas the user must keep in mind) is a clear loss for OAuth, especially if the service exposes out it's internal security architecture in the OAuth security screen.

But.

I was just asked by a service I won't name to hand over my Google username and password so that it could access my Google Reader data[3]. And I said "no", because my Google account is not trivial or simple and there are real consequences to handing it over to somebody I don't fully trust.

OAuth is a sledgehammer, and it does not make sense to use a sledgehammer when you can get away with something less. Developers should use their very limited time and money to go simplify their design, not add in significant new complexity where it's not needed.[4]

But if, after serious reflection, you decided you do need a sledgehammer, OAuth appears to be shaping up into a good choice.

[1] http://oauth.net and also http://en.wikipedia.org/wiki/OAuth (please help with the lame Wikipedia entry) I've always assumed it stood for Open Authentication, but I don't see that written down anywhere.

[2] Twitterank is a fun example because Brian Oberkirch tweeted, as a joke with a point, that he set it up as an evil password collector. But it got picked up by and republished as a non-joke by ZDNet. I love ZDNet's lame attempt to substitute boilerplate "rumor has it" for fact checking.

[3] Google is actually moving towards OAuth for all its services, but Reader doesn't have an "official" API yet, and OAuth doesn't work. Developers must use a sort of backdoor ClientLogin hack.

[4] Note to complainers: There is some accidental complexity in OAuth that can be overcome via better libraries and documentation, but the base problem is the inherent complexity of exposing a more complex authentication scheme, and that exposure is at the heart of the claimed benefit of OAuth.

I built this great product, now what?

2008-10-15T16:04:00.001-07:00

I've found that people who write great software generally do it because they can't help it. It's more fun than a barrel of monkeys and they'd do it even if they weren't getting paid. Nobody likes the final grinding detail steps of productization, but with a good team the development phase is generally a hoot.[1]

But "build it and they will come" isn't really a business plan. At some point most companies will need to talk convincingly about their great idea, either to get funding or make a sale. Do you pitch an investor with a demo, or a slide deck? How do you talk to the press? Is something like Demo or TechCrunch 50 worthwhile?

If you've been haunted by questions like the above, then show up this Saturday October 18th for a few hours and find out all the answers:

http://www.texasstartupblog.com/2008/10/15/pitch-camp-dallas-this-saturday-10am-4pm/

[1] In the same way a dog sled expedition to the South Pole is a hoot.

{s,S}emantic Web Workshop

2008-08-10T11:13:00.001-07:00

Big-S, little-s, whichever direction you're coming from, parts of the semantic web are here today. It's not the "Grand Vision"[1] of Tim Berners-Lee, but don't let that stop you from taking advantage of some very cool technology that can enhance your site or enable your application right away. The slide deck below is from my presentation at BarCampHouston3. The idea was a quick intro then a workshop where attendees had the chance to actually experiment with the technology hands-on right then and there. No projector and no network access in the presentation room almost scuttled that, but we eventually borrowed the necessary infrastructure (thanks Giovanni, Michael and Stormy!) and got to the presentation part, at least:

[1] http://www.w3.org/DesignIssues/Semantic.html In post-singularity world, the Semantic Web understands you.

MySQL, PostgreSQL and BLOB Streaming

2008-07-20T10:16:00.000-07:00

You can store surprisingly large binary objects in your relational database[1], but even if your chosen system is perfectly OK with a 100 megabyte BLOB, you're going to have to handle all that data on its way to and from storage.

With a 2 MB image file you can get away with reading the data fully into memory on each access. With a 100 MB file it's a little trickier. Not only is that a pretty big byte array, you may end up with multiple copies of it (your application's copy, the driver's copy, the database's copy, etc)

The obvious answer is to stream the data. JDBC's BLOB interface has the convenient-looking "getBinaryStream" and "setBinaryStream" calls[2]. But if you actually try to use the calls you may get a nasty (and database-specific) surprise.

MySQL, for instance, implements getBinaryStream in their JDBC driver as[3]:

return new ByteArrayInputStream(getBinaryData());

You see the problem. There's no point in streaming the data when the driver is just going to read it all into memory anyway. There are some workarounds[4], but ultimately there are server-side limitations that you can only be avoided via a brand-new database engine[5].

PostgreSQL has a reputation for being more mature than MySQL, and that's the case with handling large binary objects. Of course, another part of PostgresSQL's reputation is a Perl-like "multiple ways to do everything" approach, and there are a couple of different ways to handle BLOBs.

There are two basic API approaches for this sort of thing: either pretend the huge bolus of binary gunk is just another column value stored along with all the other data for the row (whether it's really stored that way or not), or put a pointer ("locator") to the data in the relational row, and provide some other interface for actually working with the data.

PostgreSQL does both, both at the database level (it has two different low-level mechanisms[6]) and the API level (it has several different APIs that expose out the underlying mechanism in various ways). Annoyingly, you can sort of mix and match the APIs and underlying implementations.

The following discussion assumes the use of oid's (locators), not bytea's (in-row blobs, see the footnote above), which means at the database level we're going to expose out the locator (oid) in the column. In other words, if you do a "SELECT" on the column, it's going to give you a bunch of integer pointers, not a bunch of binary content.

CREATE TABLE resource_record
(
id serial NOT NULL,
content oid,
...
)

Luckily, PostgreSQL's JDBC driver will (mostly) allow you to treat the locator column like it really holds the BLOB directly:

 Blob content = ....
InputStream bodyStream = body.getBinaryStream();
...

Well, assuming you're inside a database transaction, which is generally the only time you can stream to or from a JDBC BLOB[8]

But it's not quite perfect. The lower-level Postgres calls[9] allow BLOBs to be shared, so deleting a row doesn't necessarily delete the corresponding blob. That conflicts with normal JDBC semantics that assume the the BLOB is part of the row and should be deleted when the row is deleted. Postgres' JDBC driver should probably implement those semantics, but it doesn't, so you have to run out and delete it yourself. One solution is to add the semantics back in via a trigger or rule:

CREATE OR REPLACE FUNCTION on_resource_delete() RETURNS trigger
AS $on_resource_delete$
BEGIN
  PERFORM lo_unlink( old.content );
  RETURN null;
END;
$on_resource_delete$
LANGUAGE plpgsql
IMMUTABLE
RETURNS NULL ON NULL INPUT

CREATE TRIGGER resource_delete_trigger
AFTER DELETE
ON resource_record
FOR EACH ROW EXECUTE PROCEDURE on_resource_delete()

If you're interested, the data is stored over in the system table "pg_largeobject"[10]

Since this is PostgreSQL, the discussion wouldn't be complete without a mention of vacuum. Vacuum is the PostgreSQL "garbage collector". If you don't run it, the disk space associated with deleted BLOBs won't be recovered. In production that may or may not matter, but during development it can a pain to lose the disk space.

On my OS X machine, the file associated with the pg_largeobject table turned out to be:

/Library/PostgresPlus/8.3/data/base/17456/2613

If you load up a couple of fairly large BLOBs, you can probably figure out what file is being used on your system. It's comforting to watch it grow and shrink.

So, although there are other options, it definitely works to:

Make the column an oid
Add the trigger if you want your BLOB deleted with its row
Use normal JDBC BLOB interface and stream away
Run a full vacuum occasionally if you need to recover disk space
Use a cache (like Squid) if you're serving the data back out over HTTP

[1] If you've chosen the right database, have some time to spend tuning, and are willing to install something like Squid if you're serving the data back out over HTTP, then single blobs of 100's of MB are not undoable. Whether it's good or evil is a different question.
[2] http://java.sun.com/javase/6/docs/api/java/sql/Blob.html
[3] mysql-connector-java-5.1.6/com/mysql/jdbc/Blob.java:108
[4] MySQL also has a (seriously crippled) locator-based version: http://dev.mysql.com/doc/refman/6.0/en/connector-j-reference-implementation-notes.html, see [5] for why it's problematic.
[5] http://www.blobstreaming.org/ Even if you fix the client side to not read the entire BLOB into memory, the server side still does, and insists and transferring the entire thing every time, even if the client only wants a tiny piece. Which is why you need the whole new engine.
[6] http://jdbc.postgresql.org/documentation/80/binary-data.html gives an overview of PostgreSQL BLOB support, and how there are a lot of options.
[8] Which is painful, because it means you may have to keep the transaction open a very long time if you're, say, streaming 80 MB of data in over http from a client on a slow network connection. That's the sort of thing that drives people to stream to a temporary file first, then from the temp file to the database.
[9] The lo_* routines, see http://www.postgresql.org/docs/8.3/interactive/largeobjects.html
[10] The data is stored as very many 2k chunks: http://www.postgresql.org/docs/7.4/interactive/catalog-pg-largeobject.html

Tessera at the Semantic Web Austin Launch Party

2008-06-16T21:07:00.000-07:00

Ever tried to write an application that sits on top of a fine mist of questionably accurate factoids spread in varyingly dense clumps across the whole Internet? It's a pain. The {s,S}emantic Web is a terrible platform for applications. Developers need real APIs.

I'll be at the Semantic Web Austin Launch Party (Tue, June 17th) talking about Tessera, an experiment in wrapping the distributed social web (XFN, FOAF, hCard, etc) with OpenSocial, the API everybody-but-Facebook is using to expose out their social networking platform.

Tessera is nowhere near complete, but it's far enough along to run simple OpenSocial gadgets, and I hope to have the OpenSocial version of Popcitation up and running by tonight. Or not. But there's certainly enough there to make for some interesting discussion. Hope to see you in Austin!

Some Experiments with the Future of Social Network Technology

2008-06-04T17:30:00.000-07:00

I gave a presentation at the recent Dallas Social Network Technology event (yes, we're still trying to come up with a good name. Pictures here.). We ended up with something just over twenty people, but it was exactly the crowd we were hoping for: developers, startup-istas, entrepreneurs and business people actually "doing something" with social network technology. A quick canvas of the attendees indicated that a get-together maybe once a quarter or so, with a meetup for beers slightly more often, might be about right. There was some talk from the general direction of Alamofire about a game-focused event sometime in the fall, and I suspect there will be another general event between now and then.

The presentation was combined with a live demonstration of the Popcitation Facebook application, and both a Google and Facebook login to the Popciation destination site. (All of it, but especially popcitation.com, is a development spike to test out new technology, there's a good chance it won't work quite right for you.) I got about 10% of an OpenSocial version done, I'll try to finish that up in the near future.

Closing thought: it would be cool to have an OpenSocial container with DISO as the backend, huh? Need to do some more research along those lines, for all I know that's the standard assumption...

Coding and Comments

2008-05-22T06:31:00.000-07:00

Some recent Twitter posts by Adam Keys got me thinking about the role of comments in code. Good and consistent coding style that reflects a solid development philosophy helps eliminate the need for many kinds of comments. Capturing requirements in a formal-ish way (through the use of techniques like Behavior and Domain Driven Design) eliminates even more. If important requirements can be captured formally in a "live"[1] way, they generally should be.

But (a) you're not always in a situation where formal-ish requirements capture is possible and (b) there is some information that can't be captured formally.

It's clear what needs to be done in the long term about (a), but in the mean time if you're going to have static text, it's good to have it as close to the code as possible. Comments let you do that.

It's not so clear what can be done about (b). Even if you've read the code and grok it 100%, there are situations ("Although it's irrational, as of Oct. 18 2005, the accounting department insists we use HTTP basic auth without SSL") in which the code is not enough. Anything to do with people and their motivations, really.

So, imagine yourself as a slightly-less-clever-than-you-think-you-are future reader of your own code. Imagine that you're generally familiar with the application, have read the code in some detail, and understand the development philosophy (hopefully) baked in throughout the system. If you're still thinking "WTF?!" then put in a comment. If it takes a full paragraph to explain the awful circumstances that led you write something impenetrable, then do it. Your future self will thank you.

[1] In post-Singularity world, code executes you. Until then, you can't really refactor, trace, test or formally analyze static natural-language text like comments or (shudder) external requirements docs. Natural language isn't "live".

Dallas Social Network Technology Event

2008-05-15T14:57:00.000-07:00

Based in or around Dallas? Serious about developing an application based on social network technology? Was Facebook Dev Garage too Facebook specific for you? We're planning a get-together in early June you might be interested in. It's looking like the evening of Tuesday, June 3rd starting at 7:00 and going till about 9:00 at the ritzy Sabre campus in scenic Southlake, TX:

Sabre Holdings
3150 Sabre Dr. Southlake, TX 76092
(Campus Directions: Gate , Parking , Entrance)

When you get to the security gate, tell the guard you're with the Social Network Technology event, and you're in like Flynn.

Wiki at:

http://socialdevthing.pbwiki.com

Facebook event page at:

http://www.facebook.com/event.php?eid=16975616668

Current agenda is a quick round of introductions at 7:00, some demos, some discussion, and a followup at a local tasty beverage establishment. Leave a message on the Facebook event wall if you're interested in demoing or have a particular topic you'd like to focus on.