esjewett.com

Statistical misunderstandings and Google Book Search

esjewett — Sun, 06 Sep 2009 22:56:42 +0000

Well, there I was, chatting away with (hopefully still) a friend about configuring a Twitter search widget for a blog and up comes the Google Books meta-data topic, in the form of a link to Google Books: A Metadata Train Wreck. This is the kind of article that really gets at me. Which is to say that it is an article holding a position that I mostly agree with in principle, but which does the position the disservice of making pretty questionable arguments.

So with that, I'll lay out two (or 5) things that really get my goat when people start talking about large data sets and meta-data.

The misunderstanding about the whole point

There seems to be a misunderstanding between the scholarly community and Google about what Google Books actually is. I think it is this misunderstanding that leads to claims like the one about "miscategorization" of translators as authors. It seems clear to me that the Google Books team made a conscious decision to put translators and authors into the same search field. As such, it's not an error so much as it is a semantic disagreement. I'm willing to bet that Google maintains author and translator metadata separately on the backend and just concatenates the fields for the purposes of searching.

In brainstorming, I came up with a few things Google Book Search and the whole library digitalization project might be, from Google's perspective:

A vehicle for advertising and referral revenue for Google.
A project to create a dataset that Google can use to train its translation and semantic knowledge engines.
A dumb idea that will never help Google and will eventually be abandoned.

It's possible that Google really conceives the project as at least partially having the goal of enabling scholarship, but I think #2 is probably the real goal, having watched how Google works for about a decade now. If I'm right about this, then Google Book Search is just a cover; a way of making the data gathered from libraries publicly available. If #1 is right, then having the author and translator in the same field is exactly the right thing to do, since lots of people will mistakenly search for the translator instead of the author.

Would it be good for Google to provide a scholarly interface to Google Books metadata that splits out translator and author? Yes. Does Google have a responsibility to provide this interface? I don't see how.

The misleading or useless appeals to statistics and data

The second point is perhaps the more important one: There is often a misunderstanding or ignorance of statistical methods and the type of weird stuff that happens within large data-sets. I see this occasionally when the scholarly/library community blogs about Google Books, but I see it in a lot of other places too. This (I assume) ignorance results (again, I assume) in a belief that appeal to common sense or personal experience is a valid argumentative technique when large data sets are in play. From experience with large data sets, I have a tendency to believe that common sense and personal experience are worse than useless when large datasets are involved. In any case, this belief results in two major problems and we see examples of both in the above-linked blog.

Statistical problem #1: Claims about the number of errors in a particular set of meta-data without any comparison to existing meta-data sets that might give us a baseline of the number of errors we may expect. For example, if there are 2 errors of type X per 1000 records in Google Books, that doesn't tell me anything unless I also know that there are 0.3 errors of type X per 1000 records in the UC library system. That tells me Google Books has some catching up to do. On the other hand, maybe there are 5 errors per 1000 in the UC library system, in which case Google Books is doing pretty good. The blog (and most of these sorts of blogs) fails to give any useful metric for me to compare against.

Statistical problem #2: The sample size is usually left out. If there are 300 errors of type X in Google Books and only 100 errors of type X in my local public library ... well that probably means that Google books is 100x better than my local public library because it's got a lot more books in it. But I can't tell because no one bothers to say how many books are in Google Books and how many are in my local public library.

In summary: A metric like "572 errors of type X" is useless because I don't know the sample size or have anything to compare to. "2 errors of type X per 1000 records in Google Books as opposed to 0.4 errors of type X per 1000 records in the Harvard University library system" on the other hand is incredibly useful as it provides a basis for comparison and understanding.

To be completely clear

Again, I'm sympathetic to the desire for a great digitally accessible and searchable book repository. I'm sympathetic to the underlying concern about the devaluation of knowledge and education that drives a lot of these arguments (for example, this blog, which said friend later forwarded). But arguments made in an undisciplined and sometimes misleading manner actually do more to hinder the cause than to further it, at least in my mind.

Trackback URL for this post:

http://www.esjewett.com/trackback/46

The infrastructure coop and the web

esjewett — Sat, 15 Aug 2009 15:23:17 +0000

(In which I take a moment to meander on a topic about which I know very little.)

In Doc Searls' recent writing on "The Ultimate Alignment" he writes on the disconnect between the interests of the companies that are building the web and the people who are using it. To summarize, companies have (or are supposed to have) their investor's best interests in mind rather than the best interests of their customers and users.

Theoretically, our efficient market system is supposed to align these two groups of interests. Or at least mediate between them. In theory, theory and practice are the same. In practice they tend to differ and one group of interests has a tendency to be subverted. When it comes to situations like the recent Facebook acquisition of Friendfeed or cable and telecom companies colluding with the recording industry to restrict peer-to-peer connections, the disconnect between the interests of users and investors comes into stark relief, and we see that users can be left out in the cold when interests conflict.

Customers have generally been getting the short end of the stick, or the pointy end if they're especially unlucky. This can, and probably should be attributed to a lack of customer organization and a resultant lack of bargaining power; something that is starting to be addressed with varying success by the social web, and which Searls sees culminating in a real capability for direct customer negotiation, which he calls Vendor Relationship Management.

A somewhat alternative approach proposed by Dave Winer, which Searls writes about in the above linked article and to which he appears sympathetic, is the tale in which the customers seize the reins of power and re/self-organize companies into a sane and impeccably moral engine of economic progress, for the good of all. [Ed. Sometimes he has a hard time keeping the lid on the sarcasm, especially in entries written on airplanes. Must be the altitude.]

Despite the sarcasm, I'm incredibly sympathetic to this idea, so after all of that throat clearing, I'd like to think about how it would work. What Searls and Winer are suggesting is a customer-owned coop.

Coops are cooperatives

My understanding of a customer-owned coop is that it is a business which is wholly owned and capitalized by its customers and the purpose of which is to provide a specific service rather than to turn a profit. If additional capital beyond what is required to deliver the service is generated, that capital is returned to the customer/owners. If an existing customer would like to become an owner then they can invest the requisite amount in the coop that is required to receive an equal share. Usually there is some ownership incentive provided, such as the ability to make use of the coop's services, or a discount on those services.

There are lots of kinds of coops, from housing and utility coops with services available only to owners to grocery stores and book stores which operate as commercial entities and often provide only slim discounts to owners. Public companies and governments are arguably coops, though they are not usually acknowledged as such and are usually not strictly customer/user-owned. (Though this is usually fairly close to the truth in the case of democratically elected governments with an electoral process that is relatively uninfluenced by monetary concerns ... and we know there are plenty of those around. [Ed. This sarcasm is getting a tad out of control, isn't it?])

Coops are often formed to establish some vital piece of infrastructure that a community feels is not adequately provided by existing commercial or governmental entities. A community that is lacking a quality grocery store will sometimes establish one as a coop. A university community that feels its access to academic books is restricted or provided at far too high a price by a commercial book store might establish a coop bookstore, effectively buying itself a distribution channel that did not exist before. Same goes for utilities.

In this way of thinking, coops are usually focused on infrastructure, with a special focus on distribution infrastructure during the 20th century. Distribution of food, books, natural gas, or electricity.

Open source is a bit like a coop ... and how

Open source software development is like a coop. The buy-in is contribution, of code, testing, bug-wrangling, documentation, publicity, or community process. The ownership privileges are varied. Contributions of code usually buy the contributor partial ownership of the code-base and the accompanying control, depending on the license - a privilege that cannot be purchased in any other way. Same goes for contributions of documentation vis a vis the body of documentation. All other contributions buy a less well defined voice in the community process.

Open source projects often resemble coops in another manner in that they are usually established to fill a need that a community feels is not adequately met (or is not met at an appropriate price) by existing commercial offerings. Some of the most successful open source projects have been and continue to be focused on infrastructure.

Open source as barter coop and the problem therein

This ground has been pretty heavily tread before, but the above description is of a coop established via barter, rather than exchange of more abstract monetary instruments and the signing of binding legal contracts. Importantly open source projects are only open to ownership by individuals who are prepared to deal in a particular instrument of barter. Open source is a bazaar, but it is a bazaar that is only accessible to those who can pay the entrance fee in a particular currency. It is a bazaar, frequented only by merchants and large distributors, not by individual shoppers. It has the best prices and the best goods, but good luck getting to them.

And therein, of course, is a problem with the open source cooperative model. The strength of many cooperatives is that any interested user can become an owner (and often does), and this is the model that Winer and Searls would like to see, it seems. But in the open source world this is not the case. In meat-space cooperatives, and bazaars for that matter, the answer is money. Anyone with money can buy a share in a coop grocery store. You don't necessarily need to be able to contribute broccoli or organic lamb to participate.

Coops as cooperatives (and the trouble with staying that way)

Often coops seem to work best when they are providing services to a local community. This may be partially the case because it reduces the interest in joining a coop for reasons other than taking advantage of the primary service that a coop provides. This local focus can be used as a hedge against the possibility of a group of disinterested investors taking over the coop and turning it into a profit-making operation, divorced from a direct interest in providing a service to its customers and instead focused on providing a service to its owners (now two different groups of people). Viola! Modern commercial entity.

For this reason I very much doubt that founding a new company and immediately IPOing as Winer suggests would not accomplish the goal of creating a company that is truly aligned with its users and customers to provide a service they desire or require. Some guard or incentive must exist, as is the case with coops, to keep the coop owned by its users and customers, yet still open to membership from new and interested users and customers.

Open source projects don't have the local angle going for them, but truthfully there is precious little to gain from participating in an open source project if ones goal is not to improve the project. Ego and a bullet on the resume perhaps. Through narrow focus and severely restricted ownership, these projects make sure that all owners are user/customers, even if they cannot guarantee (and in fact do not aspire to a reality) that all users are owners.

And so we come to the point

We have open source projects, which are fairly successful at making sure that they are guided primarily by input from user/customers but which can do a fairly bad job of making sure that all users/customers are well-served. We have commercial entities which do a good job of serving their owners but a bad job of making sure that their owners are also their customers, resulting in divergent interests. We have local coops, which tend to do a pretty good job of both but don't tend to scale well past a local community.

And then we're talking about essentially a distributed infrastructure coop that shares some of the best features of local coops and commercial entities. What structural features could a web-infrastructure cooperative boast that would meet the following necessary requirements?

Any user/customer can (and often does) become an owner
Only user/customers become owners (with the possible exception of a small minority ownership by other interested parties)

I don't know the answer to this question. In fact, a part of me doubts that there is an answer and that we will be better served attempting to scale community processes. But it seems it is the crux of the problem of the dis-alignment in web infrastructure.

Trackback URL for this post:

http://www.esjewett.com/trackback/45

Make a webhook out of anything

esjewett — Fri, 07 Aug 2009 23:28:24 +0000

(or ... "How I learned to stop worrying about doing it right, and just make the damn thing work")

I've had this problem for a while:

I use a great service called Instapaper (try it, seriously) for keeping track of my reading list. Which is great. But I want stuff to happen to certain items once I'm done reading something in Instapaper. They should be posted to Twitter, stored in Evernote, or squirreled away in Diigo, del.icio.us, or Pinboard.

This just isn't very achievable. While Instapaper is totally awesome, it does not provide automatic posting to all (or any) of these sites. It doesn't even provide Webhooks, which might provide the ... hook ... to allow for this sort of posting via some service like Yahoo! Query Language (YQL) or Tarpipe. What Instapaper provides is an RSS feed of shared items.

I've previously reviewed how to use YQL to post to Tarpipe, which solves part of the problem. It is possible to use this technique to consume a feed in Yahoo! Pipes and then post each item to Tarpipe. But this isn't really what I want to do, because it will post each item to Tarpipe each time the feed is read. Which is going to result in a lot of duplicate Tweets, Evernote notes, or whatever else I'm having Tarpipe do.

What we need is a Yahoo! Pipe that will only call the special Tarpipe (or other) YQL when there is a new item in the feed. Pipes isn't very good at this, but in steps Google Reader. Pretty much all that Google Reader does is query a feed occasionally and keep track of when a new item appears.

My strategy (and it works, bless Google and Yahoo!'s hearts) is to use Google Reader to check a Pipe and cache the items it has seen in a publicly accessible label. This is a little circular, but the very Pipe that Google Reader is checking pulls the feed I want to webhookify and compares the contents of the feed to the contents of the Google Reader label. If the Pipe sees any items in the feed that aren't yet stored in the Google Reader label, it does its magic on only those items.

I've made the Pipe that does this public at http://pipes.yahoo.com/esjewett/feed_to_webhook_using_google_reader but getting it working take a little doing, which is described below with screenshots. I'll demonstrate using a public feed (Daring Fireball's main article feed, because just about every single article is worth bookmarking), but you could use it on any feed that Yahoo! Pipes can access. I use it on my Instapaper starred items feed, among other things.

Step 1

Determine your Google Reader User ID by selecting your Shared Items feed in Google Reader:

Once you've selected this feed, take note of the URL in the address bar of your browser. It includes a string that is your Google Reader User ID. The ID contains only numbers. That "F" in front of it is not part of the ID and the "%" after it is not part of the ID. Copy this ID down somewhere as you'll need it later.

Step 2

Create your Pipe by cloning http://pipes.yahoo.com/esjewett/feed_to_webhook_using_google_reader

Step 3

Populate the user input fields of the Yahoo! Pipe with the feed you want to use ("http://daringfireball.net/feeds/articles" in our case), the Google Reader User ID from step 1, and the label you are going to make publicly accessible in Google Reader in a later step. The label you choose is important. It needs to be a label that is used for only this purpose and only this feed. Make it unique and call it something that will remind you of its purpose. I'll call mine "Daring Fireball Articles".

Once you've done all this, click the "Run Pipe" button.

Step 4

You should at this point get a list of the latest Daring Fireball articles, or whatever is in the feed that you've chosen to use. Click the button to add the Pipe results to Google Reader.

Click the "Add to Google Reader" button.

Step 5

Now you should be in Google Reader staring at your newly subscribed feed. Click the "Feed settings" button.

Then choose "New Folder..." from the bottom of the list of option, and name the folder whatever you put in as the "Google Reader Label" above. In our case, it is "Daring Fireball Articles".

You should see the feed on the left sidebar, under the folder you just created.

Step 6

Now we have all the infrastructure in place to actually do something with this feed. But we have not yet defined the action that our webhook Pipe will execute. So we need to tweak this pipe slightly. Go back to the pipe. You'll find your cloned version (And you did clone it didn't you?) at http://pipes.yahoo.com/pipes/person.info, at the top.

Edit the pipe.

At the lower-right corner of the edit screen is a loop operator with no module or pipe defined.

Drag any pipe or valid module into this loop. This action will be called exactly once for every new item in the feed you have just defined. If you want to post to Tarpipe, I recommend taking a look at the Pipe http://pipes.yahoo.com/esjewett/post_to_tarpipe_1_0_api (you'll have to clone it as well), which will post each item to tarpipe, using fields you specify as the title and body of the post. But you could call an arbitrary pipe that makes a call to a web service or even YQL.

I am using this Pipe to call a Tarpipe that posts ever Daring Fireball article into Evernote automatically (the Tarpipe workflow key is fake, so don't get any ideas :-)

That's it.

Using this method of setup the pipe will not process existing entries in the feed, but it will process any new entries through the pipe you have assigned to the loop.

One limitation of this particular pipe is that it will not work reliably for feeds that are updated often. This is simply because Google Reader doesn't poll often enough. I have observed that Google Reader polls this feed every 4-8 hours. If more than 8 items are added between polls, older items will not be picked up by Google Reader and will not be processed by the pipe.

Ok, that's it. For real this time.

Trackback URL for this post:

http://www.esjewett.com/trackback/44

Why use OAuth in clients? (OAuth FAQ Part 3)

esjewett — Fri, 24 Jul 2009 16:20:34 +0000

There is an ongoing meme that OAuth provides little or no security benefit in client applications. I believe that this assertion is incorrect.

Client applications are applications that run on my computer or phone and so are implicitly "trusted" by me. (I use quotes because I think that the assumption of user trust for client applications is wrong, but that is neither here nor there.)

I already addressed this somewhat obliquely in my OAuth FAQ Part 1 and Part 2, but it bears repeating, which I just did on the OAuth Google Group here.

To quote myself:

One major security benefit of using OAuth for client apps is that the client is only provided with an access token for the service and not the user's password.

If the access token and consumer secret are compromised, then either can be revoked, either by the user (in the case of the access token) or by the provider (in the case of the consumer secret). In most provider implementations a request authorized with an access token is not allowed to update certain aspects of the user's account, such as the password.

If the client requires the user to input their password (for example, Google's ClientLogin protocol) and the client becomes compromised, then the password is exposed, allowing full access to the user's account. Game over.

In my mind, this is *the* reason I want clients I use to use OAuth rather than a username/password login scheme.

I would be happy with another token-based login scheme as well, but OAuth is a perfectly good, publicly reviewed standard and I see no reason why a provider should cook up a bespoke token-based authorization scheme when OAuth is available and works for clients.

In other words, developers should use OAuth in their phone and desktop applications and users should demand it. I do.

Trackback URL for this post:

http://www.esjewett.com/trackback/43

Tarpipe with YQL

esjewett — Tue, 21 Jul 2009 00:42:51 +0000

I've been complaining for months that there is apparently no way to get a Yahoo! Pipes -> Tarpipe connection. What I'm really looking to do is drive a Tarpipe workflow using an RSS feed, so when a new item shows up in the RSS or Atom feed it will kick off a Tarpipe workflow for that item.

Unfortunately Tarpipe doesn't natively support feeds to kick off a workflow. It allows for the use of email or a specifically formatted request to one of its two APIs. Yahoo! Pipes meanwhile can consume RSS feeds second to none, but it is limited in the after-effects that it can trigger. Pipes does provide a web service operator that can send a request to an arbitrary URL, but the person building the pipe has very little control over the format of the request, so this is not a satisfactory way to interact with arbitrary APIs, and it doesn't work with Tarpipe's API.

For a couple of months I've been wondering if the relatively new Yahoo! Query Language (YQL) might be the missing link. In perusing the YQL documentation yesterday I noticed that YQL now supports INSERT, UPDATE, and DELETE operations in addition to its SELECT operation. Given the built in ability to make state-changing requests to APIs specified in YQL's Open Data Table (ODT) format, I decided to give it a shot.

I'll be darned, but it works.

I threw up an Open Data Table definition for the Tarpipe 1.0 API on Github, based on the provided YQL examples and the very understandable YQL documentation. Open Data Table definitions are XML files that specify the possible operations on a "table" as well as optional Javascript to be executed when the operation is called.

The complete Open Data Table for the Tarpipe 1.0 API (with the exception of the "image" field) is now available on my Github fork of spullara's fork of the yql-tables repository. Whew. Gotta love Github for creating a simple interface to a tangle of forks.

The stable raw XML file is here.

This definition specifies an ODT that responds to a query of the format
INSERT into tarpipe.process_1_0 (workflow_key,title,body) VALUES ("workflow_key_here","This is my title","This is the body")

Given a valid Tarpipe workflow key, which is found in the definition of a Tarpipe workflow based on a REST receptor, this call will kick off the workflow with the provided title and body parameters.

Since this ODT is not currently included in the standard environment, we also have to specify the ODT definition to use, so the complete query we need to send to YQL looks like
USE "http://www.esjewett.com/files/tarpipe.process_1_0.xml"; INSERT into tarpipe.process_1_0 (workflow_key,title,body) VALUES ("workflow_key_here","This is my title","This is the body")

And that's it. I (or rather Github) have already provided the hosted ODT definition, so all you need to do is run the YQL query above in order to kick off a Tarpipe workflow. The query can be run in any of the manners allowed by the YQL system, including the console available here. Simply replace the text 'workflow_key_here' (keep the double-quotes) with your Tarpipe workflow key and you should be good to go.

Next up, I'm hoping we'll be able to go through how to use this techinque to wire Yahoo! Pipes directly into Tarpipe workflows. I'm not quite there yet as it's tough to get Pipes to ignore old entries, but I think we can figure something out...

UPDATE (7 Aug, 2009) - Due to an obscure incompatibility between Ruby on Rails versions higher than 2.1 and the Yahoo! infrastructure, the github.com versions of the open data table XML files no longer work. I am now hosting the XML file necessary for this blog at http://www.esjewett.com/files/tarpipe.process_1_0.xml. I've changed the code and links above to point to this file.

Trackback URL for this post:

http://www.esjewett.com/trackback/42

Introducing jsglue

esjewett — Wed, 06 May 2009 23:03:01 +0000

Last weekend I pushed out the very first version of something I'm calling jsglue to Github. It now lives here: http://github.com/esjewett/jsglue/tree/master

jsglue is in essence a framework for implementing web-connective applications a la Yahoo! Pipes and Tarpipe. Currently it is at best a compliment to those programs and at worst totally useless. In the future I would like to see it or something like it become an alternative to these tools, for a few reasons that I'll eventually get into in later posts.

jsglue does three things:

It allows you to register a handler to a path.
- The handler consists of a path and two pieces of javascript - one that constructs a response to a request sent to that path, and one that constructs one request (and in the future multiple requests, optionally) to another URL.
It accepts HTTP requests to paths with registered handlers.
- When this happens, it creates a response using the handler javascript for this purpose, and it adds a job to a stack that will be processed later.
It provides a program that can be run periodically to process the stack of jobs that has built up, sending off new requests as specified by the javascript in the handler associated with each job.

That's it.

Why do I care?

Well, hopefully that will become clear of its own accord. But the key is that the full contents of the original request are exposed to the javascript processing script that constructs the new request. As such, you can do pretty much any kind of processing you like within this handler code, which is user-defined.

So why do you care? Let me count the ways:

1. Receive a request in JSON and spit it back out multi-part form-encoded (in fact, right now this is pretty much the only thing you can do). Ever tried to connect up Yahoo! Pipes with Tarpipe? It doesn't work. With this, it can.

2. Webhooks are great. Webhooks are the facility to have a web application issue an HTTP request to an arbitrary URL when some event happens in the web application. That sounds boring, but it's actually awesome. Webhooks are great, except that no one speaks the same language so every webhook-based solution is bespoke. Bespoke is great in a suit or a coffee mug, but it's bad in web infrastructure. Yahoo! Pipes can't understand webhook calls. Tarpipe usually can't understand them. Most other web applications can't understand them. There needs to be some sort of middle-person.

3. If I'm going to run a ton of my personal data through some middleman web application, you should have the option of running that web application yourself. I'm not saying you will, but I think it would be nice if you could.

Okay, 3 ways is enough for now. We'll get to more later.

Why do you *not* care?

Well, there are lots of reasons for that too.

1. This is dorky. No, there is not a UI. No, it doesn't do much of interest. It's an infrastructure prototype more than anything else. The idea is really that we need infrastructure for building applications that can do this sort of thing. I don't have a lot of time to spare, so I'm willing to just put a framework out there, and maybe a REST-only web-application if I can get the components running on a hosting service (harder than it sounds). I'll leave it to someone else to put the UI on top of it. I'm not convinced that the "pipe" metaphor is correct (I'm partial to "tubes" myself), but I don't have a better idea, so someone else will have to have that idea.

2. This code sucks. Yes it does. I urge you to fork it, improve it, or throw up your hands in disgust and start over. I just want something that does this. I don't really care if it's written by me.

3. There's no way this execution model will fly on a public site, and no one is going to run this on their own server. This is sort a feature of this design that allows your users to execute arbitrary javascript on your server. As such, I think this will primarily find use on private servers, or as a back-end engine for a public site where the inputs are carefully cleansed. Not a recipe for ultra-popularity, I'll grant. But that's not really the point either.

So what's it made out of?

Currently, there are only four main ingredients:

Ruby is the implementation language. It's role in jsglue is to serve as duct-tape for the other components.
Datamapper is the database interface, allowing you to use pretty much any supported database (I'm using SQLite at the moment).
Sinatra for the HTTP web-service interfaces. These interfaces are pretty much a direct mapping onto the database. (REST-ful? Maybe.) (Incidentally, how is it that a minuscule Ruby web-framework beats out FRANK SINATRA in the Google rankings?)
Johnson for the Javascript processing.

That's it. It's a couple hundred lines of code. I haven't really counted, or put it on Ohloh.com for that matter, but it can't be more than that. It's got some unit tests. It's going to be changing quickly as I make it more multi-purpose.

I'll document and post examples as they become available.

Trackback URL for this post:

http://www.esjewett.com/trackback/41

A tour of testing with an SAP focus (in the end)

esjewett — Tue, 28 Apr 2009 02:43:16 +0000

As might have been assumed from the my post on automated testing in SAP systems a couple months ago, I've been delving into testing in the SAP landscape. I'm beginning to put together a series of presentations and workshops on the subject, the first of which I was delighted to deliver last week.

Being the first in the series, this presentation focuses on an overview of the leading edge of the field. It would be nice to be able to emulate the sort of testing techniques we can use in Ruby in SAP BI and EPM application development. Nice, but not necessarily realistic. I can dream!

Testing Sap: Modern Methodology

View more presentations from Ethan Jewett.

Trackback URL for this post:

http://www.esjewett.com/trackback/40

How to get Johnson built

esjewett — Fri, 03 Apr 2009 02:38:05 +0000

Johnson is a Ruby wrapper of the SpiderMonkey JavaScript interpreter. In practice, this means you can use Johnson to evaluate JavaScript statements within a Ruby program.

It works really nicely, but there is no released gem and building the development gem is a bit of a headache. The instructions in the readme at the Johnson Github site appear simple but have several prerequisites that must be fulfilled before the whole process works. Here's what I did on Leopard. I'll try recreating on Windows tomorrow.

[sudo] gem sources -a http://gems.github.com
[sudo] gem install jbarnette-johnson

Multiple errors occur - if you see gems missing (like hoe, for example) then install them.

[sudo] gem install jbarnette-johnson

Errors occur, probably due to a failure to compile the native gem. Usually something about a missing rake/extensiontask.

[sudo] gem install rake-compiler
[sudo] gem install jbarnette-johnson

Yet another error occurs - a missing Manifest.txt file. Download from Github at http://github.com/jbarnette/johnson/tree/master and put it into the directory in the error message.

[sudo] gem install jbarnette-johnson

Success!

[Apr. 4, 2009 - Fixed typo, changing "rake-compile" to "rake-compiler"]

Trackback URL for this post:

http://www.esjewett.com/trackback/39

OAuth Q&A Part 2

esjewett — Thu, 26 Mar 2009 23:08:52 +0000

This the continuation of my ongoing OAuth Q&A, now with fewer links and more editorial commentary. See Part 1 as well.

If I have access to an authorized token and the consumer key and secret, I can make "authorized" API requests galore. So OAuth is no more secure than the username/password pattern.

That's not a question. Way to start off part 2 with a big fail!

[Ed. - Ahem . . . You're asking the questions as well as answering them.]

[Me - I deny that. But okay, let's try again at a real answer.]

This is only partly correct.

First, OAuth provides the a pattern for providers to scope access much more granularly than when a single username/password pair gives total access to an app. For example, most OAuth providers do not allow access to administrative functions when authorizing with an OAuth token, so a rogue client cannot change a user's password. Because of this, the user's access to the app can be guaranteed and damage can be limited to the scope of access granted to the client.

Second, because the user's access can be guaranteed and because client access can be managed at the level of the token, authorization for a given token can be revoked in a self-service manner. The scenario here is that a user grants access to a malicious app that vandalizes the account. The user realizes this, logs in to the provider, revokes the consumer's access, and repairs (hopefully) the damage. Because the malicious consumer cannot change the password on the account, the provider does not have to become involved in the initial response to the vandalism. Follow-on activities that the provider may need to become involved in (like account restoration or consumer key revocation) still exist, but the need for instantaneous response is lowered.

Hey, you just said "consumer key revocation", but back in Part 1 you said that the provider can't assume a consumer key uniquely identifies a particular class of consumers. What gives?

It is correct that the provider can't make this assumption based on the OAuth specification. However, the provider can impose requirements around the consumer key on consumer developers, and the provider can reserve the right to revoke a consumer key for whatever reason they choose. This could be required in the case of denial of service attacks or malicious consumers using a particular key. Providers should work closely with consumer developers to clearly lay out what actions the provider will take in these cases and what options consumer developers have.

Providers should recognize that the case of desktop and web apps are significantly different in the case of key revocation and they should plan accordingly. In the case of web apps, it should be relatively easy to replace a compromised consumer key and secret. In the case of desktop apps, the consumer developer may face the task of updating thousands or millions of installed applications in the case of a key revocation.

Let's assume that a user has installed a malicious desktop app that wants to use the API of my web app. What's the point of requiring OAuth, since this malicious desktop app already owns the user's system and will have access to anything the user types?

I'm not sure where this assumption comes from. Well, let me amend that. I know exactly where this assumption comes from, but it is still incorrect. Let me assure the reader that there are many platforms where the installation of a malicious application does not translate into ownership of the host system. The reader will find this sort of behavior in any properly configured Unix-like operating system including Linux, on the iPhone, in any modern browser, in Mac OS X, and even in some locked down versions of the Windows operating system.

On these platforms, even if the user is tricked into installing a malicious client or going to a malicious website, the user may still be careful to verify that they are only entering their password into the real site of the provider. There are a lot of options for verifying this, but the best way is for the user to make sure that he or she always enters there password into their provider's site using a modern browser with anti-phishing technology and using an https connection.

Trackback URL for this post:

http://www.esjewett.com/trackback/38

Background processing in Web Dynpro for ABAP

esjewett — Fri, 13 Mar 2009 22:47:56 +0000

This week I was writing a Web Dynpro ABAP interface for a long-running process. In general it is probably not a good idea from a performance perspective to run intensive processes synchronously in the context of a Web Dynpro view. It's also a really bad idea because it makes the user wait for the process to finish. If the process runs longer than the server's time-out setting then you get a nasty error message.

I knew that there was a way to handle this in ABAP, but I don't really live in the language, so it took me a little while to figure it out. Here's the skinny:

The answer is to write a function module to carry out the processing for you. Enable the function module for RFC (and by extension, set all of the parameters to "Pass by value"). You then call the function module from a Web Dynpro controller method in the background. This registers the function module for execution as an RFC function call in the background.

Once you've done all of your "CALL FUNCTION xxxxx IN BACKGROUND TASK." statements, use the "COMMIT WORK." command to trigger the execution of these functions. You end up with a code snippet that looks something like:

call function zsample_function in background task
* exporting
*    parameter =
* importing
*    ret =
    .

commit work.

Now your function will start running and control will be returned to the user in the Web Dynpro UI.

One important thing to remember is that this method provides no feedback to the user by default. Usually it will be a good idea to at least return a message explaining that the task has been started. If possible further feedback should be provided for long-running jobs. I was able to do this by querying a status table that my job populated, but other options are available.

Help documentation - http://help.sap.com/saphelp_nw70ehp1/helpdata/en/8f/53b67ad30be445b0ccc968d69bc6ff/frameset.htm

Trackback URL for this post:

http://www.esjewett.com/trackback/37