Paul James

E-mail Sign On

2008-12-05T20:26:29Z

E-mail Sign On

Single sign on, a great thing, saving us from "yet another password", or is it? OpenID is great, I have huge respect for all the people involved in the generation of this awesome specification, but is all this really needed? After all, we all already have a password and a distributed system for proving we know it.

Yes, e-mail.

E-mail as login credentials

Imagine this.

You come to my site and I need you to log in for some reason.
So I prompt you for your e-mail address and you kindly provide it.
I send you an e-mail and then tell you to go check your inbox.
You go grab your mail, however you like to do that, and follow a link in the e-mail to a unique one time signin page.
That page gets your browser to set a cookie and effectively logs you in.

Simple, and no username or password in sight, apart from the ones you use to access your e-mail already. If you need to log in again, you just follow the process again. Simple.

Plus most sites do this already with forgotten password functionality so it's not a foreign experience to Web users.

This is basically what OpenID does but without the human intervention of having to open an e-mail and click a link. I think having to check your e-mail and click a link is a pretty good compromise for not having to set up some new authentication system that may or may not be usable on the site you're visiting.

Benefits

Everyone has an e-mail address, I mean, why reinvent the wheel when we already have a system for doing this that everyone already uses. You're not going to get every Web user to start using OpenID, it's just to much bother when they can just remember (or get their browser to remember) a username/password pair.

One of the problems with any single sign on provider is that you have to trust that provider excusively (sure you can run your own OpenID server but who actually does that?). Everyone already implicitly trusts their e-mail provider, at least they do with their e-mail, which in all sense and purposes is all that counts due to forgotten password e-mails. I know that I'd prefer to only have to trust one entity, rather than an OpenID provider too.

Disadvantages

The main advantage of OpenID is that once you've authorised a site, you don't have to lift a finger again, the negotiation occurs in the background between the site and your OpenID provider without you doing a thing.

But this too could be handled via Web-mail providers and some browser hackery:

Rather than tell you to go check your e-mail in point 3 above, if I can detect that you are using a Web-mail provider, I can stuff an iframe into the page pointing to a specific Web page provided by your Web-mail provider that supports this technique.
This page will of cource be on your Web-mails domain and so as long as you are logged into your Web-mail, will be authenticated as you and can thus show the latest e-mail in your inbox sent from me.
You can then click straight away back into my site without leaving your browser or even having to do any funky redirection or message passing magic.

Not quite as seemless as OpenID, but a hell of a lot simpler, and the infrastructure from a user point of view is already there.

How I detect that you're using a Web-mail provider and where their little iframe page is is a little more tricky. We could do add a TXT entry to DNS for the domain to point to the pages URL, or maybe eventually we can just rely on the browser to provide the users e-mail address and Web-mail provider details automatically on the click of a button. Who knows, there's plenty to inovate around in this space.

Conclusion

Yes, existing identity providers opening up their systems to support OpenID helps since it gives many "normal" folk an instant OpenID, but many of these users don't understand what OpenID is or that they can use their Yahoo ID to log into OpenID supporting sites.

Using e-mail as an authentication mechanism makes sense. It makes sense for users and it makes sense for site developers, and it can be implemented and used today, without any additional infrastructure or extra support from 3rd parties.

Although I've never used it, apparently Craigslist do something very similar to this. You can create a posting without signing up to an account and all you have to provide is an e-mail address to which they send you a link that allows you to update your posting.

Some would call this security by obscurity, others would call it capability-based security, I call it clever.

Introducing the RMR Web Architecture

2008-11-28T17:43:01Z

Introducing the RMR Web Architecture

I talk a lot to people about REST, but with the acronym having become a commonplace among Web folk, I often find myself having to explain that what they have read about as REST or think of as REST is actually POX, HTTP-RPC or whatever you want to call it.

I'm also an advocate of the idea that REST isn't just about "Web services" but is also a good model for building fast sustainable Web applications. As such, in my quest to educate the ideas of REST, today I want to outline a different way to build Web apps, a RESTful way, an alternative to the popular MVC model, a model I call RMR, or Resource-Method-Representation.

Model-View-Controller

The current popular way of building Web applications is using the MVC pattern. This pattern separates the application into three parts:

The Model models your business objects, the "things" in your application, wrapping up data handling and logic.
The View is the output to the client.
The Controller manages the application flow manipulating data from the Model and generate a view.

This architecture has been widely embraced by Web developers even though it was designed for desktop applications. As far as I'm concerned, MVC doesn't really work on the Web, it doesn't model and expose resources in a convenient way (URL routing is evil and actions are methods on controllers exposed as resources rather than methods on resources themselves.

It just doesn't model resources, the fundamental element of the Web is totally ignored. So how can we fix this?

Resource-Method-Representation

We all agree that slicing our code up into pieces is a good idea, separating concerns allows us to concentrate on the particular task in hand and make maintenance easier, how can we separate it that makes more sense in a Web context than MVC?

Resource

We start with resources. The Web is made up of resources, so to make our application work with the Web we should model resources. A resource is an object within a RESTful system, they contain information, are identified a unique identifier (URLs in HTTP), and respond to the standard interface (in HTTP the standard HTTP methods of GET, PUT, etc.).

So in an OO language, a HTTP resource can be thought of as an object with private member variables and a number of public methods that correspond to the standard HTTP methods. From an MVC point of view, a resource can be thought of as a model with a bit of controller thrown in.

Method

Each request is routed to a resource automatically since each resource has a unique URL, and the method corresponding to the HTTP request method is run. This acts like an MVC controller, doing any operations necessary to prepare the request response based on the request.

Representation

Finally, the last part of the puzzle is the response. In a RESTful system, resources are exposed to the client as representations, where a representation is in essence a concreting of the resource into a wire format.

This Web page you are looking at right now for example, is a HTML representation of the resource that is this article. There could be other representations, you could view this article as a ASCII text file, or a PDF document, if I'd made them available to you.

So the representation is like a view in MVC, we give it a resource object and tell it to serialize the data into it's output format.

Putting it into Practice

So lets create some example code of how we might set up an RMR system. We'll do it in pseudo-code to keep things quick and simple.

Front Controller

First we'll need a front controller to handle all incoming requests and to do the routing:

request = readRequestData(); resource = loadResourceForThisRequest(request.url); response = callRequestMethod(request, resource); response.output();

This is pretty similar to an MVC front controller, we read data from the request and do something depending on what's there. The only real question is where do we load a resource from for a give URL?

A simple resource it might just be a class that pulls some data from somewhere, or no data as all the data may be in the representation (as with a HTML page).

The other case is that we have a collection of data we want to expose as resources, like a bunch of weblog posts, or the users of the application, or stock in a warehouse,.. you get the idea. In is case we probably want to map our collection URLs to a resource class that pulls a row from a database table.

Resource Class

To program our application, we write resource classes to model our application data, so what could a base resource class look like:

class Resource {
    
    private resourceData = [];
    
    method constructor(request, dataSource) {
        // load data from data source
    }
    
    method get(request) {
        return new Response(200, getRepresentation(request.url, resourceData));
    }
    
    method put(request) {
        return new Response(405);
    }
    
    method post(request) {
        return new Response(405);
    }
    
    method delete(request) {
        return new Response(405);
    }
}

Our class will load it's data on construction and then run one of the request methods. The methods that correspond to PUT, POST and DELETE requests all return a 405 response object (Method Not Allowed) since we don't want them to do anything by default, while our GET method returns a representation that corresponds to the request URL and the data loaded into the resource.

Routing

The final piece of the puzzle is how we route incoming requests to their respective resource classes. There are many ways to do this and the way you choose depends on personal preference.

We could follow a convention, saying that a URL of a format /something maps to a resource class called Something, and then for collections of resources we could say that any URL of a format /something/item would map to a single resource class representing the resources in the collection called something like SomethingItem.

The other extreme would be to map each URL or collection of URLs explicitly to a resource class.

Conclusion

So given a bunch of resource classes, some representations, and a way to map URLs to resource classes and resource methods to representations, we can build up a Web application in a RESTful way. Add in some HTTP goodness like conditional GET and cache headers, and we've got a pretty good system.

Virtual World

2008-10-28T23:04:28Z

Learning From the Web; Building an Open Virtual World

The time of Virtual Worlds might just be upon us. Technology has come far enough that it's now possible for clients to render complex 3D environments and to transfer a good amount of data over the network.

You only have to look at online gaming to see that we've been in this position for some time for a small portion of society, but now with faster and cheaper hardware we're finally reaching the point where many people's everyday desktop is perfectly capable of offering a 3D massively multiuser virtual environment.

The Walled Garden

So what do we have out there today? In 2003 Linden Labs launched Second Life and There Inc. launched There, both are closed worlds run completely by their parent companies. There are a number of other 3d worlds that have appeared since, many aimed at children and teenagers, but all of which are closed. This limits the their usefulness and cuts them off from the factors that made the Web the successful tool it is today.

The Lessons of the Web

So can we build an open 3D world, a 3D Web if you want to call it that, based upon the things we've learnt from the Web?

Sure we can, but first lets look at what it is that has made the Web what it is:

Simple open protocol and data formats
Open source server and client code
Stateless protocol to enable scalability
Decentralised architecture
Hypermedia
Constrained protocol grammar

So with that in mind, what parts do we need to build a 3D Web?

Application protocol

If we want our 3D world to be part of the Web, or at least a close cousin of the Web, it makes sense for us to use HTTP to move data from server to client. HTTP is great, it does it's job really well and is one of the three pillars of the Web.

So it makes perfect sense use HTTP to move the representations of our 3D world to our clients, but what formats should our representations take?

Content description language

There are two major 3D description formats, X3D and 3DMLW.

X3D is the 3rd evolution of the original VRML project started in 1994. It is a full blown 3D modelling language able to be generated by 3D modelling software, it's big and quite scary.

3DMLW is a lighter weight format, it simply describes a 3D space and allows you to place objects within it. These objects can either be 3D primatives defined in the source file itself or 3D models in a number of existing common 3D formats.

The thing I like about 3DMLW is that it mirrors HTML in structure, being a simple document structure that links in more complex media via hypermedia (the power of the URL).

Code on demand

Javascript, Lua

Hypermedia

XHTML, Xlink

<?xml version="1.0"?>
<document>
    ...
    <content3d id="content" x="0" y="0" width="100" height="100">
        ...
        <object xlink:href="http://www.peej.co.uk/another.3dmlw" x="1659.999878" y="-17.869379" z="18" pitch="0" yaw="0" roll="0" />
    </content3d>
</document>

Presence and chat

XMPP, IRC

<?xml version="1.0"?>
<document presence="irc://irc.example.org:194/#example">
    ...
</document>

PRIVMSG #example :LOC -2.958862 13.826897 13.263217 48

LOCATION #example :-2.958862 13.826897 13.263217 48

Hypermedia as the Engine of Application State

2007-12-29T18:54:51Z

Hypermedia as the Engine of Application State

REST defines four constraints upon the architecture of a Web application:

A universal syntax for resource-identification
A set of well-defined operations
Having a shared set of media-types
Hypermedia for application state-transitions

The first three are well understood but the fourth tends to slip through peoples thoughts when in fact it's one of the most important and easiest to grasp of the RESTful concepts.

There's always talk of a need for some kind of description language for the Web, a way for clients to know what resources are out there and which actions can be done upon them. But of course, in a RESTful system, you don't need such a thing as hypermedia and the well-defined set of operations (the HTTP methods) take care of it.

Using hypermedia

So, lets say we have an application, maybe it's a ordering process application that allows clients to place orders for different kinds of widgets.

We could have a client application that has all of the processing data baked into it, it could know the IDs of all the widgets up front, it could know how to format and submit an order, but this would closely tie our client application to our server application creating tight coupling between the two.

RESTful systems avoid this kind of tight coupling by using hypermedia to guide the client application. So instead of having a hard coded client, we have a generic "order processing" client that knows how to interpret the representations our server produces and take certain actions depending on their contents.

So lets presume that we want an automated system to place an order for widgets everyday based on the level of widget stock in our own stores, we wouldn't want to run out of widgets now would we. An interaction with the service might look something like this:

Client request

GET /
Host: widget-store.example.com
Accept: application/x-order-process+xml

Server response

Etag: "1f0417-14d6-7f24a1c0"
Content-Length: 369
Content-Type: application/x-order-process+xml

<?xml version="1.0"?>
<shop checkout="/checkout">
	<catalogue>
		<item stockNumber="123" name="Small Widget" href="http://www.peej.co.uk/products/123" />
		<item stockNumber="456" name="Medium Widget" href="http://www.peej.co.uk/products/456" />
		<item stockNumber="789" name="Large Widget" href="http://www.peej.co.uk/products/789" />
		<item stockNumber="abc" name="Sparkling Widget" href="http://www.peej.co.uk/products/abc" />
	</catalogue>
</shop>

So our generic client application makes a HTTP GET request to our widget stores primary URL (aka its homepage). This is the only thing we need to tell our application up front, as long as both the widget store and our application can understand each others representations we're in business, this is why we have standard for document formats (like HTML, CSS, JPEG, etc. or in this case the freshly created just for this task "application/x-order-process+xml").

The respresentation our application gets back outlines the widgets available from the store and how to place an order. Our application knows that item nodes within a catalogue node are widgets we can purchase, it also knows that the checkout attribute of the shop node contains the URL that begins the order placing process.

Client request

GET /products/789
Host: widget-store.example.com
Accept: application/x-order-process+xml

Server response

Content-Type: application/x-order-process+xml

<?xml version="1.0"?>
<product>
	<stockNumber>789</stockNumber>
	<name>Large Widget</name>
	<stockLevel>2</stockLevel>
	<price currency="GBP">7.99</price>
	<description>This widget is larger than the standard widget in both size and price</description>
</product>

Our client application knows that we're running out of "789" widgets, so we need to order some more. So we get the URL for the "789" widgets from the stock list and GET a representation that shows our application that they have 2 in stock. Luckily we don't use many "789" widgets so we only need to order 1 to keep our stock levels happy.

If we also need some other widgets we can repeat this procedure building up a list of stock numbers and quantities of all the widgets we require. If they don't have enough in stock, our application can issue some kind of warning, maybe it can e-mail you to warn you that we might be about to run out of something.

Once our application knows what it wants to order, it can GET the checkout URL:

Client request

GET /checkout
Host: widget-store.example.com
Accept: application/x-order-process+xml

Server response

Content-Type: application/x-order-process+xml

<?xml version="1.0"?>
<order
	submit="/checkout/submit"
	method="post"
	type="application/x-order-process+xml"
>
	<input name="name"/>
	<input name="company"/>
	<input name="address"/>
	<multiple name="item">
		<input name="stockNumber"/>
		<input name="quantity"/>
		<input name="totalPrice"/>
	</multiple>
	<input name="total"/>
</order>

Now if you're thinking "that looks like a HTML form with different tags" then you're right. HTML forms are HTMLs way of showing the client (in its case a HTML browser) how to send data to the server, and this is our "order process" applications way.

Using this, our application can formulate a representation to send to the server to place our order:

Client request

POST /checkout/submit
Host: widget-store.example.com
Content-type: application/x-order-process+xml

<?xml version="1.0"?>
<order>
	<name>Jack Bauer</name>
	<company>CTU</company>
	<address>Los Angeles</address>
	<item>
		<stockNumber>789</stockNumber>
		<quantity>1</quantity>
		<totalPrice>7.99</totalPrice>
	</item>
	<total>7.99</total>
</order>

Server response

Content-Type: application/x-order-process+xml

<?xml version="1.0"?>
<confirmation orderNumber="123-456" href="http://www.peej.co.uk/orders/123-456"/>

But why go through all of this rather than just hardcode the required representation into our client? Well we could, but then the client would be tightly coupled to the server, so if the server changed, the client would need to change too. Not a problem when you control both the server and the client, but if like in our example, the widget company runs the server and we run the client, then we'd be in trouble if the widget company decided that an extra step was required in the ordering process, say an extra confirmation stage.

Client request

POST /checkout/submit
Host: widget-store.example.com
Content-type: application/x-order-process+xml

<?xml version="1.0"?>
<order>
	<name>Jack Bauer</name>
	<company>CTU</company>
	<address>Los Angeles</address>
	<item>
		<stockNumber>789</stockNumber>
		<quantity>1</quantity>
		<totalPrice>7.99</totalPrice>
	</item>
	<total>7.99</total>
</order>

Server response

Content-Type: application/x-order-process+xml

<?xml version="1.0"?>
<confirmation
	orderNumber="123-456"
	href="http://www.peej.co.uk/orders/123-456"
	submit="/orders/123-456/confirm" method="post"
	type="application/x-order-process+xml"
/>

Client request

POST /orders/123-456/confirm
Host: widget-store.example.com
Content-type: application/x-order-process+xml

<?xml version="1.0"?>
<order>
	<name>Jack Bauer</name>
	<company>CTU</company>
	<address>Los Angeles</address>
	<item>
		<stockNumber>789</stockNumber>
		<quantity>1</quantity>
		<totalPrice>7.99</totalPrice>
	</item>
	<total>7.99</total>
</order>

Server response

Content-Type: application/x-order-process+xml

<?xml version="1.0"?>
<confirmation orderNumber="123-456" href="http://www.peej.co.uk/orders/123-456"/>

So as long as our client application knows how to understand and generate "application/x-order-process+xml" documents, and as long as the server also only generates and receives "application/x-order-process+xml" documents, then everything will keep on working, even if things change at either end.

All of this and no resource description in sight. All the co-ordination is dictated by the fact that we have a uniform interface defined by the HTTP and URI specifications, and that we have decided upon a standard grammar for our representation exchanges.

Web services as Web sites

None of this is rocket science, none of it is new, we've been building Web applications for human consumption using HTML as our representation grammar for the last 15 years, so why change anything for robot consumption?

When it comes to building a Web "service", remember that you are really just building a Web site with a different representation format. The most important thing you can do is to get your format hammered down from the start (or pick an existing format if it suits your needs or extend an existing format if it doesn't so at least you can gain some network effects from it, Atom is a good place to start for many uses).

So next time someone goes on about how RESTful systems can't possibly work because they need some kind of description language to enable clients to be wired to servers, remind them of hypermedia and point them to any part of the Web for a real World example of REST in action.

Is Bit Torrent the future of the Web?

2007-03-13T00:00:00Z

Is Bit Torrent the future of the Web?

Håkon Lie of Opera Software spoke at a Silicon Valley WebBuilder event at Yahoo earier this month talking about the future of the browser. One thing he talked about was experimental native support for video formats, bypassing the 12 year old and crumbling Netscape plug-in format. This seems like an obvious idea seeing as the amount of video we have online these days, why not roll these things into a browser and make a more seemless experience.

This got me thinking about what else could be in the future for the browser and for the Web in general, not the obvious things in the near future like HTML5 support, CSS3 or native media format support. My mind went straight to Bit Torrent.

The obvious place for Bit Torrent is for file downloads, again Opera are miles ahead of the game by baking a torrent client into Opera in 2005, but I'm thinking of something all together more radical.

A major problem with the Web as it is today is bandwidth. We're all familiar with the Slashdot effect, as soon as something becomes popular the servers melt and server admins get very nervous. Bit Torrent on the other hand loves it when the heat is on, so surely there's a way we can leverage that power in the Web.

Using Bit Torrent to power the Web

The obvious way to use Bit Torrent would be to fetch inline assets associated with a Web page; images, Flash movies, and other static content we'd otherwise be HTTP GETting. I'm not an expect in how the Bit Torrent protocol works, but I'd imagine that it wouldn't be too difficult to provide all the assets for a page as a torrent file and cause each browser accessing the page (and sitting on the page after it's loaded) to join the swarm for that torrent. This way, any assets on the page would be fetched by Bit Torrent from the swarm so as soon as our Slashdot friends arrive our bandwidth and our servers are spared the majority of the load.

Of course we'd have to implement this in a backwards compatible way, the HTML link element could provide the solution, something like this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
	<head>
		<title>Page to be Slashdotted</title>

		<link type="text/css" rel="stylesheet" href="http://www.peej.co.uk/default.css">
		<link type="application/x-bittorrent" rel="torrent" href="http://www.peej.co.uk/heavy-lifting.torrent">
	</head>
	<body>
		<h1>Welcome Slashdotters</h1>

		<p>Here's the big image you were after:</p>
		<img src="http://www.peej.co.uk/big-image.jpg">
	</body>
</html>

Upon spotting the "torrent" link element, our Bit Torrent enabled browser would leap into action, downloading the torrent file and joining the swarm. Then for any inline resources that we need to download, we can check the swarm and grab the file from it rather than issuing a HTTP GET to the server.

Our server would obviously have to act as a Bit Torrent tracker and seeder, but this would be easy to implement as an Apache module, automatically seeding each asset of a particular mimetype.

I'm sure that either people are working on this right now, or there's something fundamentally wrong with my idea, either way, there are still plenty of Web problems out there to be solved.

Living Without Sessions

2006-09-27T00:00:00Z

Living Without Sessions

When we talk about sessions in Web app development, we're usually talking about server stored data about a particular client. Some Web frameworks use session state to track and hold information about the user throughout their journey through the site, however they go against the RESTful principles and should really be treated as a bug.

Developers became used to having sessions available to them, so when systems grew, became more complex, and started spreading over multiple servers, more and more hacks had to be introduced to keep the session support working, when in reality, sessions should never have been introduced in the first place.

This article isn't to argue the pros and cons of server sessions, you can research that for yourself on the Web, we're going to look at an example system that is commonly built using sessions and how we can avoid sessions and come up with a better system.

Building a shopping cart with sessions

We're going to look at a standard shopping cart application, it will do the usual, display a bunch of products to the client and allow them to place them into their cart.

With sessions, we'd create a session, tie it to the client, and use it to store the IDs of each product the user places in their basket. Great, until we get a gazillion users and our server burns up in a firey memory related hell or we try to load balance across multiple servers so we need to buy expensive load balancers with layer 7 sticky session support. Yuk! There must be a better way, well infact there isn't a better way, there are two.

Cart on the client

When you think about someone in real life going into a shop and placing items into their shopping basket, where is the basket? It's with the user. So why don't we model our online shop to mirror the real life scenario.

Web browsers used to be a document reader for displaying hypertext documents transfered over the HTTP protocol, but nowerdays they are an application platform thanks to the development and deployment of Javascript within the HTML document and the browser. So we can use Javascript to extend our clients browser to be able to store their shopping cart until they reach the checkout.

We have to do a little work to make the browser hold it's state between page requests. We have a few choices:

Place the catelogue part of the site in a frameset with a "basket" frame that stores and displays the users basket to them. Pretty easy to do but not great since frames are a usability nightmare and we have to do extra work to provide bookmarkable URLs to individual products.
Expect the client to also support cookies and stuff the cart data in there. Very easy to do and they can persist for as long as we need, but cookies weren't designed for client state storage and they'll be sent to the server with every request when we only need them to be available to our client.
Wait until the WHATWG Web Application 1.0 spec is finished and implemented in browsers and use the new sessionStorage object.

Cart on the server

On the server, isn't that the same as storing a session on the server?

Yes and no. We can store the users cart on the server as long as we do it in a RESTful way we won't hit any of the problems associated with user sessions. The problem with sessions is that we're storing client state on the server, however if we don't treat the clients cart as being part of their state and rather treat it as being part of the shop itself, then we can get around the problems with sessions.

Imagine a shop where rather than being self-service, there is a shopkeeper who upon being asked, goes and fetches the items you want and rings them up on the till. Now if we were to model that as our online shop we'd see that the client no longer has a basket as part of their state, the basket is part of the shop.

So how would this work? Looking at the interactions between the client and the server, it might look something like this:

Client	Show me your products.
Server	Here's a list of all the products you can buy at this shop.
Client	Good, okay, I'd like to buy 1 of http://example.org/shop/product/X, please place it in my basket, my username is "JohnDoe" and my password is "secretPassword".
Server	Okay, I've added 1 of http://example.org/shop/product/X into your basket, you can review your basket at http://example.org/shop/users/johndoe/basket
Client	I'd like to buy 1 of http://example.org/shop/product/Y as well, please place one in my basket, my username is "JohnDoe" and my password is "secretPassword".
Server	Okay, I've added 1 of http://example.org/shop/product/Y into your basket as well, you can still review your basket at http://example.org/shop/users/johndoe/basket
Client	Actually I don't want http://example.org/shop/product/X after all, please remove it from my cart, my username is "JohnDoe" and my password is "secretPassword".
Server	Okay, I've removed http://example.org/shop/product/X from your basket, you can review your updated basket at http://example.org/shop/users/johndoe/basket
Client	Okay I'm done, ring 'em up, my username is "JohnDoe" and my password is "secretPassword".
Server	Should I charge that to your expense account?

The thing to notice about this conversation is that it is stateless, every action from the client is independent of any other. This means that at any time, the user can run off and do something else, come back a few days later and carry on. It also means they could get some other service to add things to their shopping basket easily.

Ain't that just a session?

So how does this differ from storing the cart in a user session on the server? After all doesn't the conversation above also apply no matter whether we're storing the cart in a session or in a resource?

Firstly a users session is transitory, it's there when the user is there, but will be cleaned up and lost once they leave, in our RESTful design the cart is as integral as a user account, so whatever means we use to store and process user accounts we use to store and process the users cart (so it might be a MySQL database cluster with Memcached in front of it).

Secondly we can grab hold of the users cart since it's a resource that has a URL, so we can query and manipulate it at will, pass the URL to other services, etc.

The differences can seem very subtle, but it's the subtleties that make the difference, since we're explicitly creating and adjusting resources on the server all the problems of session handling disappear and are covered by our resource handling solution (our DB cluster for example).

Conclusion

Avoiding sessions is a bit of a purest stance, but it does lead to a more scalable and usable Web app. Keeping the clients state on the client is always a good idea, you can't scale better than by utilising someone elses computer, of course depending on your application and the technologies you are using your mileage may vary.

If you do find you need to store transitory client data on the server, think about re-working that data or the way your app. works so that the data has meaning, give it a URL and turn it into a resource the user can manipulate.

HTTP Caching

2006-09-27T00:00:00Z

HTTP Caching

HTTP has a very thorough and well supported caching mechanism, but in this age of the dynamic Web page, it often goes unused when it is needed the most. So what do we, as Web programmers, need to do to make sure our pages are cached correctly? Let's have a look.

The reasoning

What is the point of caching? The idea is that on the Web, it is often better to have stale data than to wait for the network. Data only changes every so often and even if it has changed, it's not always important that the client has the latest data, so caching it either at the client, or at an intermediary, isn't a problem and should infact be a good idea. RFC2616 says:

Caching would be useless if it did not significantly improve performance. The goal of caching in HTTP/1.1 is to eliminate the need to send requests in many cases, and to eliminate the need to send full responses in many other cases. The former reduces the number of network round-trips required for many operations; we use an "expiration" mechanism for this purpose. The latter reduces network bandwidth requirements; we use a "validation" mechanism for this purpose.

Allowing clients and intermediaries to cache our pages will lighten the load on our servers and create a faster user experience. One of the problems with dynamic Web applications is that your Web server thinks it is sending new data every request and so can't set the correct caching headers, you need to do a little extra work yourself within your application to make sure caching is working for you.

There are two types of caching in HTTP, we'll look at both, how they work, how they work together, and how to make sure your Web application is taking advantage of them.

Expiration

The expiration model is a way for the server to say how long a requested resource stays fresh for, user agents should cache the resource response and re-use it until its cache is no longer fresh.

Expiration is excellent for resources that change at known times or the change very rarely and whose staleness does not matter. For example, images and style sheets that are used across a site often do not change very often and so should be sent with a expiration date of at least 24 hours. The user agent will then only download them once, no matter how many pages of the site they visit.

If we are serving resources from a dynamic data source, for example say we have a graphic that portrays the current weather outside, if we know that the data only updates once an hour, we can set the expiration date to an hour so that clients only request it once per hour.

Expires header

The simpliest way of doing this is with the HTTP Expires header, it just contains the date and time of when the resource will become stale:

Expires: Sun, 25 Jun 2006 14:57:12 GMT

Cache-Control header

The Expires header has a few problems like requiring the server and client to have clocks that are in sync. So HTTP 1.1 replaced it with the Cache-Control header that offers more flexibility:

Cache-Control: max-age=86400, must-revalidate

The cache control header has a number of clauses that can be used to control the way the client caches the resource.

max-age=86400: The number of seconds the resource will be considered fresh, similar to the Expires header except the number of seconds from now is used rather than a specific date/time.
s-maxage=86400: The same as max-age, except that it only applies to shared caches.
public: Makes the response always cacheable even if it wouldn't normally be cacheable (behind authentication, etc.)
no-cache: Forces caches ask the server for validation before releasing a cached copy, so if the server says that the cached version is still fresh it is served from the cache.
must-revalidate: Forces caches to obey any freshness information you give them about a resource.
proxy-revalidate: The same as must-revalidate except that it only applies to shared (proxy) caches.

Validation

The validation model allows a client to ask the server whether a cached version of a resource is still fresh. If the client doesn't have a fresh cached version, the server will respond with a fresh version of the resource, while if it does, the server will say so and send nothing.

This is useful as it allows our server to tell a client it already has the freshest version and not do any processing. If the resource is dynamic, we can save our server from having to do all the work involved in producing the response since the client already has it in its cache.

Last-Modified header

Like the expiration model, there are two HTTP headers that control validation, the Last-Modified header is defined in HTTP 1.0 and sends the client the date/time when the resource was last modified:

Last-Modified: Sun, 25 Jun 2006 14:57:12 GMT

When a client sends a "If-Modified-Since" request header, the date/time sent in that header can be compared with the resources last modified date/time, and if it matches, a 304 Not Modified HTTP response sent.

Etag header

The Last-Modified header suffers the same problems as the Expires header, and thus was replaced in HTTP 1.1 with the Etag header. An Etag is a string that uniquely identifies a resource, they should be generated by the server in a way as to change whenever the resource does, a common Etag value is the MD5 hash of the resource or of the resources URL and its modified date/time stamp.

When a client sends a "If-None-Match" request header, the Etag value in that header should be compared to the resource and if it matches, a 304 Not Modified response sent.

Using caching

So to make sure your dynamic resource behaves well when it comes to HTTP caching, you need to support validation and optionally send a expiration header from within your script.

Supporting validation in PHP this is pretty simple, sending the correct headers is just a case of:

function sendValidationHeaders($mtime, $etag)
{
    header('Last-Modified: '.gmdate('D, j M Y H:i:s', $mtime).' GMT');
    header('ETag: '.$etag);
}

And doing the actual validation just requires a function like:

function isModified($mtime, $etag)
{
    return !((isset($_SERVER['HTTP_IF_MODIFIED_SINCE']) &&
        strtotime($_SERVER['HTTP_IF_MODIFIED_SINCE']) >= $mtime) ||
        (isset($_SERVER['HTTP_IF_NONE_MATCH']) &&
        $_SERVER['HTTP_IF_NONE_MATCH'] == $etag));
}

If you want to tell clients when your resource expires, you need a function like:

function sendExpireHeaders($seconds)
{
    header('Expires: '.gmdate('D, j M Y H:i:s T', time() + $seconds));
    header('Cache-Control: max-age='.$seconds.', must-revalidate');
}

Other caches

As well as HTTP caching, your Web application may also want to use various application caching mechanisms. If you're hitting a database often you may want to place an object cache like Memcache in front of it so that often requested data can be cached in Web server memory rather than being re-requested from the database. If your dynamic app has lots of pages that don't update very often you may want to place a reverse proxy like Squid in front of it to save re-generation of all non-changed pages across users. The setting up and using of other caches are beyond the scope of this article, an intro to using Squid as a reverse proxy can be found here and the PHP memcached extensions documentation is here.

For more info on HTTP caching, check out section 10 of RFC2616 and for an easier read, have a look at Mark Nottingham's Caching Tutorial.

HTTP Authentication with HTML Forms

2006-02-03T16:28:00Z

HTTP Authentication with HTML Forms

Authentication in Web applications has been highjacked, HTTP defines a standard way of providing authentication but most apps use the evil spawn of Netscape, otherwise known as cookies. Why is this? Cookies aren't designed for authentication, they're a pain to use for it, insecure unless you know what you're doing, non-standard, and unRESTful.

Warning: The solution outlined in this article is experimental and might be a complete lie, be warned that your mileage may/will vary.

The main reason people walk away from using HTTP authentication is that they want control over the look of the login form and most browsers display an awful looking dialog box. So what we need is a way for HTML forms to pass HTTP auth data when it's submitted. The HTML spec provides HTML forms as a way to create queries and to POST urlencoded data to a URL, but can we subvert it?

With the power of Javascript we can

We could add an onsubmit event to our login form that pushes the username and password values within our form into the URL in the forms action attribute. That way our login request would supply the users credentials in the URL and avoid the server returning a 401 response and causing our browser from showing the HTTP auth box.

Great, and pretty easy. We could even write the HTML form out with Javascript and provide a simple link to non-Javascript enabled browsers. But there's a problem, IE doesn't support usernames and passwords in URLs, they were removed due to a security scare, and anyway, the HTTP spec doesn't say we're allowed to have URLs with usernames and passwords in them so we can't guarentee that they work anywhere else either.

So is there an alternative way that doesn't require us to mung the username and password into the URL? Yes, our new friend, XMLHTTPRequest, it can submit the correct HTTP auth headers for us. Rather than adjusting the URL the form submits to, we can use XMLHTTPRequest to do a request before the form submits supplying the entered username and password. This will set up the browser with the HTTP auth credentials so it'll also send them with our actual form submission login request.

An example

Enough talking, here's some code. This is our login function that we bind to our form submission:

function login()
{
    var username = document.getElementById(this.id + "-username").value;
    var password = document.getElementById(this.id + "-password").value;
    this.http.open("get", this.action, false, username, password);
    this.http.send("");
    if (http.status == 200) {
        document.location = this.action;
    } else {
        alert("Incorrect username and/or password.");
    }
    return false;
}

It sends our XHR request with the given username and password, and then redirects the client on success or displays a Javascript alert on error. We need our standard getHTTPObject() function that I introduced here, and then some code to create our login form and set everything up:

window.onload = function()
{
    var http = getHTTPObject();
    if (http) {
        var anchors = document.getElementsByTagName("a");
        for (var foo = 0; foo < anchors.length; foo++) {
            if (anchors[foo].className == "httpauth") {
                createForm(anchors[foo]);
            }
        }
    }
}

function createForm(jshttpauth)
{
    var form = document.createElement("form");
    form.action = jshttpauth.href;
    form.method = "get";
    form.onsubmit = login;
    form.id = httpauth.id;
    var username = document.createElement("label");
    var usernameInput = document.createElement("input");
    usernameInput.name = "username";
    usernameInput.type = "text";
    usernameInput.id = httpauth.id + "-username";
    username.appendChild(document.createTextNode("Username:"));
    username.appendChild(usernameInput);
    var password = document.createElement("label");
    var passwordInput = document.createElement("input");
    passwordInput.name = "password";
    passwordInput.type = "password";
    passwordInput.id = httpauth.id + "-password";
    password.appendChild(document.createTextNode("Password:"));
    password.appendChild(passwordInput);
    var submit = document.createElement("input");
    submit.type = "submit";
    submit.value = "Log in";
    form.appendChild(username);
    form.appendChild(password);
    form.appendChild(submit);
    jshttpauth.parentNode.replaceChild(form, jshttpauth);
}

This looks for all anchors with the class of httpauth and replaces it with a nice HTML form with the login() function bound to it's onsubmit event.

So now if we set up a HTML page like the following, the login link will be replaced with a login form (if the user has Javascript support) but authenticate the user using HTTP auth:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
    <head>
        <title>Javascript HTTP Auth Test</title>
        <script type="text/javascript" src="http://www.peej.co.uk/jshttpauth.js"></script>
    </head>
    <body>
        <a class="httpauth" id="private" href="http://www.peej.co.uk/private.php">Log in</a>
    </body>
</html>

If our private.php looks like this, then we'll see that everything works as we'd like:

<?php

define('USER', 'user');
define('PASSWORD', 'password');

if (!isset($_SERVER['PHP_AUTH_USER']) || !isset($_SERVER['PHP_AUTH_PW'])) {
    header('WWW-Authenticate: Basic realm="My Realm"');
    header('HTTP/1.0 401 Unauthorized');
    echo 'You hit cancel, good on you.';
    
} elseif (
    isset($_SERVER['PHP_AUTH_USER']) && $_SERVER['PHP_AUTH_USER'] == USER &&
    isset($_SERVER['PHP_AUTH_PW']) && $_SERVER['PHP_AUTH_PW'] == PASSWORD
) {
    echo "<p>Hello {$_SERVER['PHP_AUTH_USER']}.</p>";
    echo "<p>You entered '{$_SERVER['PHP_AUTH_PW']}' as your password.</p>";
    
} else {
    header('HTTP/1.0 400 Bad Request');
    echo "<p>You shall not pass!</p>";
}

?>

The thing to notice about this script is the third part of the 'if' statement. If the request doesn't have any auth details we send a standard auth response, if it does and the details are correct then we let them in, otherwise we send back a 400 error. This stops the clients browser from asking the user for details again allowing the Javascript to process the failure.

This also means that if the client doesn't have Javascript and fails to authenticate correctly, it will never be asked for new credentials as long as it keeps sending the bad auth data (which browsers will do).

Problems

It's not all good just yet, this technique works in IE6 and Firefox but is known not to work in both Opera and Safari, so if you care about those browsers you may want to think again about using this (or to spend some time investigating why it fails in those browsers).

HTTP Digest Auth

HTTP Digest is a way of authenticating a client while never sending the clients password over the wire. To use Digest rather than Basic HTTP Auth, we only need to adjust our PHP script to implement digest:

<?php

include 'digest.php';

$HTTPDigest =& new HTTPDigest();
$users = array(
    'user' => md5('user:'.$HTTPDigest->getRealm().':password')
);

if (!$HTTPDigest->getAuthHeader()) {
    $HTTPDigest->send();
    echo 'You hit cancel, good on you.';
    
} elseif ($username = $HTTPDigest->authenticate($users)) {
    echo "<p>Hello $username.</p>";
    echo "<p>This resource is protected by HTTP digest.</p>";
    
} else {
    header('HTTP/1.0 400 Bad Request');
    echo "<p>You shall not pass!</p>";
}

?>

Conclusion

So there we have it, there is no need for HTTP authentication to be shunned, even for aesthetic reasons. HTTP auth offers lots of advantages over using cookies:

It's simple to implement and no hassle to use, and clients love it.
It carries no baggage unlike cookies.
It's tried and tested, it's a standard and it works, your cookie based security model won't be as good.
We can use HTTP Digest which is pretty secure.
It's about as RESTful as you can get with authentication.

So we should be using it, there's no excuses anymore.

I have put up an example of this code as a demonstration.

Note: I was first introduced to this idea by Christian Jensen and Jan Algermissen on the REST Discuss mailing list, so credit should be sent their way.

Update: Dimitri Glazkov made me revisit using HTTP Digest with this technique and realise that it does actually work, thanks Dimitri.

Update: Travis Estill and David Kleinschmidt reminded me that 401 responses shouldn't be returned without an Auth header and so a 403 is a better response code. This also helps to make Safari behave too.

A RESTful Web service, an example

2005-10-30T08:00:00Z

A RESTful Web service, an example

It's often hard for people to "get" REST, this is mostly due to the fact that REST isn't a tangible thing like a piece of software or even a specification, it's a selection of ideals, of best practices distilled from the HTTP specs.

I've always found that the best way to understand something is to see an example, to see the principles in action first and worry about the details later once I understand the general gist. So here's a little example of a RESTful version of a simple Web service you might already know about, the Delicious API.

Delicious has a simple REST API, or rather, a simple POX over HTTP API, that is, it has a perfectly usable HTTP and XML based API for accessing your bookmarks and tags, but it isn't very RESTful. Why not?

First class objects aren't exposed as resources, so we can't access our bookmarks or tags directly.
HTTP methods not used correctly, everything is done via GET even operations that change things.
Resource representations not interconnected, you can't traverse from a list of bookmarks to a single bookmark.

So (ignoring why) how can we make it better, make it more RESTful? Lets start with describing what it is the Web service does.

It allows us to:

Get a list of all our bookmarks and to filter that list by tag or date or limit by number
Get the number of bookmarks created on different dates
Get the last time we updated our bookmarks
Get a list of all our tags
Add a bookmark
Edit a bookmark
Delete a bookmark
Rename a tag

From this we can identify two first class objects; bookmarks and tags, so we'll expose both of these as resources at the URL paths of http://del.icio.us/api/[username]/bookmarks and http://del.icio.us/api/[username]/tags where [username] is the username of the users bookmarks we're interested in. We'll still use HTTP Authentication to limit peoples access to bookmarks, but we've added the users username into the URL since otherwise we'd be using HTTP Auth as part of our resource identification instead of just the URL and that'd be unRESTful.

We'll need to define some XML document formats to represent our bookmarks and tags. It doesn't matter exactly what they look like as long as they convery the correct information and we define mimetypes to identify them, but I'll include examples of my versions of them as we go along to make things clearer.

Table of XML document types we'll invent
Mimetype	Description
delicious/bookmarks+xml	A list of bookmarks
delicious/bookmark+xml	A bookmark
delicious/bookmarkcount+xml	Count of bookmarks per date
delicious/update+xml	When the bookmarks were last updated
delicious/tags+xml	A list of tags
delicious/tag+xml	A tag

Getting Bookmarks

In the POX Delicious API bookmarks are accessed by a RPC style request to http://del.icio.us/api/posts/get with a number of optional querystring parameters that influence the returned results. To do a similar thing RESTfully, we'll define a similar resource at http://del.icio.us/api/[username]/bookmarks/ that returns a list of bookmarks. We'll also define an infinite number of resources at http://del.icio.us/api/[username]/bookmarks/[hash] that represent our individual bookmarks where [hash] is the Delicious hash used to identify a bookmark.

Get all bookmarks

URL	http://del.icio.us/api/[username]/bookmarks/
Method	GET
Querystring	tag=	Filter by tag
	dt=	Filter by date
	start=	The number of the first bookmark to return
	end=	The number of the last bookmark to return
Returns	200 OK & XML (delicious/bookmarks+xml)
	401 Unauthorized
	404 Not Found

The bookmarks XML should include the URLs of each individual bookmark so a program can easily traverse to a single bookmark from the list of bookmarks. An example delicious/bookmarks+xml document might look something like:

GET http://del.icio.us/api/peej/bookmarks/?start=1&end=2 <?xml version="1.0"?> <bookmarks start="1" end="2" next="http://del.icio.us/api/peej/bookmarks?start=3&end=4"> <bookmark url="http://www.example.org/one" tags="example,test" href="http://www.peej.co.uk/http://del.icio.us/api/peej/bookmarks/a211528fb5108cddaa4b0d3aeccdbdcf" time="2005-10-21T19:07:30Z"> Example of a Delicious bookmark </bookmark> <bookmark url="http://www.example.org/two" tags="example,test" href="http://www.peej.co.uk/http://del.icio.us/api/peej/bookmarks/e47d06a59309774edab56813438bd3ce" time="2005-10-21T19:34:16Z"> Another example of a Delicious bookmark </bookmark> </bookmarks>

You'll notice the next attribute of the bookmarks element, this contains a URL to the next "page" of our returned results. Since a user might have thousands of bookmarks and we don't want to return them all, we page the results and provide next and previous attributes to allow us to get the the next and previous pages (we have no previous attribute in this example since we are on the first page). These URLs use the start and end querystring parameters to limit the results to a specific page.

Get info on a specific bookmark

URL	http://del.icio.us/api/[username]/bookmarks/[hash]
Method	GET
Returns	200 OK & XML (delicious/bookmark+xml)
	401 Unauthorized
	404 Not Found

Since a bookmark has a number of tags, it'd be RESTful to have the bookmark XML provide links to each tag:

GET http://del.icio.us/api/peej/bookmarks/a211528fb5108cddaa4b0d3aeccdbdcf <?xml version="1.0"?> <bookmark url="http://www.example.org/one" time="2005-10-21T19:07:30Z"> <description> Example of a Delicious bookmark </description> <tags count="2"> <tag name="example" href="http://www.peej.co.uk/http://del.icio.us/api/peej/tags/example"/> <tag name="test" href="http://www.peej.co.uk/http://del.icio.us/api/peej/tags/test"/> </tags> </bookmark>

Delicious provides two further retrieval "operations" on your list of bookmarks, a count of the number of bookmarks created on each date, and the time the user last updated their bookmarks. We'll implement these as two resources within the bookmarks path that return this information.

Get count of posts per date

URL	http://del.icio.us/api/[username]/bookmarks/count
Method	GET
Querystring	tag=	filter by tag
Returns	200 OK & XML (delicious/bookmarkcount+xml)
	401 Unauthorized
	404 Not Found

Get last update time for the user

URL	http://del.icio.us/api/[username]/bookmarks/update
Method	GET
Returns	200 OK & XML (delicious/update+xml)
	401 Unauthorized
	404 Not Found

Modifying Bookmarks

In the original API, you add a new bookmark by issuing a RPC style GET request to an add URL. Not very RESTful. Modifying is done by using the same add URL, and deleting is done in the same way but via a delete URL. We'll be more RESTful by using the correct HTTP methods on existing bookmark resources to add, modify and delete bookmarks.

Add a bookmark to Delicious

To create a bookmark we'll POST an XML representation of a bookmark to our bookmarks URL.

URL	http://del.icio.us/api/[username]/bookmarks/
Method	POST
Request Body	XML (delicious/bookmark+xml)
Returns	201 Created & Location
	401 Unauthorized
	415 Unsupported Media Type

Since to create a new bookmark Delicious doesn't need all of the information that we've added to our bookmark XML format, we'll allow some of the XML attributes and nodes to be optional:

POST http://del.icio.us/api/peej/bookmarks/ <?xml version="1.0"?> <bookmark url="http://www.example.org/one" time="2005-10-21T19:07:30Z"> <description> Example of a Delicious bookmark </description> <tags> <tag name="example"/> <tag name="test"/> </tags> </bookmark>

The 415 Unsupported Media Type error is returned if an incorrect XML document is sent.

On success, a 201 Created response is returned along with a location header containing the URL of the newly created bookmark resource.

Modify a bookmark

To modify an existing bookmark, we'll PUT an XML representation of a bookmark to the URL of our bookmark.

URL	http://del.icio.us/api/[username]/bookmarks/[hash]
Method	PUT
Request Body	XML (delicious/bookmark+xml)
Returns	201 Created & Location
	401 Unauthorized
	404 Not Found
	415 Unsupported Media Type

Since we don't want people to create new bookmarks using PUT, only update existing bookmarks, we'll return a 404 Not Found error if the resource doesn't already exist.

Delete a bookmark from Delicious

To delete a bookmark we'll use the HTTP DELETE method.

URL	http://del.icio.us/api/[username]/bookmarks/[hash]
Method	DELETE
Returns	204 No Content
	401 Unauthorized
	404 Not Found

On success, we'll return a 204 No Content code since we won't return an response body.

Finishing the API

Okay, so now that we've defined our new RESTful API, how do we use it? Well, REST says to "use hypertext as the engine of application-state transitions", meaning, given a starting URL, we should be able to fetch a resource and use the URLs within it to navigate to the other resources that we want. We shouldn't have to guess or construct any URLs ourselves since URLs are transparent and don't have to follow any particular formula or pattern (and since software can't easily spot patterns and construct URLs by itself).

So we're missing a homepage for our API, let's create one. This will be the entry point for our RESTful API, by giving this URL to a program that knows about our Delicious XML data formats, it can find all the data it requires without having to know anything about the rest of the API.

URL	http://del.icio.us/api/
Method	GET
Querystring	start=	The number of the first user to return
	end=	The number of the last user to return
Returns	200 OK

GET http://del.icio.us/api/?start=1&end=2 <?xml version="1.0"?> <users start="1" end="2" next="http://del.icio.us/api/?start=3&end=4"> <user name="Peej" href="http://www.peej.co.uk/http://del.icio.us/api/peej/" bookmarks="http://del.icio.us/api/peej/bookmarks/" tags="http://del.icio.us/api/peej/tags"/> <user name="Joshua" href="http://www.peej.co.uk/http://del.icio.us/api/joshua/" bookmarks="http://del.icio.us/api/joshua/bookmarks/" tags="http://del.icio.us/api/joshua/tags"/> </users>

Each user then needs to be defined as a resource, we failed to spot the user as a first class object when we started, so we'll add it now.

URL	http://del.icio.us/api/[username]/
Method	GET
Querystring	start=	The number of the first bookmark to return
	end=	The number of the last bookmark to return
Returns	200 OK
	401 Unauthorized
	404 Not Found

GET http://del.icio.us/api/peej/?start=1&end=2 <?xml version="1.0"?> <user email="paul@peej.co.uk" name="Paul James" href="http://www.peej.co.uk/http://www.peej.co.uk/"> <bookmarks href="http://www.peej.co.uk/http://del.icio.us/api/peej/bookmarks/" start="1" end="2" next="http://del.icio.us/api/peej/?start=3&end=4"> <bookmark url="http://www.example.org/one" href="http://www.peej.co.uk/http://del.icio.us/api/peej/bookmarks/a211528fb5108cddaa4b0d3aeccdbdcf"/> <bookmark url="http://www.example.org/two" href="http://www.peej.co.uk/http://del.icio.us/api/peej/bookmarks/e47d06a59309774edab56813438bd3ce"/> </bookmarks> <tags href="http://www.peej.co.uk/http://del.icio.us/api/peej/tags/"> <tag name="example" count="2" href="http://www.peej.co.uk/http://del.icio.us/api/peej/tags/example"/> <tag name="test" count="2" href="http://www.peej.co.uk/http://del.icio.us/api/peej/tags/test"/> </tags> </user>

The user resource could have a PUT method to allow updating of users settings and a DELETE method to remove the user and all their bookmarks and tags.

An example of using the API

Let's finish this example RESTful API with an example of a client program using it.

I've been messing around with Delicious and have created a number of bookmarks for testing purposes. I don't want them anymore, so our pretend program is going to delete them for me. I tagged them all up with the tag "test" so I'd be able to find them later, I'll use this tag to track them down and remove them.

GET http://del.icio.us/api/

First, we start like every client, by fetching the APIs homepage. We parse the response XML and find the tags URL of the user with the name "Peej". This gives us http://del.icio.us/api/peej/tags/.
GET http://del.icio.us/api/peej/tags/

So we get the resource that represents my tags. We parse the XML and see if the tag "test" is mentioned. If it isn't, then we GET the URL in the "next" attribute and try again. Eventually we'll get the URL http://del.icio.us/api/peej/tags/test
GET http://del.icio.us/api/peej/tags/test

Now we've got the details for the tag "test", so we parse the XML and get a list of bookmarks. If there is a "next" attribute, we'll follow that to get the next page of bookmarks. Eventually we'll have a list of all bookmarks tagged with "test".
DELETE http://del.icio.us/api/peej/bookmarks/[hash]

Finally we can walk through our list of bookmarks and call the DELETE HTTP method on each of their URLs.

Now this is quite involved and we could take shortcuts by assuming that we know the URL of the tag we want is http://del.icio.us/api/peej/tags/test. What it does show is that with a RESTful API it is possible for a piece of software to use the Web in a similar way that a human does to perform a task, there's no need for description languages like WSDL, REST is self descriptive.

Conclusion

Most "REST" APIs around today are not very RESTful, most are as RESTful as equivalent SOAP or XML-RPC services. If you are using HTTP you are being RESTful to some degree since HTTP is a REST protocol, but to take full advantage of the platform, APIs should embrace RESTful practices as much as possible.

I hope this article has been intersting, and I hope your next Web software product will embrace REST principles for its machine and human interfaces.

Content Negotiation

2005-07-23T07:00:00Z

Content Negotiation

Content negotiation is something I've been looking at lately as part of the development of my RESTful Web management system I'm working on in my spare time. The idea is as old as the Web itself and is simple to understand, for a single resource you have multiple representations in different data formats and which one we send depends on the accept headers sent by the client.

We all know it's good practice to trim file extensions from your URLs so that you are not tied to a particular data format. Content negotiation says that a URL like http://www.example.org/example would be translated by conneg to http://www.example.org/example.html if the client wants HTML and the file example.html exists.

By translated, I don't mean changed, this all happens transparently to the client. Also, an interesting affect is that now we have multiple URLs for the same resource. We have a single URL that will in effect alias the URL that best suits the client, and a number of URLs for each data format.

I came across this article on the MSDN Magazine site, the first article in the "inside MSDN" column about the new MSDN site, Designing URLs for MSDN2. Someone's going to fix the mess that is MSDN I thought, excellent, maybe someone will tidy up those URLs and remove all those .aspx extensions.

Some people wonder why we bother with the .aspx extension, as it ties us to a particular technology. Others wonder why we aren't using ASP.NET when we tell them they should (we are using ASP.NET, but without the .aspx it might appear as though we're not). To make both groups happy, we chose to use the .aspx extension, but you can leave it off if you feel like it.

So sod having the ability to deliver content in the format best suited to clients, and lets use our URLs to advertise our server technology. And while we're at it lets not use language negotiation to deliver the correct language for the clients locale, but instead use a made up bracket notation which we'll append to the URL.

Conneg is great, for example, you can have one URL for your homepage and deliver HTML to Web browsers and RSS to feed readers, all totally transparently to the clients. People should be using this stuff, just a shame that a high tech. company like Microsoft isn't leading the way.

With a recent thread on the REST Discuss mailing list I thought it'd be useful to recap the issues it raised.

The idea behind content negotiation is that a resource may have multiple representations in different formats, so a client can recieve the best representation for it's abilities. So for example requesting http://example.org/theNews could return a HTML document for a HTML browser and a PDF document for a PDF viewer.

But what happens when I want to PUT a new HTML version of my resource, one that fixes a mistake in my markup, if I PUT to http://example.org/theNews then how does my server know which representation to alter? Am I replacing both representations with a single new resource, or just updating one? That's why one of the golden rules of REST as mentioned in Roy's thesis is:

The key abstraction of information in REST is a resource. Any information that can be named can be a resource: a document or image, a temporal service (e.g. "today's weather in Los Angeles"), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author's hypertext reference must fit within the definition of a resource. A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time.

So for our example URL of http://example.org/theNews that returns either a HTML or a PDF document, we also have two other resources, one for the HTML document and one for the PDF document. By convention (and because it makes conneg easier) we'll reference these at http://example.org/theNews.html and http://example.org/theNews.pdf respectively.

In this way, we get the power of being able to say to a client, "go to this URL and there'll be something useful you can consume", but we can also do our CRUD operations to manage each individual representation.

A lot of this is academic in the real world, content negotiation is one of those things which will rarely be used, however it does a good job at showing why the golden rule of "everything should be a resource" is important. If we might want to be able to access it, edit it, delete it, make it a resource, URLs are cheap.

XML is not a Data Transfer Format

2005-06-18T07:00:00Z

XML is not a Data Transfer Format

A project I've been having a small part of at work involves a Flash element talking to a bunch of heavy chunking PHP scripts thrashing a MySQL data to within an inch of it's life. The Flash comes back to the server every now and then asking for new data for that particular user (and sends some data back too).

It was decided without any thought as to why by the Flash team to format this data as XML. The reason being that they already had libraries for parsing XML into native ActionScript variables.

The problem with this is of course that XML is a markup language for formatting documents and adding meaning to data, not a data transfer or serialisation format. It's just too verbose for doing that, especially when you consider that this campaign is going out to a large userbase and every ounce of server power and bandwidth is precious.

XML is useful if the data you are transfering is actually a document and needs marking up like you would a document, but if it's not, if you're just doing data transfer then there ar far better formats which are just as easy to parse (JSON for one rings a bell). In our case, the data sent to the Flash is always the same, so a simple string of characters pushed over HTTP would have worked great, have made data parsing and generation fast, and have cut the bandwidth used by a factor of 5 at least. When you're talking about half a million requests an hour, cutting the client/server bandwidth by 5 is a big deal.

The moral of this story is, use the correct encoding format for the correct situation. XML is great, but if all you have is a hammer everything looks like a nail. Another reason for REST/HTTP's success, content type neutrality.

3 Tiered REST Architecture

2005-03-01T08:00:00Z

3 Tiered REST Architecture

I've been working on RESTifying my website management software a little since I want to learn a little bit about XUL and thought writing a XUL interface to manage my site would be a fun project. A clean RESTful HTTP interface is a great way to allow a client side application to talk to the server.

I quickly ran into a problem however. My software already placed resources at the center of the system, but as well as just storing the resource content data it also stores some metadata about that resource. So my problem was that sometimes I want to fetch the resource data and other times it's metadata.

A multitude of solutions

As with all good architectures, there are a number of ways of doing the same thing. I couldn't decide on which was the best, there are multiple ways of asking for the metadata rather than the resource content, and multiple ways of delivering it.

Requesting metadata

Use a HTTP HEAD request.
Use a new HTTP method (META for example).
Use a querystring value.
Use a custom request header.

Metadata response

In the response body.
As custom HTTP response headers.

So the question was, which of these was the best, most meaningful, more RESTful solution?

I liked the idea of sending a HTTP HEAD request and getting custom headers back with the meta data. HTTP headers make perfect sense for metadata, since they are name/value pairs too, and HEAD makes sense since we're asking for header information only and we don't want the overhead of sending all the metadata with every HTTP response.

I also liked the idea of creating a new HTTP method for getting back the metadata. This would allow the metadata to be sent in the response body thus opening up content negotiation so we could return the metadata in multiple formats if required.

The solution

The solution I decided upon... none of the above.

It suddenly dawned on me that I wasn't thinking in a RESTful way. It's very easy to fall back into a procedual way of thinking when what you really need to do is take a step back and re-factor things into more resources. So that's what I did.

On retrospect it's obvious that metadata for a resource is another resource entirely, it just so happens to relate to an existing resource. With that change implemented I suddenly have HTTP GET and PUT methods for recieving and updating resource metadata, as well as POST for appending a single piece of metadata and DELETE for removing all metadata.

So the moral of this story, think RESTfully, at all times. Hit a problem, backtrack and think about it from a resource/URL perspective. HTTP is simple, it doesn't do very much, but that's its power, don't try to shoehorn complex ideas onto HTTP requests, break them down into simpler parts that can be modelled using REST and HTTP, and you'll save yourself a lot of pain and come out of it with a more elegant RESTful system.

The 3 tiered REST architecture

Today on the list was a post by Donald Strong that describes something that I've thought for a while but never coherently enough to put into words. There are two types of transaction within a REST system.

The first uses GET and POST(p) in what Donald describes as an untrusted transaction, the kind of transaction carried out everyday by the Web browser. The client has no direct access to manipulate resources but POSTs to a URL that processes the input and changes resource states as needed. The client is dumb while the server is intelligent, making sure that resources are always left in a valid state.

The second transaction type is trusted, using GET, PUT, POST(a) and DELETE, the types of transactions that Web services and rich clients want to perform. The client has total control over the servers resources in rather the same way as a SQL database client does. In this case the server is dumb with reguard to the resources and the client is intelligent.

Read Donalds post for more in depth information and a better explaination (complete with ASCII art).

What Donald then goes on to say is that if you use these two types of transaction together you end up with your classic 3 tier software architecture. The dumb client acts as the presentation layer talking to a server via GET and POST(p). This server is the application or business layer, doing all the heavy lifting and manipulating state by talking via GET, PUT, POST(a) and DELETE to another server, the data layer.

Now this is very clever, but when I think about it, it is what I've been thinking all along. Kind of. My theory has always been that there is no harm using POST(p) on a "do some process" URL to carry out a PUT, POST(a) or DELETE on the Web browsers behalf. It's the structure that Tonic uses, all resources on the server can be manipulated by the four HTTP methods, but they can also be changed via GETting a HTML form from a "process this" URL and then POSTing data to it. All this URL does is manipulate the sites resources (files in this case) using the standard four methods (although not via HTTP since it's on the same server).

In this setup, the "process" URLs HTML representations are the user interface, the "process" URLs processing code is the business logic, and the resources we're manipulating are our data. This makes it very easy for us to bolt a new frontend onto our system in place of our HTML (for example a XUL interface, a shell script, or a GUI application) by automatically giving us a nice RESTful interface.

What are Web Services?

2005-02-22T08:00:00Z

What are Web Services?

Justin Sampson posted a great reply to a question on the REST Discuss mailing list this evening, here's a quote:

The fundamental insight I take from REST as it relates to Web services is that well-behaved Web services should feel a lot like well-behaved Web sites. A Web service is just a machine-readable Web application, and a Web application is just a dynamic Web site. The principles of the Web architecture that made it so successful apply to all three.

Stop trying to build RESTful Web services and starting just doing things the way you've been doing them for years, if it ain't broke, don't fix it.

What REST is Not

With all the coverage REST is getting, lots of systems are being touted as RESTful, so I thought it was about time I wrote not about what REST is, but what REST is not.

A messaging system

REST is not a protocol, it is an architectural style, it is a way of transfering representations of resources between two points. A resource can be anything, an application object, a database record, etc. and a representation is a way of expressing that resource in a useful way, whether that be as a HTML file, an XML document, an image, a comma separated value, etc.

A messaging system is a way of passing messages between entities, SOAP is a messaging system. Messaging systems, RPC systems, are useful, they allow our applications to talk to each other over the network by sending short messages between one another. The problem with messaging systems is that they work fine when you control both ends of the equation but when it comes to scaling them up to the size of the Internet, where anyone can send a message to anyone else, they quickly fall apart.

HTTP may look like a messaging system at first glance, but it's not, it's a representation transfer system, no messages are ever sent back and forth, only requests and responses as representations of the resources modelled on the server.

XML over HTTP

Just because you are sending XML over HTTP does not mean your Web service is RESTful, a lot of Web services that claim to be REST services are nothing more than XML over HTTP. To be RESTful requires that you embrace "the cult of REST". This includes:

Everything is a resource and every resource has a URL. If you need to manipulate it, it needs to be a resource.
Resources can be manipulated by the four CRUD methods GET, PUT, POST and DELETE.
Resources can have multiple representations in different formats, but they must all represent the same data, the data that is the resource.
Resources should link together to allow auto-discovery and provide context. Just because it's not HTML being sent doesn't mean you forget everything that's good about the Web, a Web service is just a machine readable Web site.
URLs should encode the nature of the resource and indicate any heirarchy between resources. Querystrings should be used for queries, not for returning different resources.

If you don't cover all these points then you're not being RESTful, you're just delivering XML over HTTP (which is fair enough, just don't claim to be RESTful).

Tidy URLs

URLs without large querystrings has been a trendy thing in Web circles for the last 4 or so years and it is one of the things REST says is "a good thing". But when it comes to REST it isn't just about being neat and tidy, the poritions of a URL have a meaning, it's not a random string of characters that happens to point to a file on your Web server.

URLs do not point to files on Web servers, they point to resources. The querystring portion of the URL is used to pass in query parameters to a URL, so if you have an index file listing a number of other resources a querystring can be used to limit the URLs returned in the listing (kind of like a WHERE clause in SQL). Just because your URLs are tidy, doesn't mean you're being RESTful.

REST Data Format Confusion

2005-01-15T08:00:00Z

REST Data Format Confusion

One of the problems people have with the REST architectural style is its lack of definition of a common data format, but that is actually one of the powerful features of REST based systems.

It's very attractive to programmers to have a single all in one protocol and encoding format, programmers like to think in terms of functions and parameters which is what you get with XML-RPC and SOAP. They have their own way of representing common data formats in XML, meaning you can pass values back and forth without having to write or decode any XML, the libraries do it all for you.

The problem comes when you want to pass something more than a simple data value back and forth. If you define a data format like XML-RPC or SOAP do, then you'd better make sure it covers all posibilities. How for example do I pass a binary file back from an XML-RPC request? I have to encode somehow into the XML-RPC XML response. Fun.

REST doesn't work that way, which is alien to most programmers. It works with URLs and representations, so rather than calling a function with a set of parameters, you're fetching the state of a resource. Working with REST requires you to write applications in a RESTful way, rather than just bolting HTTP based RPC onto a procedural way of thinking.

With a REST based system, my HTTP response body for that request is just the image itself, no messing around with XML, HTTP handles it all, and why not, it's proved technology, millions of images are already transfered that way everyday.

XML is great, it gives us a way of formatting and encoding arbitory data in a way we can easily get it back with standard tools. But XML is not the answer to all of our problems, it's a tool suitable for some problems, you wouldn't use a hammer to fit your broken window, don't use XML to do things other formats do better. That said, XML is great, the best thing about it is the X, the extensible, the fact it lets us define our own formats for our data, with SOAP and XML-RPC we throw away that power.

Mixing transport with data encoding is limiting and dangerous. What if HTTP had also defined its data encoding instead of HTML being a separate layer on top? Then the Web wouldn't have any graphics, no CSS or Javascript, no binary file downloads, and no XML or Web Services.

One of peoples favourite arguments about having a standard data encoding is that it means it saves you from writing extra code. Brent Simmons said recently:

[REST] doesn't say anything about how data is formatted. So you have to parse documents. With XML-RPC, assuming you have a library, you never have to parse anything.

Well, assuming you have an XML library, you never have to parse anything, that's the reason for XML, to create a standard document format you can use off the shelf libraries to parse. It doesn't matter what tags are in the XML, a decent XML library will get you the values and structures from the XML document in a nice native format for your programming language without you having to lift a finger.

Further more, REST doesn't stop you from using a standard XML data format if that's what's best for your application, although it is most likely you'll end up with a meaningless response package that only makes sense in the context of your application and is less RESTful than it could be. Instead of XML containing users with addresses, you'll have arrays full of strings, not very semantic, not very RESTful, not really what everyone had in mind when XML was invented.

So get over the common data format idea, embrace custom response formats that make sense with respect to the resource being sent, and think RESTfully when designing your applications from the ground up.

XMLHttpRequest, REST and the Rich User Experience

2004-10-23T07:00:00Z

XMLHttpRequest, REST and the Rich User Experience

Over the last few years Javascript has come of age. Gone are the inconsistent browser implementations, so not only is CSS mature enough to be used seriously, but so is Javascript. And now with the major browsers all supporting the XMLHttpRequest object, we can create a truely interactive interface to our web applications.

Using a combination of Javascript with the XMLHttpRequest object, a server side language like PHP and REST to do the talking in between, we can add dynamic elements to otherwise clumsy HTML forms.

Using the XMLHttpRequest Object

XMLHttpRequest is a Microsoft addition to IE5+ that was cloned by the Mozilla Project and now by Apple for Safari and Opera (at the time of writing support in Safari and Opera are new and incomplete, but this should improve with time). It allows Javascript to send a HTTP request, but unlike using an Image object, the XMLHttpRequest object allows you to receive and process the response, it will even parse out an XML response for you (thus the XML in XMLHttpRequest, although any response file type will do).

Creating and using the object is easy. Depending on whether the client is using IE with ActiveX or a standards compliant browser, there are different ways of instantiating the object, so this function wraps the process up into an easy to call package:

function getHTTPObject() {
    if (typeof XMLHttpRequest != 'undefined') {
        return new XMLHttpRequest();
    }
    try {
        return new ActiveXObject("Msxml2.XMLHTTP");
    } catch (e) {
        try {
            return new ActiveXObject("Microsoft.XMLHTTP");
        } catch (e) {}
    }
    return false;
}

Note: This is an updated version of my original Javascript as written by Simon Willison.

Then it's just a case of setting the objects properties and letting it go:

var http = getHTTPObject();
http.open("GET", "http://example.org/test.txt", true);
http.onreadystatechange = function() {
	if (http.readyState == 4) {
		doSomethingWith(http.responseText);
	}
}
input.http.send(null);

And Now the REST

There are a couple of XML based RPC formats available and suitable for sending information to our Javascript (XML-RPC, SOAP), however they are all large, bloated and don't use HTTP as it was designed. REST on the other hand is small, simple and a proven technology, and perfect for linking our Javascript with our serverside functions.

Since we're getting information from our server, we use our XMLHttpRequest object to issue a HTTP GET request to the URL of our serverside script.
The script extracts request data from the URL querystring and processes the request responding with a plain text formatted file, or as a simple XML document.
Our XMLHttpRequest object receives the response and parses out the data into Javascript variables. Our client has it's new data without the HTML page (the browser or the user) having to issue a separate HTTP request.

Obviously a new HTTP request has happened, but it has occured transparently in the background without our HTML page changing, allowing us to manipulate our pages DOM with the information received without having to do a full round trip to the server (and thus refreshing the page).

An Example

Time for an example. Lets say we have a HTML form that collects a users address, a usual form for an e-commerce system. Now to make our form more user friendly we'd like to allow the user to enter their postal code and get their address automatically entered into the form.

In the past this would have required a round trip to the server, reposting form data and refreshing the page (or using a popup window and some Javascript), but using the XMLHttpRequest object we can do it all transparently to the user.

So first we need our form, here's a quick XHTML file:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> <head> <title>Your details</title> <script type="text/javascript" src="http://www.peej.co.uk/address_from_postcode.js"></script> </head> <body> <form method="post" action="process.php"> <fieldset class="address"> <input type="text" name="address[postcode]" class="postcode" /> <input type="button" value="Lookup address" class="lookup" /> <textarea class="address"></textarea> </fieldset> <fieldset> <input type="submit" value="Save" /> </fieldset> </form> </body> </html>

address.html - HTML page markup

We've included our Javascript file address_from_postcode.js in the header and given our address fieldset a classname of address that we'll use to find and attach our behaviour to it's children. We do this rather than using an ID so that we can have multiple address fields on the same page.

window.onload = function() {
  var url = "http://example.org/address_from_postcode.txt?postcode=";
  var fieldsets = document.getElementsByTagName("fieldset");
  for (var foo in fieldsets) {
    if (fieldsets[foo].className == "address") {
      var textareas = fieldsets[foo].getElementsByTagName("textarea");
      for (var bar in textareas) {
        if (textareas[bar].className == "address") {
          fieldsets[foo].address = textareas[bar];
          break;
        }
      }
      if (fieldsets[foo].address) {
        var inputs = fieldsets[foo].getElementsByTagName("input");
        for (var bar in inputs) {
          if (inputs[bar].className == "lookup") {
            inputs[bar].http = getHTTPObject();
            inputs[bar].working = false;
            inputs[bar].onclick = lookupAddress;
          }
          if (inputs[bar].className == "postcode") {
            fieldsets[foo].postcode = inputs[bar];
          }
        }
      } else {
        alert("No address textarea defined within address fieldset!");
      }
    }
  }

  function lookupAddress() {
    if (!this.working) {
      var http = this.http;
      var address = this.parentNode.address;
      this.http.open("GET", url + escape(this.parentNode.postcode.value), true);
      this.http.onreadystatechange = function() {
          if (http.readyState == 4) {
            this.working = false;
            address.innerHTML = http.responseText;
          }
      }
      this.http.send(null);
      this.working = true;
    }
  }

}

address_from_postcode.js - Javascript behaviour layer

This code does the following:

Search the DOM for all fieldsets with a class of "address".
For each fieldset, get the first textarea with a class of "address".
Also get the first inputs with a class of "lookup" and "postcode".
Attach to the lookup button, a HTTP object and an on click event.
On click of the button, make sure we're not already waiting for a HTTP response and if not issue the HTTP request.
On reception of the HTTP response, set the address textarea to the recieved value.

So now to add this complex behaviour to our HTML forms, all we need to do is include the Javascript file and add the class names to the relevant elements.

Of course this is just a simple example, I've been using it to dynamically update dropdowns and autocomplete fields from existing data, the number of applications of this type of rich forms are unlimited.

HTML Overlays

2004-09-15T02:34:00Z

HTML Overlays

XUL overlays is a great feature of the excellent Mozilla application platform, and now Laurent Jouanneau and Daniel Glazman have created a version for HTML using Javascript to do the legwork.

The idea is to remove navigational elements and other pieces of HTML document that are repeated across multiple pages, and place them in separately downloaded files that can be cached and combined with the rest of the pages content by the browser. A link tag is used to specify which overlays to use, and a Javascript script is used to get the overlays and patch them into the document. The idea is good and the implementation is solid, being based upon the proven technology of XUL overlays.

Backwards compatibility

I have a problem with the backwards compatibilty of the implementation. Without Javascript, the browser doesn't load the overlays and you lose your navigational elements, not only bad for those with JS turned off but also for Google.

How about instead of using XML we use HTML, the id's in the overlay file matching up with the id's in our host document, and the rest of the HTML being disgarded by the Javascript. That way we can use another mechanism to include our overlays if Javascript isn't present (iframes, object tags, a simple hyperlink) and they are human readable in everyones browsers.

This may remove some of the flexibility, and it doesn't answer all of the questions, but personally I think this would be a saner way to introduce HTML overlays.

The problem is that we still have to parse out our HTML. The useful thing about using XML is that XMLHttpRequest will parse it and return you a nice DOM tree. We could send our HTML overlay as XML to XMLHttpRequests and as HTML for regular requests, but this makes things messy and adds some unwanted complexity. Mozilla and the latest version of Opera provide us with a DOMParser object we can use to turn our HTML string into a DOM tree, but that doesn't help IE users.

A better solution

The best solution I can think of is to add a new <overlay> element to our HTML overlays to make them easy to extract with a regular expression. Browsers displaying the HTML would ignore it and our Javascript to grab the overlays from the HTML would be quick and simple.

<overlay id="myOverlay"> <p>This is content in an overlay.</p> </overlay>

I've put up a demo of my version of HTML overlays in action.

REST

2002-05-02T07:00:00Z

Representational State Transfer (REST)

REST (REpresentational State Transfer) is a phrase coined by Roy Fielding in his dissertation Architectural Styles and the Design of Network-based Software Architectures. It is an attempt to describe the undocumented architectural design principles behind the Web. In fact you are using the World's largest and most popular REST system right now, yes, the World Wide Web.

"The World Wide Web architecture has evolved into a novel architectural style that I call 'representational state transfer.' Using elements of the client/server, pipe-and-filter, and distributed objects paradigms, this style optimises the network transfer of representations of a resource. A Web-based application can be viewed as a dynamic graph of state representations (pages) and the potential transitions (links) between states. The result is an architecture that separates server implementation from the client's perception of resources, scales well with large numbers of clients, enables transfer of data in streams of unlimited size and type, supports intermediaries (proxies and gateways) as data transformation and caching components, and concentrates the application state within the user agent components." - Roy Fielding

With all the talk of Web Services by the big software companies of this World, REST has (or will, maybe) come back into the limelight as an HTTP RPC protocol (like SOAP and XML-RPC).

The Web As We Know It

What REST basically says is "the Web is cool, HTTP is cool, and it already does everything we want and it works! Why do we need anything more?"

The Web as defined by Tim Berners-Lee consists of three elements:

URI (Uniform Resource Identifier) - The way of uniquely identifying resources on the network.
HTTP (HyperText Transfer Protocol) - The protocol used to request a resource from a URI and respond to requests.
HTML (HyperText Markup Language) - The content format of resources to be returned.

The interesting aspect of this mixture is the HTTP.

We all use it everyday, but why is it so interesting. Well HTTP is a very cleverly designed protocol, it defines a small global set of verbs (the HTTP Methods: GET, POST, PUT, DELETE, etc) and applies them to a potentially infinite set of nouns (URIs). Most of the time when using the Web, you use the HTTP GET method to retrieve documents, if you're submitting some form data you may use the POST method, there are however a whole handful of other useful methods defined but rarely used on the Web but perfect for REST based Web Services.

HTTPs power comes from its simplicity and extensibility. The ability to distribute any payload with headers, using predefined methods and its built-in support for URIs and resources, makes it really special. URIs are the defining characteristic of the Web, the mojo that makes it work and scale. HTTP as a protocol keeps them front and center by defining all methods as operations on URI-addressed resources.

Tell Me The REST

Stupid acronym puns aside, the idea of REST is to take what we all take for granted in the Web (ie. HTTP and URIs) and use it with other payloads (other than HTML, images, etc.) and other HTTP Methods than the usual GET and POST. REST defines identifiable resources (URIs), and methods for accessing and manipulating the state of those resources (HTTP Methods).

HTTP messaging formats like SOAP and XML-RPC are supposedly the next big thing in the ever-developing world of the Web and the Internet. The idea of being able to link applications together using a standard open RPC format over HTTP connections shows great promise for distributed computing and interoperability, but the major problem with them is they purposefully break the HTTP spec by adding another layer of abstraction onto HTTP rather than using the protocol as it was designed. REST says this extra layer is an unnecessary layer of complexity.

SOAP and XML-RPC are both designed to operate from a single URI with methods being invoked from within the request payload. This fails to use URIs as they were designed to be used. A URI is a unique resource on the network and as such, each method in your RPC call should use a unique URI.

Using REST this is the case, and depending on your request method, different actions can be invoked. REST uses HTTP as it was designed, if you want to get some data you use a HTTP GET request, if you want to delete a record from a database you use a HTTP DELETE request, etc.

Advantages of REST

It uses well documented, well established, well used technology and methodology.
It's already here today; in fact it's been here for the last 12 years!
Resource centric rather than method centric.
Given a URI anyone already knows how to access it.
It's not another protocol on top of another protocol on top of another protocol on top of...
The response payload can be of any format (some may call this a disadvantage, however the Web copes with it, it's just a case of defining the application grammar).
Uses the inherent HTTP security model, certain methods to certain URIs can easily be restricted by firewall configuration, unlike other XML over HTTP messaging formats.
REST makes sense, use what we already have; it is the next logical extension of the web.

Differences Between REST and RPC

REST is not RPC, RPC says, "define some methods that do something" whereas REST says, "define some resources and they will have these methods".

It is a subtle but vital difference, when given a URI anyone knows they can interact with it via the predefined set of methods and receive standard HTTP responses in return. So given http://www.peej.co.uk/ I know I can issue a GET on it and receive something meaningful back. I may then try a PUT on it to change it and receive a meaningful HTTP error code since I'm not authorised to meddle with it.

Conclusion

The Web of yesterday was all about transferring documents over the network to a users eyes, the Web of today adds the delivering of Web based services (ala e-mail, recruitment, etc.) to the mix, the Web of tomorrow will extend this information and service source from just being a user experience to being a generic client system.

SOAP, XML-RPC, and other XML messaging over HTTP protocols may be the way this new era of the Web is created, the big (and not so big) corporates certainly think so, but REST shows that the infrastructure to do these types of machine to machine communications is already in place and has been since HTTP, the URI, and the Web was invented.

References

XHTML

2002-02-14T08:00:00Z

XHTML

What is it, why is it, when is it, and how to use it.

Warning: This article was written in 2002, 3 years after XHTML became a standard. I no longer believe that XHTML is a good choice for authoring on the Web. For more information please read Ian Hickson's "Sending XHTML as text/html Considered Harmful" for more information.

Case Of The X

HTML4 as been around for a while, it is the basis of the modern web, however it is far from perfect and far from supported. Most people know about browser incompatibilities, but what most people do not realise is that HTML has a standard grammar; it is just that browsers do not comply.

Actually it is not the browser author's fault. To explain this we need first to look at a little history.

HTML History Lesson

SGML (Standard Generalized Markup Language) is a grammar used to mark-up documents for formatting, it was invented in the 1980's as a standard way of defining formats on electronic documents.

HTML is a SGML application. That is, it is a set of mark-up that conforms to the SGML grammar and tells people (or computers) how to format a document. SGML says an element must look like this, an attribute like this, etc.

As HTML evolved and the web became more popular, the SGML roots of HTML started to slip and browsers started allowing HTML authors get away with not conforming to the HTML standard. This was good for developers as it gives them an easier ride, however over time it adds inconsistencies between browsers and leaves us in the current mess of browser conformity.

How This Relates To XML

You've probably all heard of XML. It is a grammar like SGML, but it has been specifically designed for the Internet. It too defines how to structure documents in a rigid way, which makes it easy for programs (called XML parsers) to interpret XML documents. XHTML is an implementation of HTML as an XML application, in similar ways that HTML is an SGML application.

The ultimate goal of XHTML is to allow browsers to do away with their HTML parsers and basically become XML (and XSL and CSS) parsers. This has two benefits:

Browsers based on an XML parser can parser and display any XML document which it has a stylesheet for, whether it be XHTML, WML, or any other XML application (even one you invented yourself, thus eXtensible).

It removes incompatibilities, as the formatting of an XML document is defined in a stylesheet and an XML document is strongly typed (an illegal XML document will generate an XML parser error), so all clients will display documents the same.

But Browsers Only Support HTML 4

Wrong! XHTML is an implementation of HTML 4 in XML, thus it is HTML 4 compatible, it will be displayed fine in browsers which support HTML 4. All that XHTML really does is:

Cut out the crap
Make HTML a valid XML document
Force authors to create valid HTML documents

So XHTML Is The Standard?

Yes, XHTML has been a W3C (the web standards people) recommendation for over a year (since Jan 2000 to be exact). You can (and should) use it right now.

"But why should I?" I hear you cry:

Do you hate browser incompatibility? Then use XHTML.
Are you fed up of having to check your HTML in hundreds of browsers? Then use XHTML.
Do you want to help push for a standardised XML based web? Then use XHTML.
Do you want your documents to be future proof? The use XHTML.
Do you take pride in your HTML? Then use XHTML.

You probably think that you can write proper HTML4, I did, but if you haven't read and understood the HTML4 spec then you can guarentee that your documents are not standard, try running your HTML through the HTML validator, you will be surprised, I know I was.

So How Do I Write XHTML?

The only way to really know is to read the XHTML spec, however it isn't the easiest document in the World to read, so here are the basic rules:

Documents must be sent with the correct content type.

Web servers send HTML files with a mimetype definition of text/html, this tells the browser what type of document to expect from a given request. XHTML is not HTML and should not be sent with the same mimetype, XHTML should be sent with the mimetype of application/xhtml+xml.

However it's not quite that simple as Internet Explorer does not understand the application/xhtml+xml mimetype and interprets it as text/xml displaying your XHTML file as an XML file. Luckily the W3C allow us to send HTML conforming XHTML documents with the text/html mimetype which IE will understand and render our document correctly.

Best practice is to get your server to check the types of files the browser will except and send the content as application/xhtml+xml to browsers that support it, and as text/html to those that don't.

Documents must start with a DOCTYPE definition.

This rule exists for HTML4 too, but browsers let you get away with missing it out, so most people do. The first line of any XHTML document should look like this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

This defines the document as XHTML1 and tells the browser which DTD (Document Type Definition) to use. There are three DTDs for XHTML, Transitional is the best to use if you are moving from HTML4 and want maximum backwards compatibility.

All tags must be lower case.

Yes, I don't care if you've used upper case tags before, XML is case sensitive so <B> is different to <b>, the XHTML DTDs define all tags in lower case only, so for your tags to work they must be lower case.

All attributes must be in quotes.

That is:

<a href=http://www.example.com>Example</a>

is not valid XHTML. You must put quotes around the attribute body such as:

<a href="http://www.peej.co.uk/http://www.example.com">Example</a>

All tags must be closed.

That is if you open a tag like <p> then you most also close it with </p>, makes sense really.

Empty tags must be closed.

This is probably the most alien concept from HTML 4. A tag is empty if it does not contain text between itself and a closing tag, however in a valid XML document when you open a tag you must also close it. So:

<img src="http://www.peej.co.uk/http://www.example.com/image.jpg">

is not valid XHTML. You must close the tag like this:

This is however annoying, so XML defines a shorthand way of closing empty tags like this:

<img src="http://www.peej.co.uk/http://www.example.com/image.jpg" />

Even the humble line break tag must be closed, ie:

<br />

You must nest tags correctly.

So you cannot open two tags, and then close them in a different order, eg:

<b><i>This is bold and italic</b></i>

is not valid, you must nest them correctly like:

<b><i>This is bold and italic</i></b>

Tag context

Last but not least and the hardest concept to get hold of, you must use tags in the correct context. This is almost impossible to get right without practice or learning the XHTML spec inside out, so the best way to check if you have done something wrong is to check with the W3C validator.

Quick Review

XHTML is HTML4 compatible
Don't forget the DOCTYPE
Don't forget to close all tags
Use W3Cs validator to check your XHTML (or if I've failed to convert you at least your HTML).

Now go forth and prepare your web sites for XML.

XHTML Links & References

Cross site scripting

2001-09-27T07:00:00Z

What Is Cross Site Scripting?

Cross site scripting is the name given to web site vulnerabilities arising from the embedding of malicious HTML tags into a HTML document which is generated dynamically on the server. By entering HTML tags into an input field of a web site (either via a HTTP POST or GET i.e. as a form post or through the query string), these tags can then be included in a HTML page served from the web site.

Web pages contain both text and HTML markup that is generated by the server and interpreted by the client browser. Servers that generate static pages have full control over how the client will interpret the pages sent by the server. However, servers that generate dynamic pages do not have complete control over how their output is interpreted by the client.

These attacks do not affect the security of the server itself, as they only create malicious tags or code that is sent to the client. However, these tags can be used to annoy users with large graphics and/or sound files, change the expected look of a web page, change the functionality of the web page and thus capture sensitive user information, etc.

The heart of the issue comes back to the age-old knowledge that you cannot trust the client for anything.

Script Tag Include

Sites that host discussion groups with web interfaces have long guarded against a vulnerability where one client embeds malicious HTML tags in a message intended for another client. For example, an attacker might post a message like:

Hello message board. This is a message. <script>malicious code</script> This is the end of my message.

When a client with scripts enabled in their browser reads this message, the malicious code may be executed unexpectedly. Scripting tags that can be embedded in this way include <script>, <object>, <applet>, and <embed>.

This type of vulnerability is easy to stop by simply filtering/encoding any user input for the offending tags, however as we will learn, the issue is a little more complicated that this.

Malicious Link

Many Internet web sites overlook the possibility that a client may send malicious data intended to be used only by itself. This is an easy mistake to make. After all, why would a user enter malicious code that only the user will see?

However, this situation may occur when the client relies on an untrustworthy source of information when submitting a request. For example, an attacker may construct a malicious link such as:

<a href="http://www.peej.co.uk/http://example.com/comment.cgi?mycomment=<script>malicious code</script>">Click here</a>

When an unsuspecting user clicks on this link, the URL sent to example.com includes the malicious code. If the web server sends a page back to the user including the value of mycomment, the malicious code may be executed unexpectedly on the client.

Inline Frame / Script Include

The inline frame tag <iframe> added to Microsoft IE in version 4, allows the inclusion of a remote document within a parent document:

<iframe src="http://www.peej.co.uk/http://exampl.com/malicious.html">

The script tag can also be used to include a remote script block into the parent document:

<script src="http://www.peej.co.uk/http://example.com/malicious.js">

Form Tag Include

In addition to scripting tags, other HTML tags such as the <form> tag have the potential to be abused by an attacker. For example, by embedding malicious <form> tags at the right place, an intruder can trick users into revealing sensitive information by modifying the behavior of an existing form.

How To Limit The Effects Of Cross Site Scripting

So far the issue seems pretty simple. Filter an input to make sure tags that could be used for a malicious reason. However the problem is a little deeper than that, as I will explain below.

Any data inserted into an output stream originating from a server is presented as originating from that server, even if it contains output that originates from a user. Web developers must evaluate whether their sites will send untrusted data as part of an output stream, or if they will validate any data before it is included.

Untrusted input can come from, but is not limited to,

URL parameters
Form elements
Cookies
Databases queries

A combination of steps must be taken to mitigate this vulnerability. These steps include:

Explicitly setting the character set encoding for each page generated by the web server
Identifying special characters
Encoding dynamic output elements
Filtering specific characters in dynamic elements
Examine cookies

Explicitly Setting the Character Encoding

If the web server does not specify which character encoding is in use, the client can not tell which characters are special. Web pages with unspecified character encoding work most of the time because most character sets assign the same characters to byte values below 128. The question comes with the characters above 128, which of these are special?

Some 16-bit character-encoding schemes have additional multi-byte representations for special characters such as "<". Some browsers recognize this alternative encoding and act on it. This is correct behaviour, but it makes attacks using malicious scripts much harder to prevent. The server simply doesn't know which byte sequences represent the special characters.

Web servers should set the character set, then make sure that the data they insert is free from byte sequences that are special in the specified encoding. For example:


<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />

<title>Using ISO-8859-1 Charset</title>

</head>

....

The <meta> tag in the <head> section of this sample HTML forces the page to use the ISO-8859-1 character set encoding.

Identifying the Special Characters

The next two steps, encoding and filtering, first require an understanding of "special characters". The HTML specification determines which characters are "special", because they have an effect on how the page is displayed. However, many web browsers try to correct common errors in HTML. As a result, they sometimes treat characters as special when, according to the specification, they are not.

In body content:

"<" is special because it introduces a tag.
"&" is special because it introduces a character entity.
">" is special because some browsers treat it as special, on the assumption that the author of the page really meant to put in an opening "<", but omitted it in error.

In attributes:

In attribute values enclosed with double quotes, the double quotes are special because they mark the end of the attribute value.
In attribute values enclosed with single quotes, the single quotes are special because they mark the end of the attribute value.
Attribute values without any quotes make the white-space characters such as space and tab special.
"&" is special when used in conjunction with some attributes because it introduces a character entity.

In URLs:

Space, tab, and new line are special because they mark the end of the URL.
"&" is special because it introduces a character entity or separates querystring parameters.
Non-ASCII characters (that is, everything above 128 in the ISO-8859-1 encoding) are not allowed in URLs, so they are all special here.
The "%" must be filtered from input anywhere parameters encoded with HTTP escape sequences are decoded by server-side code.

Encoding Dynamic Output Elements

Each character in the ISO-8859-1 specification can be encoded using its numeric entry value.

The following example uses the copyright mark in an HTML document:

The copyright character is 169 and using the &# syntax allows the author to insert encoded characters that will be interpreted by the browser.

Encoding untrusted data has benefits over filtering untrusted data, including the preservation of visual appearance in the browser. This is important when special characters are considered acceptable.

Filtering Dynamic Content

Unfortunately, it is unclear whether there are any other characters or character combinations that can be used to expose other vulnerabilities. The recommended method is to select the set of characters that is known to be safe rather than excluding the set of characters that might be bad.

For example, a form element that is expecting a person's age can be limited to only accept numeric characters between a certain range (1 to 99 would probably do). There is no reason for this age element to accept any letters or other special characters. Using this positive approach of selecting the characters that are acceptable will help to reduce the ability to exploit other yet unknown vulnerabilities.

The filtering process can be done as part of the data input process, the data output process, or both. I would recommend filtering the data during the input process as this has the benefits of only needing to be done once, and (if the input is going to be stored in a database) can be included as part of your database input validation (to continue the example above, an age would be expected to be held in a numeric database field, trying to insert a string into this field would cause an input error anyway). The only disadvantage of filtering on input is that you must make sure that all input is filtered at every entry point to the storage device.

Examine Cookies

Cookies can often be overlooked. Since cookies are just text documents held on the client's machine, it is easy for a client to alter their cookie to allow inclusion of malicious content or send bogus information in their HTTP requests. Like anything else from the client, cookies should not be trusted and the information within them should be verified before being used.

So Why Is It Called "Cross Site Scripting"?

Since the issue isn't just about scripting, and there isn't necessarily anything cross site about it, the name Cross Site Scripting is a little ambiguous. It was coined earlier on when the problem was less understood, and has stuck, it's as simple as that.

SQL Injection

2001-08-01T07:00:00Z

SQL Injection

The checking of user input is often overlooked by web developers but is very important to close serious vulnerabilities in web systems. I am currently working on a project for a client that involves rewriting and designing their existing web application. This process started with looking at their existing system.

Within 10 minutes of investigation I was able to drop their entire database just through their web sites log in box. I was able to do this for the following reasons:

The log in form textbox had no maximum input length set, allowing me to enter a large input string including my concoction of SQL. This hole is not required as I could have written a spoof form without a maximum input length, but it made my life easier non-the-less.
There was no length checking for the username field. Their username database field is 10 chars long, and yet the server code did not check to make sure my input was 10 or fewer characters.
There was no invalid character stripping or conversion. By allowing me to place a single quote in the input text, I was able to fool their system into allowing me to run arbitrary SQL code on their database server.
The database user had full permissions on the database, including delete and drop permission.

By submitting code similar to what I've written below, I was able to execute whatever SQL I wanted on their database just through my web browser:

'; DROP DATABASE dbname; SELECT * FROM tbluser WHERE username = '

When this input is concatenated with the rest of the database query string, the following SQL query is created and executed on the database:

SELECT * FROM tbluser WHERE username = ''; DROP DATABASE dbname; SELECT * FROM tbluser WHERE username = '';

The first two characters of the user input terminate the first select statement, the next statement does the malicious drop of the database, and the last select statement is a dummy statement to use the original closing quote and make sure the whole query string is valid.

Of course, the attacker needs to know the name of the database or any other database structures that they need to reference, but these are often easy to guess by trial and error or by making the database throw up errors if they are not caught by the web application.

I hope this little demonstration of web app security is useful to web developers and makes clear the point of why checking user input from web forms is important.

This type of web application attack has been given the name of an "SQL Injection" attack since I originally wrote this article, so the title of the article has been changed to reflect this fact even though the validation of user input effects more than just SQL injection attacks.

URL	http://del.icio.us/api/[username]/tags/
Method	GET
Querystring	start=	The number of the first tag to return
	end=	The number of the last tag to return
Returns	200 OK & XML (delicious/tags+xml)
	401 Unauthorized
	404 Not Found

URL	http://del.icio.us/api/[username]/tags/[name]
Method	GET
Querystring	start=	The number of the first bookmark for this tag to return
	end=	The number of the last bookmark for this tag to return
Returns	200 OK & XML (delicious/tag+xml)
	401 Unauthorized
	404 Not Found

Paul James

E-mail Sign On

E-mail Sign On

E-mail as login credentials

Benefits

Disadvantages

Conclusion

Introducing the RMR Web Architecture

Introducing the RMR Web Architecture

Model-View-Controller

Resource-Method-Representation

Resource

Method

Representation

Putting it into Practice

Front Controller

Resource Class

Routing

Conclusion

Virtual World

Learning From the Web; Building an Open Virtual World

The Walled Garden

The Lessons of the Web

Application protocol

Content description language

Code on demand

Hypermedia

Presence and chat

Hypermedia as the Engine of Application State

Hypermedia as the Engine of Application State

Using hypermedia

Web services as Web sites

Is Bit Torrent the future of the Web?

Is Bit Torrent the future of the Web?

Using Bit Torrent to power the Web

Living Without Sessions

Living Without Sessions

Building a shopping cart with sessions

Cart on the client

Cart on the server

Ain't that just a session?

Conclusion

HTTP Caching

HTTP Caching

The reasoning

Expiration

Expires header

Cache-Control header

Validation

Last-Modified header

Etag header

Using caching

Other caches

HTTP Authentication with HTML Forms

HTTP Authentication with HTML Forms

With the power of Javascript we can

An example

Problems

HTTP Digest Auth

Conclusion

A RESTful Web service, an example

A RESTful Web service, an example

Getting Bookmarks

Get all bookmarks

Get info on a specific bookmark

Get count of posts per date

Get last update time for the user

Modifying Bookmarks

Add a bookmark to Delicious

Modify a bookmark

Delete a bookmark from Delicious

Tags

Get all tags

Get info on a specific tag

Rename a tag

Finishing the API

An example of using the API

GET http://del.icio.us/api/

GET http://del.icio.us/api/peej/tags/

GET http://del.icio.us/api/peej/tags/test