Peer Pressure

Indexed TiddlyWeb Filters

Thu, 05 Nov 2009 14:25:08 +0000

One of the core features of TiddlyWeb is its ability to use filters to constrain the tiddlers that are selected from any collection of tiddlers (bag, recipe, search results, etc.). In the early design discussions that led to the creation of TiddlyWeb filters were conceived as the mechanism a recipe would use to choose only some tiddlers from a bag. Bags are containers for tiddlers that have been grouped together for some reason. Recipes are lists of bags that lead to the creation of some useful set of tiddlers. When using TiddlyWeb and TiddlyWiki together, a recipe can create a particular application or vertical of TiddlyWiki. In that context one would use a filter to select some tiddler tagged “systemConfig” from one bag, others tagged “faq” from another, and others with modifier “cdent” from another.

When TiddlyWeb is used generally as a data store, filters are just as useful. When requesting tiddlers from any bag you can select and sort by attributes, and limit the number of tiddlers. Application developers can also make new filters as plugins (see mselect for an example).

This is all quite grand and useful but recent explorations by Mike Mahemoff while developing Scrumptious have revealed some (fairly expected) problems. Imagine a bag called “comments” containing some 10,000 or more tiddlers which are comments on URLs. Now imagine you’d like to get those tiddlers which have the field ‘url’ set to http://cdent.tumblr.com/.

The naive way to do this is to look at each one of those tiddlers, one at a time, and say “Hello tiddler, have you got your url field set to http://cdent.tumblr.com/? Oh you do? Well then I’ll have you, thanks!” This is time consuming and resource intensive.

It is also the way TiddlyWeb does filters. It’s like this for a few different reasons:

The original design imagined many bags, not few bags with large numbers of tiddlers.
It preserves the strict separation between the filter system and the storage system, meaning that the storage system can be simple and very adaptable: any filter can work with any store.
It makes the filter system fairly transparent: There’s no magic going on; a filter works by looking at tiddlers and making a decision.
It makes the filter system easy to extend: The contract between a filter and the rest of the system is “look at some tiddlers, return some tiddlers”. What the filter does when looking is arbitrary.

The 0.9.74 release of TiddlyWeb includes support for querying an index when doing select style filters. The support only kicks in in special circumstances (explained below) but when it does it can speed some filters up immensely. In a test (using profile/list_tiddlers.py) of 10000 tiddlers a filter that took 13.96 seconds without an index took .30 seconds with an index.

That’s great news. Here’s the bad news: In at least this initial implementation the prerequisites for the indexing system to be used (and be useful) are quite complex. Here’s the list:

The filter being performed must be a select filter (sort and limit do not).
If there are multiple filters being performed on the collection of tiddlers, the select filter must be first in the stack and only that one filter will use the index.
The collection of tiddlers must be what’s called a “natural” bag. That is, the thing being filtered is a bag that exists in the store and the entire contents of the bag is what’s desired to be filtered.
That bag should be skinny, meaning when it was loaded from the store its tiddlers contents was not determined. If you are processing recipes or working from URLs this is handled for you by the code. It’s only a concern if you are writing your own handlers.
tiddlyweb.config[‘indexer’] is set to a string which is the name of a module which provides an index_query(environ, **kwargs) callable that returns tiddlers which have been loaded from the store. tiddlywebplugins.whoosher has been updated to provide this and the next item.
Something must provide an index for index_query to query. That index needs to be kept up to date as tiddlers are changed. tiddlywebplugins.whoosher has this functionality. The sql and mappingsql have the guts to make the functionality possible, but have not yet been extended with an index_query method.

The TiddlyWeb at http://tiddlyweb.peermore.com/ has been updated to use the filter indexes. The relevant changes to the tiddlywebconfig.py are:

Add tiddlywebplugins.whoosher to twanager_plugins.
Add tiddlywebplugins.whoosher to system_plugins.
Set indexer to tiddlywebplugins.whoosher.

Then twanager wreindex is run to build the initial Whoosh index.

You can get some sense of the effects of the index by comparing the following to URLs (this is not an exact test, but gives the sense of things):

The docs bag only has a couple hundred tiddlers in it and memcached is involved, so the effect is not huge, but if you imagine bags orders of magnitude bigger…

Astute observers will note that what’s going on here is not particularly innovative: It’s simply the addition of an index to a query system. One can imagine future improvements a la SQL query optimization, wherein the order of the filters are adjusted to allow most effective use of the index, and the index is used for more queries than just those against “natural” bags. Constant evolution, constantly building on the shoulders of that which has come before.

For more details, a browse of the code will be instructive. tiddlyweb.control:filter_tiddlers_from_bag is a good entry point.

TiddlyWeb Value Proposition

Tue, 27 Oct 2009 21:39:31 +0000

A common question associated with TiddlyWeb is “What can I do with it?” or “Why would I want to use it instead of Django or something?” The answer is not simple.

Because TiddlyWeb has tools for storing data and presenting that data in various forms over HTTP, it has some of the core features of a web application framework. However, since it (intentionally) is also missing some of the core features of a web application framework, a fair piece of cognitive dissonance can happen when the tool is approached as a web application framework.

So it seems best to stay away from saying that TiddlyWeb is a web application framework, so as to manage expectations. The question remains, “What is it?”

One pathway to answering that question is to create tools with it and see what the commonalities are between those tools. A derivative of this approach is to create a killer app with TiddlyWeb: a something which demonstrates all the salient features and how to use them, both as a user and as a developer.

This approach seems to falter: The concepts in TiddlyWeb are sufficiently strange (I won’t say unique, because they aren’t that, they are common concepts, with their own spin) that in order to create a tool that uses them well you first need to either understand them in the abstract, or see some examples. Catch 22, especially if you are an examples oriented person.

FND and I tend to converse on this topic about daily, trying to come up with different strategies for answering it. Today I suggested that instead of creating a new application, from scratch, that is cool and interesting because it uses TiddlyWeb, it might be better to find some application that could be made better through the addition or inclusion of TiddlyWeb.

It turns out this process has already started and by analyzing that process we might understand TiddlyWeb better.

The process has started with TiddlyWiki. TiddlyWiki is made better for a particular set of use cases by taking advantage of TiddlyWeb features. The fundamental experience of TiddlyWiki does not change: you’ve still got a wiki in an HTML file, with these things called tiddlers, some of which are special plugins that change behaviors.

What TiddlyWeb adds when used with TiddlyWiki is:

Centralization of content storage, storing at the level of the tiddler, with revisions, protection against editing conflicts and access control to read, write, create and delete; accessible from multiple locations on the network.
A (hopefully) straightforward system of managing access control of tiddlers and making other kinds of groupings of tiddlers through the use of bags. Bags are collections of uniquely named Tiddlers.
A system for composing functional collections of tiddlers selected from multiple bags using recipes. Recipes are list of bags paired with filters. Filters are rules which limit the tiddlers pulled from a bag when processing the recipe.
Tools for viewing and manipulating tiddlers in different forms than those used in the wiki (e.g. JSON, Atom).

These four things together provide a multi-user environment for TiddlyWiki that allows multiple custom views of the same or similar content, depending on some piece of context. Different recipes can provide a different view on the same stuff, providing a different look and feel, different security handling, etc.

TiddlyWeb has many other features, but the above four are probably the key aspects which define its special sauce. When I evaluate a nascent TiddlyWeb tool for its tiddly-fitness the evaluation is done based on the extent to which the system takes advantage of and understands the above features.

These ideas should allow us to create some mental heuristics to use when approaching a TiddlyWeb project. If we have an existing project or use case that we intuit will be helped by using TiddlyWeb we can ask ourselves: 1) What aspects of this project will benefit from bags, recipes and filters? 2) How can we take advantage of different representations?

If, for some reason, we are planning to make a project from scratch, and have decided to use TiddlyWeb prior to really evaluating our use cases, then we can ask ourselves: 1) How can we best structure our data and UI to use recipes, bags and filters to provide flexible user experiences? 2) What representations can we present to allow other people to provide flexible user experiences?

Python Namespace Packages for TiddlyWeb

Sun, 18 Oct 2009 13:00:00 +0100

In Friday’s TiddlyWeb Dev/Deploy Workshop posting I said that mature plugins need to be packaged so they can be indexed by PyPI and installed via easy_install and pip. At base this is relatively straightforward, put some stuff in a directory, make a setup.py, register the package, package up a source distribution, tell PyPI about it.

This is okay if you have a good name for your package, but what about packages, like plugins for TiddlyWeb, which have names which are only meaningful in their use context? You can’t just use the name of the module, otherwise you end up with namespace collisions, trouble finding stuff, and associated beasts of chaos.

Thankfully distutils, setuptools, pip, etc conspire to support a notion called namespace_packages which can solve this issue. Unfortunately using the feature is not exceptionally well documented. I found getting started a bit frustrating, until I sort of cracked the nut. Here’s some info for reference.

First some prerequisites:

You need setuptools
You need a username and password on PyPi.
Some understanding of how to make a python package distribution. Here’s one tutorial: Use setup.py to Deploy Your Python App with Style.
Some understanding of Python packages.

That tutorial includes a section about using namespace_packages under the heading “Multiple Distributions, One (Virtual) Package”. That “Virtual” is key: if you wish to use namespace packages there much be no real distribution which occupies the namespace. In the example of tiddlywebplugins, you can have tiddlywebplugins.static and tiddlywebplugins.utils distributions which are members of the tiddlywebplugins namespace, but you must not have a tiddlywebplugins distribution. If you do the packages which are supposed to occupy the virtual namespace will not be found. The upshot of this is that if you already have a package out there using a name that you want to use as a namespace, you will need to rename the existing package (this is why the old tiddlywebplugins package is now tiddlywebplugins.utils

Packaging a Plugin

So say you have a TiddlyWeb plugin called foobar.py sitting in a directory somewhere. You’ve determined that it is a happy little plugin and the world would benefit if it could be installed easily. You’ve heard of the tiddlywebplugins namespace and you’d like to join the party. Here’s what you do.

In that directory make a tiddlywebplugins directory.
Edit tiddlywebplugins/__init__.py to include just this line: __import__("pkg_resources").declare_namespace(__name__)
Move foobar.py into the tiddlywebplugins directory.
Create a setup.py (in the original directory) that includes at least:

     from setuptools import setup, find_packages

     setup(
         version = '0.1',
         namespace_packages = ['tiddlywebplugins'],
         name = 'tiddlywebplugins.foobar',
         description = 'A TiddlyWeb plugin for foobaring the fritz.',
         install_requires = ['setuptools', 'tiddlyweb'],
         )

Do not import from the tiddlywebplugins package in setup.py. This will make installs struggle or fail later.
Learn enough about distribution to register the package and upload it. The links above point to enough documentation to figure that part out. If you can’t be bothered to read that documentation then you shouldn’t be distributing packages. We wouldn’t want you aoling all over PyPi.
When you want to use your foobar plugin in a TiddlyWeb instance or application refer to it as tiddlywebplugins.foobar.

TiddlyWeb Dev/Deploy Workshop

Fri, 16 Oct 2009 13:00:00 +0100

Yesterday I gathered with the Osmosoft gang to discuss developing and deploying applications that are built on top of TiddlyWeb. Since I mostly develop TiddlyWeb itself and not things built upon it, I tend to be fairly distanced from the issues so getting together and having a chat and messing about with some code was a good thing.

Mike has written up his summary of the things to do next. I took some notes as well, which I’ll attempt to summarize here.

To get a sense of the issues we went around the room and people reported on how they’ve been managing development and deployment and what’s working well and what’s not. There were some but not many common threads.

One common issue is that most of the people have a tendency to do a code a bit, reload a web page cycle of development and test. When using twanager server this does not work well as the server needs to be restarted with each code change. Restarting CherryPy’s wsgiserver is less easy than some desire because it can be some time before the main thread will come round to hearing an interrupt signal. The initial solution to this problem, the reloader plugin, helps but gets a bit weird when syntax errors exist in the watched Python files (which happens a lot if you have a esc:w trigger finger): The entire server will crash out and you have to start the process again by hand. This mostly makes sense from a logical standpoint but is not that helpful.

One proposed solution to this problem is to host TiddlyWeb under a simple CGI setting. This will reload and recompile the code with every request. This can be slow for production settings but plenty fast for development. The TiddlyWeb source package includes an index.cgi script for this. Ben Gillies has written some docs on how to use it.

Of course to use the CGI option you have to have some kind of web server around. Many systems come with Apache already on them or easy to install. If you prefer not to run Apache (like me) another option is Spawning a fast multi-process, multi-threaded, Python web server. There’s a TiddlyWeb “factory” for Spawning called spawner.

Paul wisely pointed out that much of this reload fiddling about can be avoided by writing and testing Python modules that perform most of the activity for you application, minus the web handling, and then using these modules in small layers of Python code that integrate with TiddlyWeb.

There was a fair amount of resistance in the group to formal and rigorous testing, which I think is extremely unfortunate, but Simon rightly points out that lots of people are like that, and TiddlyWeb is going to limit its audience a great deal if it requires developers to be totally test driven.

Another point of difficulty and contention is lack of familiarity with Python, Unix and TCP fundamentals. I’m not sure how to respond to that sort of thing other than to say that if you want to develop networked applications in Python, on Unix, then it’s probably pretty good to be familiar with such stuff.

By the far the biggest issue with development is managing the inclusion of necessary stuff in the development environment. This includes auxiliary plugins that are required for the system, TiddlyWiki content and plugins, and static content that the web app needs. Mike’s blog post has most of the plans in this area. The gist is two pieces of development:

Improving the existing devtext store so it more effectively (and correctly) supports during development manipulation of the store and inclusion of TiddlyWiki content. We’ll be talking about the details of this in IRC this afternoon.
Creating a suite of shell-based tools that provide Make like functionality for establishing and maintaining a development environment and easing deployment. The long term plan is to migrate these tools to Python as time and knowledge allows.

For myself the things that have become important are:

Making existing plugins more visible to developers so they know what tools are available.
Packaging so-called mature plugins as Python packages that are installable via easy_install or pip.
Enhancing documentation of configuration options so they are more visible.
Publishing the quick hacks and tricks I do to make my own development life a bit more zipless.

Info Condom

Fri, 07 Aug 2009 16:43:00 +0100

The recent suffering of Twitter and Facebook (and others apparently) at the hands of a huge DDOS is just one more reason to disapprove of centralized, single-source, non-federated services: They are vulnerable to just this sort of thing.

A system of federated presentation services and many Banks of Content would be far less vulnerable. Oddly, this is a well known lesson and a fundamental piece of the architecture that underlies the internet. However it is a problem that is far easier to solve at the level of technological interaction than at the level of social interaction.

One of the appeals of Twitter and the like is that you can experience moments like this: “I’m on Twitter! ZOMG are you on Twitter too!!?? You totally have to get on twitter and then we can be on twitter together!”.

That’s far more fun and engaging than: “I propagate my microblog content by URI from my robust distributed content store to multiple presentation services, it would be great if your microcontent could co-mingle with mine on the open internet and together stimulate the generation of new knowledge.”

(Actually that sounds really sexy to me, but I’ve been told I’m a bit strange.)

Recent TiddlyWeb Plugins of Note

Thu, 06 Aug 2009 13:13:19 +0100

Though the historical roots of TiddlyWeb are as a store for quine-like systems (I should write more about this), because it has a very flexible plugin system it also manages to be something of an unintentional web-app framework. Many plugins have been developed over the past several months. Many are experiments to demonstrate possibilities. Others are practical tools. Here’s a bit of info on those I’ve recently put together. Ben Gillies, Jon Robson, Mike Mahemoff, and FND have all been doing interesting work as well.

Just this morning I made markdown.py which is in the class of plugins known as wikitext renderers. These provide a method for transforming the text of tiddler stored in TiddlyWeb into some other textual form, usually HTML. The default renderer is called ‘raw’ and returns its input. TiddlyWebWiki uses the wikklytextrender plugin as its default renderer. It takes TiddlyWiki formatted text and returns HTML. The markdown renderer renders Markdown syntax to HTML. It’s possible to run multiple wikitext renderers in the same TiddlyWeb instance, which is used is determined by the value of tiddler.type. Here’s the entire content of the markdown plugin:

import markdown2

def render(tiddler, environ):
    """
    Render text in the provided tiddler to HTML.
    """
    return markdown2.markdown(tiddler.text)

ramstore.py is a StorageInterface implementation for TiddlyWeb that persists the main entities of TiddlyWeb (Tiddlers, Bags, Recipes, Users) into RAM in the same process. Persist is not really the right term because as soon as the current process goes away, so do the tiddlers. This plugin mostly exists for testing (it allows you to effectively mock a store without actually being a mock) and to demonstrate the bare minimum of what a store needs to be able to do to fully support the StorageInterface. Longer term, however, I can see it being useful as part of a layered caching solution (one thread reads or writes RAM and returns control to the web request handling layer, another thread wakes up when RAM is written and takes what is written and sends it downstream to more persistent layers).

The default distribution of TiddlyWeb provides no exposure of User entities over HTTP. userbag.py and users demonstrate two (incomplete) ways of exposing them. The former uses a combination of diststore and a simple StorageInterface implementation which presents the users on a system as tiddlers in a bag called ‘users’. The latter add /users and /users/{usersign} routes with handlers that query the store. At the moment both are read only.

Maybe I’ll write another one of these.

Tools Together

Tue, 14 Jul 2009 14:10:22 +0100

Brain Dump

As TiddlyWeb has matured, one of the things I have noticed is that people are often wanting it to do more than it does. The generic form of the query is something along the lines of “Wouldn’t it be more complete if it did X”.

This is interesting to me because in a way it is contrary to the TiddlyWeb point. At base TiddlyWeb is a tool for manipulating chunks of stuff called tiddlers on the web. It is one of many tools in the toolbox for doing web stuff.

An analogy may help. There are lots of tools out there that spin and accept attachments: screwdrivers, socket wrenches, drills, etc. You can make a drill be a screwdriver if you want, but sometimes it is overkill. TiddlyWeb is an extensible web service: with extensions it can serve up static content, use templates, do all the usual web framework stuff. But that doesn’t mean it should be used for all the usual web framework stuff. Maybe, maybe not.

TiddlyWeb comes out of the Unix tradition: small tools that do one job well and can be integrated with other small tools through pipelines. This is true in how TiddlyWeb can interact with other tools and also internal to the architecture of TiddlyWeb: separated layers with strict contracts for interaction. This means it is easy to extend and it interoperates nicely with other stuff.

The most performant and scalable implementation of a large scale service that involves TiddlyWeb likely involves TiddlyWeb; a separate web server (or two or three or four) serving static content, doing some URL rewriting, some caching; a memcached server; a database server; and maybe another framework (django, turbogears, rails, etc) presenting UIs that might use the TiddlyWeb server as a datastore.

This last thing (presenting UIs) is the main area in which TiddlyWeb intentionally makes no provision (to concentrate on effective backend handling). Good web frontend presentation is strenuous, there are existing tools which make it easier. A tool approach means that TiddlyWeb can play well with those guys.

Leaping Forward

Fri, 26 Jun 2009 10:04:58 +0100

I’ve been having a bit of a debate with @Casablanca about the need or merits of designing and developing for people who use Outlook or IE. It started with:

@cdent: This #fixoutlook thing is bullshit. Just don’t use outlook!

@Casablanca: @cdent it’s not about whether you use outlook. It’s about having to design for people who do (e.g. our client’s customers)

@cdent: @Casablanca don’t design for those people! Design well enough that they’ll want to use what you’re using. If U cowtow, they’ll never learn.

@Casablanca: @cdent you mean we should tell our clients to tell their customers to change their email clients? A bit impractical, no?

@cdent: @Casablanca Just because it is impractical doesn’t make it wrong.

@Casablanca: @cdent does that mean you’ve never tested code in ie, because you don’t want to design for those people either?

@cdent: @Casablanca I haven’t personally tested code in IE for several years, if ever. Nor do I test other code in non-posix envs.

My basic assertion here is that people or organizations won’t change to things that are better if they can get the cool stuff on what they already have.

A secondary assertion is that by requesting Microsoft fix Outlook, Microsoft is not getting the punishment they deserve for causing the problems they have caused over the last…well the last very long time. There shouldn’t be a campaign to fix Outlook, there should be a campaign to boycott Outlook and boycott developing for Outlook.

I understand the economic considerations involved in the calculus used to determine “shall we do the work to make this work in IE”. For people encumbered by relationships with the corporate world, it’s kind of inevitable that someone along the chain will declare an operating environment where things must work. Likely that is an environment where openness, consensual standards and interoperability are meaningless tokens, not thriving actors in the system.

I try to avoid those kinds of relationships, and think other people should as well. I’m well aware that some people can’t and some people don’t, and more power to them. In fact it works out quite well for me: Because I work on open code in open environments, if there are people who are interested in making the stuff I do work in non-standard environments, they have easy access to do so. The pain stays with the people who need or want to suffer it.

Yes, I’m being impractical. Yes, I’m being unrealistic. But I can see little to suggest that the practical realities of the world are the best ways of doing things, so I’ll continue.

S5 For TiddlyWeb

Tue, 16 Jun 2009 23:40:00 +0100

This evening FND and I were discussing how one of these days I’ll have to do a TiddlyWeb presentation: make slides, talk to people, gesticulate in awkward ways while standing in front of people and moving around for no particular reason. I remarked that I was going to use 3x5 cards to lay out my thoughts because they are physical and I think better with physical stuff.

This led to more discussion about how in the TiddlyWiki universe slide presentation systems are a bit of theme or motif. I said I had considered making a tool to present slides out of TiddlyWeb but figured I would do such a thing when I really needed it. What I had in mind was some kind of jQuery thing that retrieved content from the server as needed. In other words, a project, an investment of some kind. FND brilliantly suggested that an S5 serializer would be the ticket.

I had heard of S5 before, but never really knew what it was. So I went digging around and saw that the basics of S5 are not only incredibly straightforward, the fine folks at meyerweb.com have packaged it all up nice and easy.

So I thought to myself: All TiddlyWeb needs to do is make the s5 javascript and CSS available (either via the hosting web server, or the static plugin), gather some tiddlers, and push them into the HTML file provided for S5. I set about doing that and lo and behold it works! Here’s an example. Go have a look, it gives a bit of an idea of what’s going on.

And here’s the code.

The serializer architecture in TiddlyWeb has been designed from the start to be extensible. By configuration or a plugin you can associate a MIME type or a file extension with a Python module that implements at least some of the SerializationInterface. A serializer takes TiddlyWeb objects, such as a tiddler or collection of tiddlers, and turns them into some stringified form of some type (HTML, JSON, an S5 file, etc.). Some serializers go the other way as well (e.g. turning a JSON string into a Tiddler object).

The S5 serializer checks for some special tiddlers to set meta-presentation-stuff like title and presenter name, and then renders each of the tiddlers it has been provided into HTML. This HTML is then pushed into the template file (jinja2 is used) and the results are sent out over the web as an HTML file. That HTML file requests back to the server for the necessary javascript and CSS.

Because TiddlyWeb already has all the bits for generating the list of tiddlers and getting them to the serializer, the serializer itself can be quite small and concentrate on just the work it is supposed to do. The code is only 55 lines long, including whitespace. And I like whitespace.

There are still some things to clean up, but I reckon it is nice start.

A demo
The code
TiddlyWeb’s home site
An old posting about TiddlyWeb serializers. That one involves cake rather than presentation slideshows.

Update: I updated the link to the demo to a new location to reflect that there’s now support for ordering slides in the presentation.

"The GIF Pronunciation Page. It’s jiff. Here’s evidence."

Thu, 11 Jun 2009 21:57:39 +0100

“The GIF Pronunciation Page. It’s jiff. Here’s evidence.”

- The GIF Pronunciation Page

Synthesis

Sun, 07 Jun 2009 16:16:42 +0100

Not long ago I decided to take a break from updating my status on Twitter and Facebook. It was a spur of the moment decision but the urge had been building for quite a while. What I said at the time was: “It’s mediating my conversations and I like the internet when it disntermediates.”

Meaning: Using Twitter and Facebook statuses gives me the action of initiating interactions with the world, satisfying something innate, but for the most part ends the experience there. On one hand this is perfectly fine: Broadcasting the ephemera of life maintains a loose contact, a finger in the ambient stream of ones social network. And every now and again someone will @you or “like this” to let you know that the world rolls on.

On a different hand, by granting a minor satisfaction of the urge to express, 140 characters of news bleeds off the pressure to say something of substance, something with significance, something that synthesizes.

This may be the most telling way in which I’m showing my age: Knowledge trumps Information; though communication is the grist of knowledge generation, action, results, doing, knowing is what matters. When I was at Socialtext and actively engaged with Blue Oxen I had occasion to play in the fields of the lords of Silicon Valley and often thought: All these Californians seem to do is travel from one conference, meetup, retreat to the next. When do they do anything? When is the synthesis? (It does happen, but it is on the fringes of the whirling soup of information being exchanged. People step out of the flow and then return with their synthesized new stuff.)

EEK (of Blue Oxen) and I, in our too infrequent conversations, often return to the topic “What happened to wikis?” Somewhere along the way the popular conception of a wiki transformed from a tool for synthesizing new understandings in a group to a tool for representing topical information which can refined. In many modern wikis, UI affordances emphasize the single page and the fact that it is editable rather than the entire wiki and its nature as an evolving (visible by recent changes) networked (via hyperlinks) corpus of shared understandings. A traditional wiki is a synthesizing tool when it exposes or strengthens the connections between distant concepts and categories. The group using a wiki learns by exposing, exploring and, eventually, refining those connections.

I get bored when I’m not learning, learning requires synthesis and synthesis requires reflection, reflection is something we pause for. Writing a tweet is vaguely like pissing with the wind: off it goes, out there with the breeze; easy, initially satisfying, but ultimately boring. So a pause. I’m sure I’ll return to it eventually but for the time being I need some reflection, some synthesis, and some doing and being.

While looking around for things I’ve thought or written related to this, I found this old email about knowledge representations from 2002 which has a bit to say about synthesis.

Don't Shape My Net Bro

Sun, 07 Jun 2009 15:24:56 +0100

I have always been, and plan to remain, fiercely of the opinion that ISPs should never be involved in traffic shaping at the application/layer 7 level. It is completely antithetical to the spirit of the network (stupid network and all that) and I think in the long run it is very bad business for ISPs.

If a customer wishes to do their own shaping to maintain quality of service (e.g. prioritizing interactive ssh sessions over retrieving web pages) more power to them. Such tools are available.

If an ISP wants to do shaping of the the total bandwidth any single endpoint gets across a switch, that makes pretty good sense.

But: If the ISP starts digging into packet headers and even packet bodies to do shaping they are introducing a huge cost for themselves. On the one hand their switches much do greater work. On the other hand there is now a presumption that the ISP knows or can know the content that is traversing their network and can be held liable for producing data about that content or controlling that content.

I get the impression that this latter point about liability is already a lost cause in the UK, which is a damn shame.

If BT or any other ISP is selling 8mbps connections to customers, then the customer should be able to expect that level of service for any type of content they get across the wire. If BT insists on being so oversubscribed that they can’t support that level of service at all times, they should shape total usage, not specific protocols or applications.

It’s interesting to me that BT would want to perform any kind of shaping. BT has an effective monopoly on the physical infrastructure that provides these services. By encouraging more and more intensive use of the internets they increase the demand for the infrastructure they provide.

Disclaimer: I do contract work with Osmosoft which is part of BT. These are my opinions. I’m sure there are lots of people within and without BT with lots of different opinions on this matter.

Internet Bank of Content

Tue, 02 Jun 2009 17:28:19 +0100

This is a followup to Invert the Web where I wrote:

Today I go to a web site and give it some output of my brain. The web site takes my content and in exchange it gives me a URL so I and other people can get to the content again. What should happen instead is that I make some content and give that URL to some websites so they can get to the content.

In typical me fashion that wasn’t very clear. Here I shall try to make parts of it more clear and other parts more muddy. Let’s expand the above.

Today when I publish a photo or make a tweet the mechanism goes something like this:

I suck the photo off my camera, do whatever tweaking I like to it, and then upload a copy of it to some photo site (e.g. Flickr). My photo then gets a URL somewhere in Flickr’s URLspace and I can tell people about that URL, or people can browse Flickr and stumble upon my photo. If things go my way and I play the game right, the photo might even show up on Explore.
I go to http://twitter.com/ or use an app like Twitterrific and type up my tweet. The content is transmitted to and stored on Twitter’s servers and gets a permalink in their URLspace.

What I would prefer is something like this, where the URL becomes a more Universal Resource Locator:

I suck the photo off the camera and store it in my own personal URLspace, getting a URL that links to the content. I then give that URL to Flickr and whatever other services provide a nice URI for browsing photos. People can still browser Flickr and stumble upon my photo and the photo may still show up in Explore. And lots of other places too.
I write my tweet and store it in my own personal URLspace. I use the returned URL and give it to Twitter, Pownce, Jaiku, Facebook, MySpace, iChat and a variety of other “status” related services.

At this stage of the conversation the first objection is usually, “Not everyone can host their own content.” Indeed this is true. It’s also true, though, that not everyone keeps all their money in their wallet, under the mattress or in a mason jar in a hole in the back yard. They use banks. And those banks provide a variety of services to provide incentives for people to deposit their money with them. So we could have banks for content.

Another objection is “I don’t want to do that two step publishing dance just to get something on Flickr.” Me neither, that’s why we have computers to automate the parts of processes which are fully describable. POSTting something in the content bank and then PUTting the returned URL to Flickr is fully describable.

Why would you want a bank for content? What would the benefits be?

All content remains yours, held in trust by the bank, forever.
If a presentation service goes out of business, your risk or loss is more manageable.
Any publication of content can be immediately rescinded by changing access policies on the URL (modulo whatever caching policies you’ve chosen).
You can (in theory) have access policies or capabilities statements on all your URLs.
Anything which can be captured by a MIME type can be stored in the bank.
You can host your own bank if you want: It’s just HTTP.
You can publish your content to multiple places by reference instead of copy. When you make changes, you don’t need to worry about going round to several different presentation services.
You have the option of maintaining several content accounts for all your various internet personas.
Banks could differentiate themselves by providing different levels of service: different ways of managing access control, service level agreements, data sharding and redundancy. Some people might opt for free content accounts in exchange for allowing some or all of their content to be anonymously mined for targeted advertising purposes or market research.
A suitably well featured bank could provide persistent and versioned storage for all your stuff, ever. Kind of like a github for everything, not just code.

And what about places like Flickr or Twitter, what we might call presentation services? What could they gain?

They don’t need to worry about storage quite as much.
They can concentrate specifically on the UI of their service, distinguishing themselves through innovations in that area.

These ideas are the latest synthesis of a variety of things which have proven to be important on the internet:

Separation of concerns. The presentation services worry about presentation and banks worry about storage.
Cool URIs.
Data ownership and portability. It’s always yours.
MIME. Lots of different types of content.
Stupid Network. Put the smarts where the smarts need to be and nowhere else.
The awesomeness of REST, or more accurately HTTP, and the open web it/they allow.
Using content by reference rather than copy (c.f. Xanadu, Transclusion, Purple Numbers).

It would probably be possible to prototype a content bank in TiddlyWeb or CouchDB or some kind of interface to S3. My own first work related to such things was closet from which TiddlyWeb got a few ideas.

#openhackday

Sun, 10 May 2009 18:37:24 +0100

I went to Yahoo’s Open Hack Day London yesterday. Forthwith my observations:

I went with Indy döt Net. We had some misconceptions about the event which may have informed our experience. Back before folk like DHH brought about the sexy extroverted geek, oldsters like myself thought of hackathons as small gatherings with lots of caffeine, pizza, a goal or two or three and lots of bandwidth.
Yahoo put on a great show. The space was large. The food was good. There were some interesting presentations. There was plenty of caffeine. There was some late night pizza. But it was, in some sense, a show (there was a band, they were fun). This is perfectly appropriate on Yahoo’s part. They’re trying to get people involved in their (by the way, very good) APIs, tools, info.
The bandwidth was not great. The usual WiFi problems: routers not giving out DHCP leases, people getting dropped off the ends of NAT tables, pipes being saturated. It seems like this ought to be a solved problem. My guess is that it goes something like this: The owner or supplier of the event space has a definition of robust network with phat pipe that is several orders of magnitude below that of the customer. Or at least I hope it is something like that. In any case, the Yahoo people worked valiantly all day long to get things working in an acceptable fashion.
Having bad network for the start of the event was a real buzzkill, especially when the stuff being shown off were tools for accessing information on the open web (YQL is cool!). I was planning to make something with YUI but since I couldn’t download the code, I used jQuery instead because I had that library on my local disk already.
FND discovered a big performance bug in TiddyWeb so we fixed it and released a new version. We might not have found the problem if it weren’t for the slow network.
Ingy and I left without staying overnight because the vibe was large and big and loud.
Tweets from the event point out that at least some people felt spectacularly productive in the time they were there. I certainly got a lot done, once I settled in. It suggests that there’s something wrong with the way we, as a society, do work. These days more and more people have the job of “knowing stuff”. Some days you’re just stupid, for whatever reason. Probably shouldn’t work on that day. Other days you’re smart. Then might be a good time to work. However this might also just be a romantic belief that supports my own predilections.

Anyway, it was a good time. Glad I was there.

TiddlyWeb for the Impatient

Wed, 08 Apr 2009 15:54:00 +0100

Update: These instructions have been updated to reflect that the wiki functionality of TiddlyWeb has been migrated into a plugin (called tiddlywebwiki). See TiddlyWebWiki for updates.

When TiddlyWeb first started being usable as an actual server, rather than a codebase against which to run some tests, it felt rather complex, maybe even onerous, to use. I hope that’s no longer true. Here’s TiddlyWeb for the impatient:

sudo easy_install -U tiddlywebwiki
twanager --load tiddlywebwiki.config instance myname
cd myname
twanager server
open http://0.0.0.0:8080/recipes/default/tiddlers.wiki or xdg-open http://0.0.0.0:8080/recipes/default/tiddlers.wiki or if you don’t have one of those commands open your browser to that URL.

In your browser you will now see a TiddlyWiki hosted on a TidldyWebWiki to which your edits and new additions will be saved.

For this to work you need a fairly modern system with Python installed, Python’s setuptools installed and generally speaking a friendly environment for doing stuff. In my experience that’s modern OS X and Ubuntu. YMMV.

Hosting a TidldyWiki usable by multiple users is but one small piece of what TiddlyWeb can do, but the above opens the door. For more info see http://tiddlyweb.peermore.com/

Update: Some people have reported that the second step fails reporting an ImportError. When this happens it means the first step did not fully install all the tiddlyweb dependencies, and you’ll have to install them separately using easy_install. The ImportError will report what is missing.

TiddlyWeb + IMAP

Wed, 25 Mar 2009 21:42:57 +0000

One of the more interesting but mostly unexplored areas of TiddlyWeb is its ability to use its storage system to adapt pretty much any datastore into something useful to the tiddlers, bags and recipes that TiddlyWeb uses. Last year some time I created tiddlywebweb, which lets you use another (remote) TiddlyWeb server as the storage for a different TiddlyWeb server. While cleaning it up last night, to get it more in line with contemporary plugin behaviors, I decided I need to make something similar to prove the point a bit stronger.

Python’s standard library includes some pretty useful tools for talking to IMAP servers and parsing email messages. Like many of the modules in the standard library, these aren’t exactly pretty or super clear, but they get the job done well.

I decided I’d start a TiddlyWeb store that uses an IMAP account on the backend. Rather than figuring out the mapping between email and tiddlers from the start, I figured I’d just get rolling and see where I could take things.

First I needed some code to talk to IMAP4 servers. imaplib does that. Second I need to figure out how to use it. So I started some code that logs me into my IMAP account, selects the default mailbox, and gets me some references to all the messages there:

imap = IMAP4_SSL(hostname)
imap.login(user, password)
imap.select()
typ, data = imap.search(None, 'ALL')

That code is straight out of the docs.

If I have some messages I want to be able to parse them. The email library does that just fine. When you fetch a specific message from the IMAP server it can be parsed from a string into a message object. For the sake of simplicity I’ve tossed aside any message that is MIME multipart:

messages = []
for num in data[0].split():
    typ, data = imap.fetch(num, '(RFC822)')
    message = message_from_string(data[0][1])
    if not message.is_multipart():
        messages.append(message)

Now it’s possible to do things like

print 'Subject: %s' % messages[1]['subject']

Okay, satisfactory. Now we know how to get at stuff on the IMAP server. Time to start making a store. We can use the same file. A store in TiddlyWeb is a subclass of StorageInterface. There is a set of methods it may support. No method is required. It’s really up to you what you want to handle. A store gets, puts and deletes various entities and lists collections thereof.

To get things rolling I decided my store was going to be readonly and work for just one user, at least at first. Making messages map to tiddlers, and folders/mailboxes map to bags seems to make sense. Recipes? Dunno. Maybe someday that would be a mechanism for handling multiple user accounts on the server.

The store methods we need to implement, then, are list_bags, bag_get and tiddler_get. List all the folders on the server, list all the messages in a folder, retrieve an individual message and map some fields to tiddler fields. Mostly straightforward, with some blips:

I don’t know about you, but I have several hundred folders in my imap account, some with an astounding number of messages in them. Anything with “list all” in the description is kind of bad news. So the first hack was to only list folders beginning with ‘a’ (in IMAP-style pattern matching this is ‘/a%’).
Any bag may only contain one tiddler with any one title. This is not true in an email folder, so when listing the contents of a folder/bag and creating a list of messages/tiddlers it is necessary to create a unique ID. I chose the union of message-id and subject. The message-id should be unique but is not particularly meaningful in lists. I thought about using the IMAP UID but these are not guaranteed to be the same between sessions.

From there it is pretty cool with the tiddlers: the body of the message is the tiddler text (we are only accepting non-multipart messages), the date of the message is the tiddler modified time, the email address part of the from is the tiddler modifier.

And so it all pretty much just works. If you go to /bags you get a list of all the folders starting with ‘a’. If you follow links, you eventually get a list of all the messages in a folder. If you click on one, you can read it. If you click on a tiddlywiki link, all those mail messages show up in a tiddlywiki, which I must say is quite wacky.

Things that would be useful:

Some form of caching. Lord can it be slow. Even since some core performance changes to TiddlyWeb in the last few days.
The same date used to set modified should set created too.
Other message headers in the tiddler as extended fields.
A TiddlyWiki plugin plus a TiddlyWeb plugin that lets you send mail from TiddlyWiki in response to messages.
tiddler_put and tiddler_get for moving and removing messages.
Some kind of role for recipes.
A something, maybe called polystore, that lets one TiddlyWeb server access multiple different stores.

What else? Give it try if you are curious and hardy. Contact me if you want some help.

The code is in the TiddlyWiki subversion repository, in my tiddlyweb-plugins dir. I’ve also started keeping track of such things on github.

Docupeopletation

Tue, 24 Mar 2009 12:00:00 +0000

Writing documentation is spectacularly difficult. At least for me. While I’m capable of being quite verbose (perhaps too verbose) in response to a question (perceived or actual) I’m crap at predicting what people will want or need before they get to the point of asking questions.

This is having an unfortunate effect in the land of TiddlyWeb: documentation is lagging quite far behind the functionality of the tool and while I’m generating new documentation on a fairly regular basis I have no way of knowing if it is the right documentation.

Any software project, especially one which is both an application, a library and an HTTP API, has several different types of documentation. There’s the documentation in the code explaining functions, classes, methods, modules and such. With Python there are tools like pylint and pep8.py which will tell you when your code is missing docstrings so TiddlyWeb has most of that covered with at least basic documentation on stuff that is public.

More complicated is the narrative that explains what TiddlyWeb can do and the documentation that explains how to do it. Making it go, getting the best out of it. That’s the stuff that’s hard to do alone; since narrative is a story it only emerges through human communication: feedback loops, a dialog of participants.

What I did on my Spring Vacation

Sun, 15 Mar 2009 13:14:00 +0000

I made raamable.

My friend Stan is going to be part of the support crew of Team Hoosiers when they once again participate in the Race Across America: a west to east cross-continental traverse of the US. On bikes.

Stan wondered about the possibility of some web based resources to help make things go. When heading across the country where every minute counts it’s good to know where you are going; what’s coming up next; weather conditions; the nearest grocery, gas, laundry, bike store.

When I was working for Socialtext most of the developers worked from home, scattered around the world. When we did get together we’d sometimes experience bursts of productivity that were very pleasant: a great sense of being in the zone. Bourbon was often involved. Intense work and intense play.

Seemed like Stan’s project was a good reason to do some good work for a good cause.

So I hunted out Matt and Zac, also former Socialtexters, and we planned a hankathon (hack and drink a thon) in Bloomington, the font of much goodness. The idea was we’d gather, mess about with some Google mashups, see what we ended up with, and see what we learned about how to have effective similar things in the future. These days it should only take a few days to establish a startup.

We made plans. Zac couldn’t come: teething baby. Down to just Matt and me. I was sick, Matt had to work, and hanging out with Stan is too rare and too fun to be wasted on staring at computers so we didn’t have the focus and concentration we originally desired. We still managed to get some things done and learn some things.

On the learning side we had some thoughts for next time:

Clear the decks. If you’re going to a multi-day hackathon, don’t bring other work.
Have a least three people. If you know exactly what you are doing, pairing or sharing work can work out very well, but if you are exploring as much as executing, an odd number keeps thinking unstable and diverse.
Spend the first day, half day, some unit of time brainstorming and finding the point of gravity. The group needs an articulated shared goal to have shared momentum.
Have a tool for keeping track of what’s up. We used a TiddlyWeb hosted tiddlywiki but didn’t initially use it all that well. Since we were in person I suspect something more tactile like a whiteboard with stickies on it might have worked better.
Get necessary infrastructure in place beforehand.

Despite being a bit discombobulated we did manage to create something.

Uncle Ingy says every good project starts with a good name, so I baked the name “raamable” into a git repo on github and started gathering the necessary pieces.

Since we knew we’d be working on this project separately before and after the hankathon, we’d need a convenient way to work in a test and deployed environment. Google App Engine makes that easy: do local development until you get it right and then deploy to the servers.

We had some known data sources: gps waypoints for the race, locations and frequencies of NOAA weather radio towers, a web service that gives elevation for longitude and latitude in the US.

First pass plan: put all that stuff on a google map and see what happens. There were some challenges.

It’s easy to put a bunch of markers on a map. The code shows that well enough. What’s harder, when there are several hundred markers, is plotting the line of the route on roads. Plotting a route requires using GDirections, which will only take 25 waypoints per use. Our solution was to loop over the markers in chunks of 25. The results in the map shown on the front page. The animation is a side effect of the process being somewhat slow. The darker blue spots happen where one block of 25 stops and another starts.

On the interactive map yellow markers are time stations, orange are turns on the route, and the towers mark nearby weather radio towers. Each GMarker that represents a turn has an added attribute altitude containing meters of elevation at that location. Each GMarker representing a radio has a frequency attribute.

The race route turns information did not come with elevation data. I initially planned to make the map queryable for elevation at any point (thus the /proxy URL in the app engine code) but then we decided an elevation profile image would be nice. Dynamic querying for that would be much too time consuming and barring catastrophe elevation doesn’t change much so storing it on the map seemed the way to go. For kicks I processed the route turn CSV file through a shell one liner using curl to get the elevation data and add it back into the file.

The elevation profile is a google chart that is redrawn each time the map is moved or zoomed. The crux here was to make sure that we only draw information for those markers which are within the bounds of the current view. updateProfile in the code shows how this is done.

There are many hundreds of radio towers in the continental US. We only want to show those which are near the route. There’s doesn’t seem to be a straightforward efficient way to say “plot only those waypoints which are near to this other set of waypoints” so we constructed our own. The entire route describes a rectangle that has northern and souther latitudes that can exclude most of the towers. So first we do that. The slope of the route (southwest to northeast), however, is too steep for this to be good enough: a larger number of towers are plotted which are too far away to be tuned in and make the map uselessly noisy. To deal with this an array of bounds, one containing each ten turns (plus and minus 1 degree of longitude and latitude) along the map, is created. The remaining radio tower locations are compared against the bounds. See _establishRouteCallback and _addTowersCallback.

We hope to do quite a bit more. We’re tracking our plans on the wiki. What’s running now is a useful tool but suffers one rather extreme flaw: it requires network access. We hope to modify things so that useful printouts can be made from the system and carried on the route.

This is an open source project, you are welcome to help out, use this for your own bike ride, or whatever. If you do, we’d love to hear from you.

Invert the Web

Thu, 05 Mar 2009 20:38:22 +0000

I was talking this afternoon with Fred. Over the past few days, he and Jon have been creating a Twitter archiver called TiddlyTweets using TiddlyWiki plugins. It’s cool: you give it a twitter username and it retrieves all the tweets and archives them in a useful fashion, preserving a link to their original URI.

I’ve been hassling Fred about this, as I like to do, because one of the reasons I have liked Twitter is because tweets have, or at least had, a sense of being ephemera. In my mind, if you feel like you need to archive your tweets, then you are doing it completely wrong.

But that’s just my opinion and Fred’s is different. He says there’s good stuff in those tweets which you might want later. This is true enough.

Today, in IRC, he said:

without starting the whole discussion again, after
yesterday's(?) archiving exchange, it struck me as
odd that you're so indifferent - after all, you're a
proponent of cool URIs, which kinda includes permanence

My response went something like this:

Every tweet has its own URI already, the “statuses” URL, so the coolness is basically there already. Fred responds that Twitter is not reliable. This is true but I suggested that archiving was a short term solution to the problem that allows the long term problem to persist. The long term problem is that ones tweets live at Twitter, not in ones own persistent store: Twitter should be a presentation engine which has access to your own microcontent.

This reminded me that in some ways what I’ve been working on for the last ten years or so with Purple Numbers, wikis in general, TiddlyWeb, persistent identifiers etc. are tools to make such things possible. What I’d like to see is an inversion of the current content relationship that is present on the web.

Today I go to a web site and give it some output of my brain. The web site takes my content and in exchange it gives me a URL so I and other people can get to the content again. What should happen instead is that I make some content and give that URL to some websites so they can get to the content.

TiddlyWeb Store Migration Plugin

Fri, 13 Feb 2009 13:03:40 +0000

Some of the people at Osmosoft are working on an additional StorageInterface for TiddlyWeb that enables development of tiddler content that is readable by TiddlyWeb, by Cook (the tool used to build TiddlyWikis out of content in the tiddlyweb subversion repository. To help make using the new store, I’ve written a plugin that can migrate TiddlyWeb data from one store to another.

TiddlyWeb can host in one process as many different references to as many different stores (objects that implement the StorageInterface) as you like, so creating a migration tool is a simple matter of enumerating all the content in one store and putting it into another.