It took a while before we restarted our podcast “Data Without Borders” but we are back on track with the third episode after the break where we talk about what happened at the Internet Identity Workshop (IIW) which happened last week.

We also got two new people on board: Eve Maler from PayPal who is also the chair of the Kantara Inititative’s User Managed Access Work Group and Elias Bizannes, like myself board member of the DataPortability Project.

Of course we also still have Trent Adams from the Internet Society (the organization behind the IETF) and another member of the DataPortability Project board: Steve Greenberg.

And last but not least we are now an official podcast of the DataPortability Project (whatever that actually means, in terms of content probably nothing and it should be noted that the opinions expressed in thep podcast are our personal opinions and not some official opinion of the DataPortability Project unless stated otherwise).

The Internet Identity Workshop

I heard a lot of very good things about the Internet Identity Workshop which happenes twice a year in Mountain View, CA. Unfortunately I never made it there though. But at least I know people who are going there and this time all of my podcast co-hosts have been there and so we talk about some topics which have been discussed there, like the future of OpenID, Webfinger, LRDD, XRD and lot of other upcoming and established standards.

But give it a listen yourself:

(Download MP3, 45 min, 46 MB)

For the shownotes check the blog entry.

You can also subscribe to us on iTunes.

We are also streaming the recording of new episodes live on fridays, 1800 UTC. More information will be posted to the Data Without Borders Twitter account.

Malthe Borch organized a session about “Otto”, his and Jeroen’s approach to make Python Web Framework programming fun again mainly be reinventing the wheel. But give it a listen yourself:

(Download MP3)

(It started by Malthe explaining that repoze.bfg was born out of forking Zope3 packages to make them work in your environment and then noticing that you don’t need them and throwing them out again or something along those lines)

Georg Greve from the Free Software Foundation Europe was talking about decision making. He started explaining how he moved from being a software developer to being a politician.

He takes the relicensing policy as an example and wonders how we reached a decision on it. What is a majority? How do we define the qualified majority needed? Here the Plone Foundation is one important piece. If we wouldn’t have the Plone Foundation it would come down to consensus. This works well in certain group but sometimes it also leads to shoutocracy where those people who shout the loudest go on doing so and the others at some point simply give up.

This is not applicable in many cases. E.g. in the Plone Foundation it is about copyright and you need unanimous agreement, you need everybody to say “Yes”.

Copyright Law

Copyright laws are man made by man made processes. So where does it come from?
It starts with national levels which have some form of majority. In a bi/multilateral situation it is unanimity and in the united nations it’s consensus.

This sounds as if the bi/multilateral decisions are easier to make than the UN ones but that’s not necessarily true. There are a lot of rules involved and it might come down to harsher negotiations. It is also complicated with certain regimes where they technically have 100% of their people behind them.

But why do copyright laws happen? There are always some reasons behind those laws. In copyright it is about “fairness” politicians say. But what does “fair” mean? Also politicians are not expert in this field but they still need to make decisions. So they rely on other people to help with that decision.

He gives TRIPS (Trade Related aspects of Intellectual Property Rights) as an example. Originally it was driven by a very very targeted agenda by large US corporation, by a coalition of the willing. Only those who wanted the maximum went along. And now governments need to enforce something which was created by a group of coporations. He mentions the U.S. Special 301 list with the “naughty” countries who do not enforce that.

He mentions a study (LAWRENCE) where 3 professors found out that lobby expenditures can receive you a 22,000 percent return on investment. This is for tax regulations, we don’t know about copyright lobbying. So it’s a worthwhile thing to do for companies.

On which side do you wanna be? What’s your perspective of this?

“Politics is too serious a matter to be left to politicians” (Charles de Gaulle)

The only way your voice is heard is that you need to be at that table where decisions are made. E.g. Plone is also affected by this. Like software patents, web standards, etc.

“Those who are too smart to engage in politics aree punished by being governed by those who are dumber” (Plato)

Inactivity is not an option. There are moments where we have to act!

Governance?

This is going in two directions, the external government between Plone and the outside world. But there is also the inside governance, how does the community work in itself?

Plone already decided to be a community instead of being a single vendor. He thinks that this is the more sustainable approach. The single-vendor approach is complicated because it’s only one company you don’t know the future about.

There are also other issues that come up. How do you address growth? This is one of the most challenging issue. How do we keep the substance strong as people get in and how do we ensure that our community remains able to make decisions. No one has found the answer to that yet though.

What about the structural “bottleneck bugs”?

There is things in the plumbing which you might want to redo at some point. KDE just has gone through such a process which was rather painful. Many projects encounter such situations. It is very hard to find volunteers to do that work. You also cannot find a customer to pay for all of this. There is no visual change.

How do we find ways to address those problems? He knows that the Plone community is thinking about this and he also helped the Open Database Alliance to find a solution for this. There the membership fees are done like this: 50% allocated by members, 20-30% allocated by board. Members can allocate fund for certain projects. All the members interested in this can pool their funds for this project.

In order for this to work you need the right choices. The ODBA has certain seats for users, developers etc.

Food for thought

1. Get active. Find ways to express your political agenda! Just being a small foundation of a few 100 people is not an excuse. You can make a difference!
2. The most important asset of Plone is YOU! In the end whether the PF is structured this or that way, the strategic choices still need to express what the community wants. He was positively impressed by what he experiences at the conference as he experiences a positive and lively community. Keep it that way!
3. Coordinate with your allies, like FSF. Talk to them as often as you can. Find common priorities and build alliances on them.

This live blogging report is without any guarantees of correctness and corrections are very welcome in the comments!

David Glick is introducing Dexterity to us, the new way of doing content types in Plone. It is aimed at non-developers and developers.

What is a content type?

• A way to categorize the items in your site
• It has a Schema which is a list of fields
• It has a workflow/permissions
• It has custom view templates
• misc settings, like placeful restrictions, comments, searchability, per-type portlet assignments

Content Management Framework (CMF)

• Underlying framework for registering types and assigning workflows
• CMFDefault contains sample types which used to be used by Plone
• It is not schema-based though

Archetypes

• Provided schema-based form generation
• Basis of Plone’s current default content types (ATCT)
• Not going away anytime soon

It has some ugly bits though.

Dexterity

Because of the drawbacks of AT, Martin Aspeli invented Dexterity.

Goals of Dexterity:

• Make filesystem content type development sane
• Make through-the-web content type development sane
• Make it possible to switch back and forth between the two

The philosophy of Dexterity:

• Reuse over reinvention
• Small over big (e.g. AT is quite big, it’s all in one package)
• Natural interaction over excessive generality
• Real code over generated code
• Zope 3 over Zope 2
• Automated testing

Installing dexterity

It’s a normal python egg you add to your buildout (plone.app.dexterity). Right now only compatible with Plone 3 but work on Plone 4 is ongoing. Then of course you also have to install it inside your site. This will give you all the products and a Dexterity Control Panel.

Using dexterity

Inside the Control Panel you can create a new content type. Right after that you can add it to a plone site but it only has Title and Description. It also contains a default view which contains also lots of meta data fields.

With “Behaviours” you can extend the functionality of the content type like choosing a better shortname etc. This is different from subclassing and schema extension. Behaviours are stored in the FTI, they can be switched on and off. Examples are thumbnail generation, versioning, ratings, “name from title” etc.

In the FTI you now can disable the Metadata behaviour and enable the name from title-behaviour.

Now we need more fields. This can be done in the Schema tab. You can give it a title and a type. A field has lots of attributes you can edit then like if it’s required, if it’s read-only, the default value etc.

Rapid Prototyping

The Schema object in memory is directly modified, so changes take in effect immediately.

Modifying a typey

You can add, remove and rename fields through the web. But, values stored in exising instances will not be automatically removed or converted. So be careful.

Some more desired refinements

• Custom add permissions
• Show the “bio” field in a separate fieldset

These cannot be done throught the web (not yet). So lets switch to the filesystem.

Filesystem roundtripping

Schema editing internally is done in XML, stored in the FTI. This can be imported and exported via GenericSetup. It also is converted into a Zope 3 Schema which is then used to edit content.

He then goes through the files which are inside an example dexterity product structure. E.g. the __init__.py stuff is gone and now works all via GenericSetup. There every type will have it’s own XML file which is the generated one we talked about earlier.

He also shows the Zope 3 schema with custom fields you can use and some dependencies like five.grok and others.

To make that fieldset you then have to change the Zope3 Schema by adding a fieldset and adding a list of the field names to that set.

There are also more form directives he doesn’t want to get into like widget, omitted, mode, order_before, order_after etc. Additionally you can set permissions for individual fields.

Custom view templates

To register a custom view you write:

from five import grok
class View(grok.View):
    grok.context(IPlonista)
    grok.require('zope2.View')

Templates are found by convention (thanks to grok). So you need to put it into “plonista_templates/view.pt”. Inside the template you also can use direct attribute access as dexterity is intelligent enough to find that data.

Enabling your filesystem content type

You can then add the package to your buildout and rerun buildout.

After restarting Plone you can e.g. see the Bio field in a separate fieldset and you also see the custom view.

The pieces

• five.grok
• plone.dexterity (core package, base content classes, FTI, default edit form and view)
• plone.autoform (Dexterity renders widgets using z3c.form, autoform makes it possible for a form to be composed by schemas and form hints from different sources, like main schema + behaviour schemas)
• plone.schemaeditor (used for UI for editing schema TTW. basically you can edit a Zope3 interface TTW. could be used seperately. IFieldFactory determines which fields are editable)
• plone.supermodel (serializer translates Zope3 schema into XML. Parser translates an XML schema into a Zope 3 Schema. It’s easily extensible)
• plone.directives.dexterity (grok directives for custom content classes)
• plone.folder (orderable btree-based folder implementation, will also be the basis for AT-based folders in Plone 4)
• plone.behaviour (a behaviour is a conditional adapter. In Dexterity, the condition is whether the behaviour is listed in the FTI for a type. ZCML directives for registering new behaviours)
• plone.rfc822 (supports marshalling of Dexterity contet into RFC 822 format, used to support access via WebDAV)
• plone.app.dexterity (pulls in everything you need. Standard behaviou: DC, related items. Provides the types control panel)

Status/Roadmap

• Core functionality is pretty solid
• Schema serialization
• automatic form generation seems to be working will
• portlet assignments works
• content rule assignments works
• Relations not working between AT and Dexterity content types in the same site
• Widgets nor as rich as AT yet but better than formlib. We have autocomplete, browser-for-content, file/image upload
• TTW schema editing works fine but needs more real-life use
• Image & File support (via plone.namedfile & plone.formwidget.namedfile). No support for image scaling yet.
• Rich Text field with text transform support
• Select field support. no way to define vocabularies through the web yet.
• WebDAV support is work in progress.
• Versioning and staging is in progress
• TTW behaviour creation would be cool but is not yet implemented.
• Link integrity checks are not implemented yet

Upcoming releases

Second alpha release was on Oct 12. First beta release coming soon.

It’s compatible with Plone3, Plone 4 compatibility is coming soon.

Documentation

Dexterity Developer Manual
Behaviours manual

Getting involved

Code on Google
Google Group

What’s new in the latest release of zc.buildout?

1. Extension build by gocept, you can use buildout really offline. It tells you if you cannot download something in offline mode but still uses download-caches etc. if you configure it.
2. “annotate” command will give you where all the values are coming from when having a very extended configuration. It also shows all the places from where the configuration is coming from and which overwrites which. He wants feedback on that.

Elizabeth Leddy is talking about scalability issues with Plone and how to approach scalability: Tools, Tips and techniques for making Plone scale.

This talk is not about just installing something like CacheFu, it’s more about specialized situations where you cannot cache, have SSL, are contantly logged in and so on.

“A responsive, scalable Plone setup has little to do with optimizing code”

We also need to know that Plone is not Drupal, so we do not get the same RPS. That means that we need talent and hardware to get Plone to scale. She also says that Plone people are kinda special, compared to PHP people.

Measure!

The first important thing is to measure what really is slow. So setup your tools to do measuring.

Tools you can use:

• munin
  monit (or supervisor)
  ganglia
  Zope Health Watcher (monitor response time!)

Whale Watchers

You can also use a whale watcher which is triggered in the abnormality case and reads all statistics it gets.

Simple(r) error reporting

It’s good to have some sort of escalation. Like escalate with email, use Google Analytics for Timeouts, 4xx and 5xx pages.

4 stages of system grief

Denial - Anger - Bargaining - Acceptance.

She recommends sharding your ZEO as much as you can which means splitting the data in smaller databases. This makes moving databases around easier. You can also pack faster, backup faster. It also helps reducing the single point of failure you have with ZEO.

How much hardware

First see what components you need and how they need to be distributed on different boxes and how these boxes need to look like. Also check the tradeoff between RAM, CPU and Disk space.

With Zope and ZEO on different computers you also have network latency. With a shard ZEO it also makes sense to put the ZEO on the same computer as the Zope. But not everybody can do this.

She then explained a munin graph of the memory usage explaining the badness of Swap. You want it shared between cache and apps. With that graph you can also find out how many Zope instances you can use on one box.

How many Zopes should I have?

• 50% avg. util
• 2 zopes/CPU
• api/async instances

(she looked at Mongrel and what they found out).

You add more hardware when in any point in time the average is more than 50% utility. 2 Zopes/CPU is a good way to start. Question: How many threads per Zope instances. Answer: it depends. She uses 1 as the OS is way better at context switching than Zope is. But be careful with asynchronous requests.

What does “asynchronous” mean?

A lot of requests in Zope take a very long time (like 2 minutes). If you can take out some time out of that, do that. Like if you have to collect data from somewhere else, do it later, pickle it up and do it separately. (Pickle, Drop, Pickup, Process, Callback).

use httpd

Serve static content from a static web server to unload plone, provide faster response time to initial requests, and enable simpler caching strategies. Esp. for non-existing images plone takes faaaar longer than an apache (404 that is).

(you might want to rewrite the access to /portal_skins/…)

CSS and sprites

use CSS sprites! less bandwidth with less response time and onle 1 request.

HAProxy

handles graceful reloads, backend health, distribution algorithms, warmup time, preserve keepalive, web based stats.

HAProxy esp. knows if a request is finished on one instance and then gives it the next request instead of going round-robin.

It also knows about Zope startup time and does not send it requests during that.

Find it here.

ZEO

It’s perfect for OO access but has problems with other data access paradigms. For this use something else.

If python <2.6: socket.settimeout(2)

If you know you have a long running request, set it higher and set it back.

Restarting

Restarting is not a sin. Let Zope eat as much as it wants but restart it regularly. Also pack your ZEOs frequently. Do it for your system, not for you.

Backups

First: Get good disks and your ZEO will fly.

• Use repozo for backups (and look out for timing, don’t do it during busy times).
• Use chunked rsync for backing up regular files (like your Plone install).
• Watch the disk

Summary

• Always setup system and error monitoring first
• choose the right softzware for YOUR hardware
• use as many zopes as possible but no more
• do it like a flickr engineer (async)
• Don’t make zope handle unnecessary requests
• Accept that zeo usually isn’t the only db in a scaled solution
• Never understimate the importance of a proper disk-RAM partnership.

Wichert Akkerman was talking about lessons he learned from other frameworks. He once was looking at other frameworks and thistalk is about a view for developers. First of all: Plone is an application, not a framework but still we build stuff on top and thus it’s a valid comparison to make.

• Zope Toolkit (Zope 3 reborn). Nobody knows what it is and is changing all the time. It has not been released yet. It’s the basis of several other frameworks.
• Grok (A Smashing Web Framework). Is an attempt to make Zope or the Zope Toolkit easier.
• BFG is a “pay only for what you eat” Python Web framework. Has the most released compared to the other, about 76 releases in a year.
• Pylons is lightweight and little.
• Django and Plone are also there, where the latter is called “Plone Portal Toolkit” by him

Templates

Every framework has one template engine. ZPT sometimes are smarter than you though. E.g. try to put a style element in your page and you have to do lots of “<tal:pain replace=”structure string:&lt;”>”. So you have to do all sorts of tricks to get it done and this is not helpful. You want to do this:

<style type="text/css">
#nav li#home {
   background: url(${logo}) no-repeat;
}
</style>

Another thing where Plone differs from all the others is that Plone already has templates. In others you start with nearly nothing. No standard views are there. No templates to customize.

There are also tools like XDV or deliverance which he thinks is still not delivering though. E.g. using breadcrumbs is tedious and you can do it with XDV using custom XSLT but nobody really will do this (or can do this). But you can customize breadcrumbs in Plone but then you have again 2 fields where you have to change stuff.

So Deliverance is interesting but it has it’s limits.

Authentication and Authorization

Plone does all that for you, it has the PAS. You get all that for free. In other toolkits you have e.g. PAU (in Zope Toolkit) which is supposed to be a more flexible way than PAU but in fact is harder to understand.
In BFG it’s far simpler. You can still cover all the usecase but you might need to write a little bit of Python. Compared to that PAU is far too complex to understand.

TG2 and Pylons both use repoze.who. It turns out though that repoze.who does the same mistake as PAS and PAU as there is too much policy in the code. In the end it restricts you.

Forms

In Plone you have many ways to do forms. You have CMFFormController, there is formlib, z3c.form, ATCT and dexterity. There are 5 different ways to do a form in Plone and this is a bit too many. All are also doing the same mistakes. You define data in the form, you have validating, the marshalling, the generation of the form, they are all built into one package. The result is something that is very very complex.

But if you split them up it becomes simpler and you can use them separately.

Pylons use formencode and you can use validator decorators on controllers to do the validation (@validate(schema=…)). But it’s not generating a form etc. It’s just validating. Pylons additionally can add error messages into the HTML.

wiggy also experimented a bit with formish which also has separate packages for the individual functions.

There is no problem to split it into different packages, you might need to write a bit of code but that’s ok and so much simpler in the end.

Data

In Zope you are usually ZODB based. BFG OTOH can do both, ZODB and SQL as not storage layer is included. Pylons uses SQLAlchemy but it’s very easy to switch to something else, e.g. CouchDB. The big difference there is that if you look at the Zope based options, then you always have a root object. That way you can have multiple sites in the same storage but you also always have to use the ZODB. This makes it difficult in case your data is more relational. Django has it’s own ORM.

In Plone you have again lots of options: Archetypes, Devilstick, Dexterity, … There still is no good story for a SQL based storage. It’s very hard to do although the storage itself is not the problem.

Configuration

Plone/Zope usually used ZCML. It was intended to configure components together. But it’s gaining some use for general configuration. There also is the zope.conf file which was used for everything but it somehow has been forgotten. Then you additionally you have all the settings stored inside the ZODB which makes it sometimes harder to deploy the site (e.g. file paths might be a problem). This is not done by any other framework.

BFG is using ZCML but only for component wiring up. The rest is done in an .ini-file, like Pylons, TG2 etc. Django uses a python settings file. This is also the plan for Plone but it’s a long way to go.

URLs

In Zope based sites everything is based on traversals. The Zope publisher is iterating over the path and maps it to objects. This works well with the ZODB as it’s a hierarchical database. There are some exceptions though for URLs which do not map to a ZODB object (”/folder”,”/folder/@@view”,”/++resource++package/public.css”). The traversal in Zope is highly complex. There are no API to generate URLs for e.g. views.

The other frameworks have usually a much simpler option, using Routes. Here you map regular expressions for URLs to a bit of code. You can then also have different names for the same bit of code, e.g. for different languages. You cannot do that easily in Zope. With routes you can also reverse engineer URLs.

The downside is if you have hierarchical content like in Zope or Plone. So it’s a tradeoff. In BFG you can use both at the same time. For Zope there are also some experiments with routes (me.grok.trails).
Having the liberty to do both is very handy.

Documentation

“Always a fun one”.

Zope Toolkit has no documentation at all. It has meta documentation, like how to make releases which hasn’t happen yet, what should be included, coding styleguides etc. That makes it a bit hard to work it. The only thing which might help is philikon’s book which is 3 years old.

Grok is well documented, has lots of tutorials but only partial Reference. It’s also not sure how it’s versioned. But still you might need the Zope Toolkit at some point with it and then you are back to no documentation.

BFG has everything. It has tutorial, fully versioned documentation. Every release becomes a new set of documentation.

Pylons is also good documented but it’s not versioned. They also use a lot of addon packages which are documented separately. It needs to be seen if documentation is kept in sync and keeps relevant.

django has again good documentation, it has tutorials, reference documentation and it’s versioned. It’s very code oriented.

Plone is catching up, the documentation team is doing a tremendous job. It has tutorial but only partial reference documentation. It’s not versioned though. It even has more documentation for e.g. buildout than buildout itself.

What’s important?

The most important thing is that you feel “in control” and you actually are. You want good documentation and not missing or misleading docs.

“Python is an excellent choice but to be fair, if you are only doing a one-off project I think you might want to use a simpler framework than Grok for your portal.” (Sebastian Ware on the grok list)

Lessons

• Modularity can confuse (Django is very integrated and this might be good as there is only one choice. With Plone you might need to learn too much. At least at the documentation experience level it’s important).
• Explicit is simpler than a framework (nobody understand how to change Plone portlets or the navtree although it aims to be very flexible.. But writing code might be simpler).
• Version documentation
• Less features, not more
• Sugarcoating is not a solution (lesson for Plone and Grok, at some point you need to look below that)

Tom Moroz, Chris Rossi and Calvin Hendryx-Parker have been talking about KARL.

Tom Moroz of the Open Society Institute (OSI) starts on a bit of background on what KARL is.

OSI has 7 ‘core’ offices in US and Europe. It has national foundations in 32 countries. They have a program and grant giving activity in over 60 countries and >20 Program Areas / Initiatives.

KARL was begun in August 2006, in May 2007 KARL v1.0 beta release, January 2008 an official launch, November 2008 KARL v2.0, June 2009 KARL v3.0. OSI first developed it for their own needs but later saw that many NGOs have the same problems so they tried to open it up for their use cases as well.

He then showed some of the features of KARL.

Chris Rossi then took over and explained a bit about the technical background. KARL moved away from Plone to use BFG and he tries to explain the reasoning:

• Plone is great at what it does. Content Management in a nice, general way. This is the sweet spot of Plone.
• What do you do when your project is outside of the sweet spot?
• Further you need to exted Plone, the more you fight the framework. Result is a high developer frustration.
• Using BFG, they were able to maintain programming paradigms familiar to Zope/Plone developers without using/fighting Plone.

The move from KARL 1 to KARL 2:

• Entirely new UI built on top of Plone — very different from Plone’s UI.
• Much framework fighting
• Much integration with non-Plone technologies, aka Frankenplone.
• Because of the Frankenplone infrastructure they ended up with bad performance and poor reliability
• It also resulted in difficult maintenance
• Difficult to provide something reasonable for non-OSI deployments.

What are the alternatives to Plone?

• Zope2, primarily used by Plone, time tested.
• Zope3, big, many dependencies, app developers encouraged to use CA.
• Grok, an attempt to make Zope3 easier to use (but it’s still Zope3 and still big and complicated).
• Django, nice, but full stack like Zope2 (you need to use their answers for all you do).
• Pylons/TG2, also nice but geared towards RDB/URL dispatch
• Several others, little traction, not Zope like.

With BFG:

• It is in the repoze namespace, but is not really repoze.
• Like Zope, but much smaller, leaner
• Speed and simplicity: “Pay only for what you eat”. If you don’t need it, it’s not there. If you do need it, it is easy to put it in.
• Documentation: An undocumented feature or API is a bug.
• Testing: 100% test coverage. Untested code is broken code.
• Collaborative culture: Using and promoting non-Zope technologies when they are useful.
• Easy to adopt for Zope developers. (Maybe non-Zope developers as well.)

KARL technologies

Python: WSGI, WebOB, Paste Deploy, virtualenv, eggs, repoze.who, Supervisor, TinyMCE, jQuery, nose, twill, mod_wsgi.

Zope: ZPT (Chameleon), traversal, views, security, ZODB/ZEO, catalog, component architecture, buildout

Facts:

• 50-150 rps per core, fully dynamic and personalized, no caching
• 50,000+ lines of code (including tests)
• 1,500+ unit tests, 8-10 seconds
• startup time: <2 seconds
• 946 MB ZODB +11.8 GB blobs, 3,700 users, 736 communities, 90k documents
• <600MB RAM

KARL project

• 13 committers, 8 organizations
• LaunchPad: 708 total issues, 35 milestones, 59 UAT, 5 evaluations, 3 migrations
• Zero open bugs
• Hosting by Six Feet Up
• Web Site: www.karlproject.org

Then Calvin talks about hosting which he started with KARL 2. He explains what the difference is and how deployment now is much easier because the framework is smaller.

They are running it on Xen and CentOS. It’s fully managed and monitored. Of course you can also run KARL yourself on your server.

KARL 3.0 migration

• Plone-based KARL 2 to the BFG-based KARL 3
• 4.7 GB of data and 62,000 pieces of content
• Created dummy content to baseline and test the migration process
• Using a GenericSetup to export XML content and built a one time content importer for KARL 3
• Migrated users group information stored in Postgres to KARL 3
• Both systems were ZODB backed, but the newer KARL system included BLOB support
• Other pilots were migrated a few weeks later

Recent & Future KARL Hosting

• Moved to new datacenter in Q3 2009 (more robust)
• Disaster Recovery Support Q1 2010
• Online Realtime Backup Q1 2010

He also mentioned EC2 but there you are dependant on Amazon’s infrastructure.

The path forward:

• On-going software development
• Development of governance model to make available to both large and small organizations
• The importance of “soft” technology and methodologies

As the site is down, here are the streams:

Main Hall

Live TV : Ustream

Eötvös Room

Live video chat by Ustream

Than Room

Live video chat by Ustream

Ortvay Room

Streaming .TV shows by Ustream

Link

• Main Hall
• Eötvös Room
• Than Room
• Ortvay Room

One repeatedly upcoming problem is what identifier a normal user would use for identifying his or her own user account. One approach is apparently OpenID in which case it is a HTTP URL. For the one or other reason though OpenID has not the success among non-geeks as one would wish for. This led to some people thinking about different ways for an identifier scheme to identify yourself on the web. And the thinking was: “People do think of their email addresses as personal identifier, so why not use those?”.

One result of this was the E-mail-Address-to-URL-Translation, in short EAUT. The idea was to these esp. to OpenIDs.

Now another idea is born at Google: WebFinger. It’s goal is a bit more than just providing an easy to understand identifier in that it also aims to retrieving information about the user, like the finger protocol did back in the days. It’s chaining together some upcoming protocols, like XRD and the proposed acct: URI scheme.

How does a WebFinger identifier look like?

It’s like an e-mail address but with an additonal acct: in front:

acct:user@examplehost.com

A WebFinger client would then use a discovery mechanism known as Link based resource descriptor discovery (LRDD) to obtain more information about this user which comes as an XRD file and most likely will point to other services a user might have data stored at (like a profile).

The acct: scheme is here used to make it a valid URI again because mailto: is not semantically correct.

While I personally do not think that inventing a new email-based identifier is solving the problem of user’s confusion on what their identifier is, I nevertheless implemented WebFinger support in the more general discovery library called pydataportability.discovery. More on the issues I see with it maybe in a different blog post.

pydataportability.discovery

pydataportability is (or going to be) a set of libraries for handling various protocols used in the open web/distributed social networking/data portability world. It contains right now basic support for Microformats (hCard and XFN), includes an XRDS-Simple parser and from today on also support for parsing XRD files as well as for discovery (which is going from a URI of a resource like document to the actual resource description file marked up in XRD).

Today I released the following components:

• pydataportability.xrd is an XRD parser
• pydataportability.model.resource models a Resource Description document and is used by the above component
• pydataportability.discovery is implementing hostmeta discovery for now and also has support for WebFinger’s acct: scheme as well as it’s “http://webfinger.net/rel/acct-desc” relationship type.

You can find more documentation on the pydataportability homepage.

Be aware though that nearly all used protocols are not stable. XRD, WebFinger and the discovery stack are much likely to change soon and thus the state of the library might not match the specifications anymore. Moreover this really is a prototype and some structures and names are also likely to change in the future.

How to install and use it

Luckily Google and Yahoo already have implemented support for WebFinger although you need to enable it for your account (at least in Google’s case, I haven’t tested it with Yahoo yet).

After installing pydataportability.discovery also provides two commands for performing discovery. This is really just an example though as the main use of this to include it in your application and do something else with the data retrieved. These commands are called “discover” and “webfinger”.

I recommend to install pydataportability inside a virtualenv. You can do so like follows (I assume you have easy_install installed):

sudo easy_install virtualenv
virtualenv pydp_env --no-site-packages
cd pydp_env
bin/easy_install pydataportability.discovery

discover command to perform discovery on any XRD enabled resource (but only via hostmeta at the moment). Or you can use the webfinger command to perform discovery on WebFinger identifiers:

$ bin/webfinger acct:mrtopf@googlemail.com
Rels:  ['http://portablecontacts.net/spec/1.0']
URIs: ['http://www-opensocial.googleusercontent.com/api/people/']
Media Types: []
Prio: 0
Templates: []

But as said before, this is really just a proof of concept. On how to use this in your own application please
check out the pydataportability.discovery documentation. You will also find out in there on how you can provide your own plugins to control the discovery workflow yourself.

More information about WebFinger

You can read more about WebFinger here:

• The Project Page on Google Code
• The Mailing List
• Eran introduces WebFinger
• Eran on implementing WebFinger