blog entries (Logilab.org) RSS Feed

EuroPython 2009

Nicolas Chauvat — 2009-07-07T12:28-01:00

Once again Logilab sponsored the EuroPython conference. We would like to thank the organization team (especially John Pinner and Laura Creighton) for their hard work. The Conservatoire is a very central location in Birmingham and walking around the city center and along the canals was nice. The website was helpful when preparing the trip and made it easy to find places where to eat and stay. The conference program was full of talks about interesting topics.

I presented CubicWeb and spent a large part of my talk explaining what is the semantic web and what features we need in the tools we will use to be part of that web of data. I insisted on the fact that CubicWeb is made of two parts, the web engine and the data repository, and that the repository can be used without the web engine. I demonstrated this with a TurboGears application that used the CubicWeb repository as its persistence layer. RQL in TurboGears! See my slides and Reinout Van Rees' write-up.

Christian Tismer took over the development of Psyco a few months ago. He said he recently removed some bugs that were show stoppers, including one that was generating way too many recompilations. His new version looks very promising. Performance improved, long numbers are supported, 64bit support may become possible, generators work... and Stackless is about to be rebuilt on top of Psyco! Psyco 2.0 should be out today.

I had a nice chat with Cosmin Basca about the Semantic Web. He suggested using Mako as a templating language for CubicWeb. Cosmin is doing his PhD at DERI and develops SurfRDF which is an Object-RDF mapper that wraps a SPARQL endpoint to provide "discoverable" objects. See his slides and Reinout Van Rees' summary of his talk.

I saw a lightning talk about the Nagare framework which refuses to use templating languages, for the same reason we do not use them in CubicWeb. Is their h.something the right way of doing things? The example reminds me of the C++ concatenation operator. I am not really convinced with the continuation idea since I have been for years a happy user of the reactor model that's implemented in frameworks liked Twisted. Read the blog and documentation for more information.

I had a chat with Jasper Op de Coul about Infrae's OAI Server and the work he did to manage RDF data in Subversion and a relational database before publishing it within a web app based on YUI. We commented code that handles books and library catalogs. Part of my CubicWeb demo was about books in DBpedia and cubicweb-book. He gave me a nice link to the WorldCat API.

Souheil Chelfouh showed me his work on Dolmen and Menhir. For several design problems and framework architecture issues, we compared the solutions offered by the Zope Toolkit library with the ones found by CubicWeb. I will have to read more about Martian and Grok to make sure I understand the details of that component architecture.

I had a chat with Martijn Faassen about packaging Python modules. A one sentence summary would be that the Python community should agree on a meta-data format that describes packages and their dependencies, then let everyone use the tool he likes most to manage the installation and removal of software on his system. I hope the work done during the last PyConUS and led by Tarek Ziadé arrived at the same conclusion. By the way, Martijn is the lead developer of Grok and Martian.

Godefroid Chapelle and I talked a lot about Zope Toolkit (ZTK) and CubicWeb. We compared the way the two frameworks deal with pluggable components. ZTK has adapters and a registry. CubicWeb does not use adapters as ZTK does, but has a view selection mechanism that required a registry with more features than the one used in ZTK. The ZTK registry only has to match a tuple (Interface, Class) when looking for an adapter, whereas CubicWeb's registry has to find the views that can be applied to a result set by checking various properties:

interfaces: all items of first column implement the Calendar Interface,
dimensions: more than one line, more than two columns,
types: items of first column are numbers or dates,
form: form contains key XYZ that has a value lower than 10,
session: user is authenticated,
etc.

As for Grok and Martian, I will have to look into the details to make sure nothing evil is hinding there. I should also find time to compare zope.schema and yams and write about it on this blog.

And if you want more information about the conference:

photos tagged europython in flickr and gallery on the website
blogs reporting on europython
recorded talks and presentation materials

Semantic web technology conference 2009

Sandrine Ribeau — 2009-06-22T10:15-01:00

The semantic web technology conference is taking place every year in San Jose, California. It is meant to be the world's symposium on the business of semantic technologies. Essentially here we discuss about semantic search, how to improve access to the data and how we make sense of structured, but mainly unstructured content. Some exhibitors were more NLP oriented, concepts extraction (such as SemanticV), others were more focused on providing a scalable storage (essentially RDF storage). Most of the solutions includes a data aggregator/unifier in order to combine multi-sources data into a single storage from which ontologies could be defined. Then on top of that is the enhanced search engine. They concentrate on internal data within the enterprise and not that much about using the Web as a resource. For those who built a web application on top of the data, they choosed Flex as their framework (Metatomix).

From all the exhibitors, the ones that kept my attention were The Anzo suite (open source project), ORDI and Allegrograph RDF store.

Developped by Cambridge Semantics, in Java, Anzo suite, especially, Anzo on the web and Anzo collaboration server, is the closest tools to CubicWeb, providing a multi source data server and an AJAX/HTML interface to develop semantic web applications, customize views of the data using a templating language. It is available in open source. The feature that I think was interesting is an assistant to load data into their application that then helps the users define the data model based on that data. The internal representation of the content is totally transparent to the user, types are inferred by the application, as well as relations.

I did not get a demo of ORDI, but it was just mentionned to me as an open source equivalent to CubicWeb, which I am not too sure about after looking at their web site. It does data integration into RDF.

Allegrograph RDF store is a potential candidate for another source type in CubicWeb . It is already supported by Jena and Sesame framework. They developped a Python client API to attract pythonist in the Java world.

They all agreed on one thing : the use of SPARQL should be the standard query language. I quickly heard about Knowledge Interface Format (KIF) which seems to be an interesting representation of knowledge used for multi-lingual applications. If there was one buzz word to recall from the conference, I would choose ontology :)

IPMI plugin for Munin python code published

Arthur Lutz — 2009-06-17T20:17-01:00

As you might have noticed we quite like munin. We use it quite a bit to monitor how our servers and services are doing. One of the things we like about munin is obviously that the plugins can be written in python (and perl, bash and ruby).

On a few recent servers we started playing with IPMI to sensor the temperature, watts, fan's rpms etc. So we went out looking for a munin plugin for that. We found Peter Palfrader's ruby plugins. There was one small glitch though, we came across a simple bug : the "ipmitool -I open sensor" can be real long to execute on certain machines, so configuring the plugin was a bit painful and running it too. Changing the ruby code was a bit tricky since we don't really know ruby... so we did a quick rewrite of the plugin in python... with a few optimizations.

It's not really complete but works for us, and might be useful to you, so we're publishing the hg repo. You can get the tgz or browse the source.

Google I/O 2009

Sandrine Ribeau — 2009-06-10T10:05-01:00

The big event of the conference was the annoucement of Google Wave, a new online communication and collaboration tool, built on the Google Web Toolkit (GWT). Another big thing the GWT, lots of application built with it, a delightful tool for Java developers.

It was interesting to see that Google App Engine (GAE) will shortly provide an API to do offline processing, with objects called Task Queue. Task queue are web hooks, tasks are pushed to the server, queued and they are pushed until the task is executed (which overpass the annoying well-known time out issue with Google App Engine). It was introduced as asynchronous, enabling low latency, reliable and scalable (where are the buzz words?). Of course, this news has been a huge relief for most of the developers in the public as that was a big missing part from GAE.

An nice presentation from the founders of FrontSeat.org about their current project Walk score. They introduced themselves as civic software developers, writing software in a civic manner. They explained how they use GAE, and why they had to use Amazon EC2 to compensate GAE gaps. The gaps they listed here were the ranking non-ability of GAE, the too long reponse time for such computation they do, the fact that no cron jobs can be done (the arrival of Task Queue might change their opinion).

All the sessions have been recorded and are available here.

And yes, as all the participants of this conference, I went back home with an Android phone :)

hgview 1.0.0 released!

David Douard — 2009-06-08T09:53-01:00

I am pleased to introduce you to the latest kid of the Logilab team: hgview 1.0.0.

hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

This version is an almost complete rewrite of hgview 0.x which had two GUI backends, gtk and qt4. This 1.0 release drops the gtk backend (we may consider reintroducing it, we haven't decided yet... by the way, patches are always welcome). Some may not like this choice, but the immediate benefit of using qt4 is that hgview works like a charm on MacOS X systems.

Edit: there was a bug in hgview 1.0.0 on Ubuntu hardy. It's now fixed, and I've uploaded a 1.0.1 version deb package for hardy.

Features

4 different viewers:
- repository navigator that displays the graphlog efficiently (works well with 10,000 changesets),
- filelog navigator that displays the filelog of a file (follows files through renames),
- filelog diff navigator that displays the filelog in diff mode to easily track changes between two revisions of a file,
- manifest viewer that navigates in the files hierarchy as it was at a given revision.
Each viewer offers:
- easy keyboard navigation:
  - up/down to change revision,
  - left/right to change file (for the repo navigator only),
  - return to display the diff viewer of the selected file,
- search quickbar (Ctrl+F or /): search in graphlog (search as you type in the currently displayed file or diff, plus a cancellable background search in the revision tree),
- goto quickbar (Ctrl+G): go to the given revision (accepts id or tag, with completion for tags),
- navigation history: alt+left/alt+right to navigate backward/forward in the history,
can be used alone or as a hg extension,
can be configured using standard hg rc files (system, user or per repository),
possibility to declare users (with multiple mail addresses) and assign them a given color to make a given user look the same in all your repositories,

Download and installation

The source code is available as a tarball, or using our public hg repository of course.

To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

hgext.hgview=/path/to/hgview/hgext/hgview.py

Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.

The Web is reaching version 3

Nicolas Chauvat — 2009-06-05T20:32-01:00

I presented CubicWeb at several conferences recently and I used the following as an introduction.

Web version numbers:

version 0 = the internet links computers
version 1 = the web links documents
version 2 = web applications
version 3 = the semantic web links data [we are here!]
version 4 = more personnalization and fix problems with privacy and security
... reach into physical world, bits of AI, etc.

In his blog at MIT, Tim Berners-Lee calls version 0 the International Information Infrastructure, version 1 the World Wide Web and version 3 the Giant Global Graph. Read the details about the Giant Global Graph on his blog.

Nous allons à PyConFr 2009

Arthur Lutz — 2009-06-05T19:57-01:00

Le 30 et 31 mai prochain (samedi et dimanche prochain) nous allons être présents à PyConFr édition 2009, nous sommes partenaire de l'évènement et allons y parler de CubicWeb. Pour être plus précis, Nicolas Chauvat y présentera "CubicWeb pour publier DBpedia et OpenLibrary". Il avait déjà évoqué ces sujets sur ce site : Fetching book descriptions and covers et DBpedia 3.2 released.

Si vous comptez y aller, n'hésitez pas à venir nous dire bonjour.

Almost reached 1000 tickets

Arthur Lutz — 2009-05-25T11:02-01:00

Logilab.org has almost reached a thousand tickets on the Logilab's open source projects. To be exact there are 940 tickets right now. What kind of tickets are they ?

Here is a quick graph of the state of the tickets in our tracker :

Graphing is neat. Maybe soon we'll get this kind of feature automatically in the CubicWeb forge, see this ticket.

Reading SPE files

Andre Espaze — 2009-05-11T16:23-01:00

If you would like to read SPE files from charge-coupled device (CCD) cameras, I have contributed a recipe to the SciPy cookbook, see Reading SPE files.

Fetching book descriptions and covers

Nicolas Chauvat — 2009-07-02T12:43-01:00

We recently added the book cube to our intranet in order for books available in our library to show up in the search results. Entering a couple books using the default HTML form, even with the help of copy/paste from Google Book or Amazon, is boring enough to make one seek out other options.

As a Python and Debian user, I put the python-gdata package on my list of options, but quickly realized that the version in Debian is not current and that the books service is not yet accessible with the python gdata client. Both problems could be easily overcome since I could update Debian's version from 1.1.1 to the latest 1.3.1 and patch it with the book search support that will be included in the next release, but I went on exploring other options.

Amazon is the first answer that comes to mind when speaking of books on the net and pyAWS looks like a nice wrapper around the Amazon Web Service. The quickstart example on the home page does almost exactly what I was looking for. Trying to find a Debian package of pyAWS, I only came accross boto which appears to be general purpose.

Registering with Amazon and Google to get a key and use their web services is doable, but one wonders why something as common as books and public libraries would have to be accessed through private companies. It turns out Wikipedia knows of many book catalogs on the net, but I was looking for a site publishing data as RDF or part of the Linked Open Data initiative. I ended up with almost exactly what I needed.

The Open Library features millions of books and covers, publicly accessible as JSON using its API. There is even a dump of the database. End of search, be happy.

Next step is to use this service to enhance the cubicweb-book cube by allowing a user to add a new book to its collection by simply entering a ISBN. All data about the book can be fetched from the OpenLibrary, including the cover and information about the author. You can expect such a new version soon... and we will probably get a new demo of CubicWeb online in the process, since all that data available as a dump is screaming for reuse as others have already found out by making it available as RDF on AppEngine!

iclassmethod decorator to define both a class and an instance method in one go

Sylvain Thenault — 2009-05-11T16:06-01:00

You'll find in the logilab.common.decorators module the iclassmethod decorator which may be pretty handy in some cases as it allows methods to be both called as class methods or as instance methods. In the first case the first argument will be the class and the second case it will be the instance.

Example extracted (and adapted for simplicity) from CubicWeb:

from logilab.common.decorators import iclassmethod

class Form(object):
  _fields_ = []

  def __init__(self):
      self.fields = list(self._fields_)

  @iclassmethod
  def field_by_name(cls_or_self, name):
      """return field with the given name and role"""
      if isinstance(cls_or_self, type):
          fields = cls_or_self._fields_
      else:
          fields = cls_or_self.fields
      for field in fields:
          if field.name == name:
              return field
      raise Exception('FieldNotFound: %s' % name)

Example session:

>>> from logilab.common import attrdict
>>> f = Form()
>>> f.fields.append(attrdict({'name': 'something', 'value': 1})
>>> f.field_by_name('something')
{'name': 'something', 'value': 1}
>>> Form.field_by_name('something')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 15, in field_by_name
Exception: FieldNotFound: something

So we get a field_by_name method which will act differently (actually use different input data) when called as instance method or as class method.

Also notice the attrdict trick that can also be achieved with the Python 2.6 named tuple.

12 ans de l'April: bilan et objectifs

Nicolas Chauvat — 2009-04-06T11:45-01:00

L'April fête cette année ses douze ans. L'association de promotion et de défense du logiciel libre approche maintenant les 5000 membres, toutes catégories confondues: personnes physiques, entreprises commerciales, collectivités, associations. Elle vient de publier son rapport moral 2008 et sa feuille de route 2014 qui fixe des objectis pour les cinq années à venir. On notera en particulier la lutte contre les quatre dangers que sont les brevets sur les algorithmes, les dispositifs de contrôle d'usage, la vente liée et l'informatique déloyale. Consultez le site de l'April pour en savoir plus sur ses actions et n'hésitez pas à adhérer ou à donner quelques heures de votre temps.

Emacs and mercurial trick

Ludovic Aubry — 2009-06-17T20:20-01:00

While using emacs I always find the need to use grep on a terminal to search for things within a specific project. This is not ideal, even within an embedded emacs shell.

Since I recently discovered the emacs command grep-find I decided to make it work nicely with mercurial projects, and here's the result:

(defun grep-project (s)
  (interactive "sSearch project for: ")
   (grep-find (concat "files=(`hg manifest`);grep -nH -e \"" s "\" ${files[@]/#/`hg root`/}"))
)
(global-set-key (quote [f4]) (quote grep-project))

Venez nous rendre visite à Solution Linux 2009

Arthur Lutz — 2009-04-03T13:30-01:00

Nous sommes dès ce matin, pendant 3 jours, présents au salon Solutions Linux 2009 au stand du pôle de compétition System@tic dont nous faisons parti. C'est le stand B4/B8, assez prêt de l'entrée sur la gauche (détails).

Nous allons présenter CubicWeb à plusieurs reprises sur le stand, ainsi que lors des conférences sur le Web2 ce mardi 31 mars de 14h à 17h30 :

Adrien présentera "Simile Widgets, des composants de haut niveau pour IHM web"
Sylvain présentera "Cubic 3.0 - une plateforme pour les applications web sémantique"

pour plus de détails consultez le programme.

Pylint at BayPIGgies

Sandrine Ribeau — 2009-03-31T09:35-01:00

I am pleased to announce that Pylint was presented during a Tools night meeting organized by BayPIGgies on thursday march 26th. This meeting has been recorded and you can enjoy the video.

One point was missing from the presentation and I'll take the opportunity now to mention it. Flymake, an on-the-fly syntax checker for GNU Emacs which has been discussed, does work in combination with Pylint (please see EmacsWiki for more informations).

photo by ten safe frogs under creative commons