Protractile

Cross compiling a Gentoo

2013-02-23T00:00:00+01:00

Note

This article is about cross compiling a Gentoo/Funtoo not building a Rasberry Pi Gentoo/Funtoo, if what you want is a Gentoo on your raspberry follow the white rabbit on wiki.gentoo.org.

If a machine can run a GNU/Linux you don't need to do cross-compilation but if the machine is not powerful enough you will still want to compile the binaries in another more powerful machine which means building a cross-compilation toolchain. Gentoo provides a script to do that backed by Portage, it builds a GCC compiler that run on your machine which is called the host and builds binaries for the other machine called the target. But no all sources can be cross compiled, some do, some don't, you need a specific configuration to be able to cross compile something and sometime it will just not compile like Python (I only tested 2.7.3). So instead what can be done is fake that we compile on the target so that the build process operate as if it was on the target machine, this can been done with a a qemu chroot or a proot <http://proot.me/> qemu and a bit of portage tweaking ;). This methods divide nearly by 6 the time needed to compile the kernel on my quad core i5.

Here what I propose it to build a Funtoo, Portage backed distribution for Raspberry Pi with x86_64 as host. It basicaly goes like this:

Build crossdev, proot and qemu with support for the target in this case arm
Build cross compiler on x86_64 and cross environnement for armv6j
Emerge @system

At this stage, you will have what is called a «stage 3», you can then emerge whatever you want and use the target directory as the base of a root filesystem of any armv6j backed machine.

Ninja cli dance

Grab a hot cup of coffee and first emerge crossdev, create a dummy overlay for it, call it crossoverlay don't forget to add it as first path in PORTDIR_OVERLAY of your make.conf.

While you are in make.conf you can add the following:

QEMU_SOFTMMU_TARGETS="arm"
QEMU_USER_TARGETS="arm"

You can now emerge proot and qemu:

emerge -q proot qemu

qemu is an emulator and proot some kind of improved-with-powerful-magic chroot.

Bootstrap the crosstools and target root filesystem with crossdev:

crossdev -S -s4 arm-hardfloat-linux-gnueabi

Now prepare the cross-environnement:

cd /usr/arm-hardfloat-linux-gnueabi/usr/
cp -r /usr/portage .
mkdir -p share/portage/config/sets
cp /usr/share/portage/config/make.globals /usr/arm-hardfloat-linux-gnueabi/usr/share/portage/config/
cd /usr/arm-hardfloat-linux-gnueabi/var
git clone git://github.com/amirouche/funtoo-overlay.git amirouche
cd /usr/arm-hardfloat-linux-gnueabi/etc
rm make.profile
ln -s ../var/amirouche/profiles/raspberrypi make.profile
emerge-arm-hardfloat-linux-gnueabi -q binutils
emerge-arm-hardfloat-linux-gnueabi -q glibc
emerge-arm-hardfloat-linux-gnueabi -q gcc
emerge-arm-hardfloat-linux-gnueabi -q portage
emerge-arm-hardfloat-linux-gnueabi -qO libtool

Portage will fail it's not significant. Open /usr/arm-hardfloat-linux-gnueabi/etc/portage/make.conf and comment the ROOT, PKGDIR, PORTAGE_TMPDIR, PKG_CONFIG_PATH, CBUILD and their should only be /host-rootfs/var/amirouche in PORTDIR_OVERLAY.

Proot needs libgcc_s.so.1 to work:

cd /usr/arm-hardfloat-linux-gnueabi
cp usr/lib/gcc/arm-hardfloat-linux-gnueabi/4.6.3/libgcc_s.so.1 lib

Until Python is compiled inside the new environnement we will use the host Python:

cd /usr/arm-hardfloat-linux-gnueabi/usr/bin
ln -s ../../host-rootfs/usr/bin/python2.7
ln -s ../../host-rootfs/usr/bin/python2.7 python

Change PS1 in the chroot to prevent errors in other consoles:

echo "export PS1=\"(qemu proot arm-hardfloat-linux-gnueabi) \$PS1\"" >> /usr/arm-hardfloat-linux-gnueabi/etc/bash/bashrc

Now proot in /usr/arm-hardfloat-linux-gnueabi:

cd /usr/arm-hardfloat-linux-gnueabi
proot -r . -w / -Q qemu-arm /bin/bash

You can now compile using the target compiler inside the proot environnement:

emerge -q @system

Is not that easy but I'm sure you will succeed at solving the issues that may arise ;)

Create an archive and keep it safe somewhere, exit the target environnement and execute on the host the following command:

cd /usr/
tar -pczf arm-hardfloat-linux-gnueabi.tar.gz arm-hardfloat-linux-gnueabi/
mv arm-hardfloat-linux-gnueabi.tar.gz /home/

Your armv6j stage 3 is ready!

TODO

Create a qemu image
Continue the install take inspiration from gentoo's Raspberry Pi Quick Install Guide
Setting up distcc
Make patches to make emerge @system working with almost empty tree.
Investigate why overriding cross environnement PATH and using the cross compiler will fail to properly build some softwares (e.g. eix)
Patch crossdev to support custom profiles
Setup a cross root environnement from a stage 3
Patch metro to support this kind of build

Installer Funtoo

2013-02-16T00:00:00+01:00

Funtoo est un fork gentil de Gentoo. Gentoo est la distribution source, par excellence, c'est à dire que tout est installer depuis les sources des logiciels. La compilation depuis les sources permet d'avoir un OS plus puissant car les binaires peuvent être optimisés pour la machine. L'autre chose qui rend une Gentoo puissante sont les drapeaux d'utilisation aka. «USE flags» qui permettent de configurer la compilation des logiciels afin de n'inclure que ce que dont vous avez besoin et donc encore une fois rendre votre OS plus rapide. Un autre interet c'est la disponibilité quasi immediate dans les recettes pour compiler le dernier logiciel à la mode, même si il est pas encore stabilisé.

Ce qui fait sa force fait aussi sa faiblesse: la compilation ça prend du temps. L'autre difficulté est la mise à jour des logiciels de bases GCC, glibc mais qui a dit que les mise à jours de distribution fonctionnaient ?

Si vous voulez découvrir ce qui fait un distribution GNU/Linux c'est l'idéal. Si vous voulez découvrir comment on fait une distribution il faut se tourner vers LinuxFromScratch pour laquelle il doit y a voir une traduction en français.

Funtoo est un fork realisé par l'initateur du projet Gentoo, où ont lieu des modifications ou vont avoir lieu des modifications majeurs, pour le moment il y a:

Metro, un logiciel qui permet de construire une distribution basé sur Portage (le gestionnaire de paquet de la Gentoo) utilisé par Gentoo et Funtoo
Gestion des profiles amélioré avec la possibilité d'ajouter des profiles Mix-In qui sont des additions à n'importe quelles autre profiles. Un profile Portage renseigne un certains nombres d'informations sur la distribution, drapeaux d'utilisation de base, masques pour les recettes pour eviter l'utilisation de recettes non stables et ceci pour différentes plateformes (x86, x86_64, arm, ppc etc...) et pour différente utilisation (desktop, server, workstation). Les MixIn vont permettre de manière transversale d'ajouter des fonctions à votre profil (Gnome, Kde, audio, video etc...) pour simplifier la configuration de votre OS.
Les mise à jours de la configuration Grub automatique avec un fichier de configuration pour les humains ;)
Des scripts réseaux améliorés
L'overlay de la distribution, ensemble de recettes de compilation et profiles sont synchronisé avec celui de Gentoo

Funtoo utilise aussi github et donc git pour herbergé l'overlay.

Pour Funtoo il y a une documentation en français aussi, ainsi qu'un guide de demarrage.

«On the seventh day, God saw the world had already been outdated, so he did a god@heaven# eix-sync && emerge -uDNav @world && exit42» ~ Genesis on Gentoo

AdminNext: A new API

2013-01-05T00:00:00+01:00

Following the first post about AdminNext, this posts dives further the idea of another API to build an admin area.

In a previous version of this article I tried to dive the user features API idea, it wasn't that interesting. The part about user features that I still find somewhat interesting resulted in an article with the same name.

Model centric

In the previous post I was talking about a groups centric admin, this was badly done. The story goes like this:

I was thinking about the admin and discovered that it was model centric
Why is it model centric ?
Can it be something else ?
What would it be ?

At the same time I was wondering about the fact that the admin API was bare metal the features it provides and the code that generate it which lead me to the user feature thing and composite views as way to abstract the features each view provides.

Putting all this together, I'm looking for an API that meets this goals:

non-model centric
focused on user

This lead me to the group-centric admin, because:

groups relate to users
groups relate to permissions
permissions relate to actions

And groups can relate to features without requiring them to be linked to an a permission. For instance an user can be part of group because he is part of community and as such require a particular treatment like say I subscribe to a subreddit, I get it on frontpage, the subreddit is available to everybody but as a member of the subreddit I can access it easily.

Group is in a gray area, it's a user feature but also a code feature, so I was back at code as a rail for thinking which was exactly what I didn't want to do.

Even more mistakes, in my previous attemps to draw a new API, I focused too much on groups as UI or how an admin that use more the group concept would look like so I had trouble to find a coherant API, while now it seems to me that I should have started from permissions.

Another thing, current django admin does rely on permissions and groups to customize the experience of each user, but this not done in way that can make the experience better, at least what I think is better, like it's explained in the following part.

Permission centric API

Like I said above, before I focused too much on groups because I think it's central in an admin experience, but the foundation of a group is users of course, but in the point of view of code it's permissions.

API overview

What ever the admin there is:

menu or several menus
dashboards
navigate, explore, filter models
CRUD operations

And the most important thing the ability to stay focused on the task which can be achived through separation of concerns.

API wireframe

About writring APIs

When I build an API I start from the user part of the API, I write things like I would like to write them myself. This also translates in me doing the same thing when I actually write the code which means I focus on usability before focusing on features which for instance lead me to write a declarative API but without any means to do the same thing otherwise. So writring the API is a way to get an idea of the why and what but not the how which needs further thinking.

Context

The site for which I will mock an API is named «Cabernet Cheese Chop». The website is a frontend for consumers but also a social network for cheese enthusiasts and allows each shop owner to run their boutique. Notable features are:

Cheese, wine, bread and butter spotlight, it's somekind of magazine for all things related to the restaurant suite.
A social network in which consumers and owners interacts together sharing and caring for the finest tastes...
ERP thing that allows to manage supplies, keep in touch with their local community and everything that needs to be done to keep a business running.

Of course all this is inter-mixed in an awesome way so that the experience is awesome.

The example is in purpose federation-like, a big entity and many related sub entities, so that it match most websites while still being possible to relate to simple websites where there is only one group of people running one business.

Writring the backend

What we are interested in here is to write a backend that allows every people taking part in this entreprise to deal with their matters. I won't write every backend features so I will focus on the following:

Local community manager answers questions regarding the local shop, gives a hand in the magazine and also keeps an eye over the social network.

This is already a bit complex since there is three user stories only for one job.

Without further ado let's write the class that will represents the local shop support story:

class LocalShopSupport(Feature):

    verbose_name = 'shop support'

    includes(SupportTicketExplore)
    includes(SupportTicketAnswer)

    dashboard = (IncomingTickets, SupportTicketFeedback)

    def filter_queryset(self, queryset, user):
         return queryset.filter(shop=user.shop)

Except it uses a syntax that is not supported by Python 3, it draws a canvas for how one can define backend features. Permissions don't appear in this Feature because it they are inherited from the includes. Let's define the includes:

class SupportTicketExplore(Feature):

    ChangeList = SupportTicketChangeList

    # inherited from SupportTicketChangeList
    # permissions = ['support.ticket.can_read',]

class SupportTicketAnswer(Feature):

    actions = (TicketAnswer, RequestForFeedbackOnTicket)

    # inherited from TicketAnswer
    # permissions = ['support.ticket.can_answer',]

We can go on and on with more classes but everything could have been done in one class:

class LocalShopSupport(Feature):

    model = support.Ticket

    class ChangeList(BaseChangeList):
        list_display = ['title', 'kind']
        filter = ['state', 'kind']
        order_by = ['submit_date', 'last_seen', 'last_answer']

        permissions = ['support.ticket.can_read']

    class TicketAnswer(ModelChangeView):
        verbose_name = 'anwser'
        form = AddCommentForm
        template_name = 'support/tickets/add_comment.html'

        permissions = ['support.ticket.can_anwser']

    class RequestForFeedbackOnTicket(ModelAction):

        in_bulk = True

        def action(self, request, queryset):
            queryset.update(state=Ticket.STATE.REQUEST_FOR_FEEDBACK)

    actions = (TicketAnswer, RequestForFeedbackOnTicket)

    class IncomingsTickets(ObjectListDashBoard):
        verbose_name = 'incomings tickets'

        def filter_queryset(self, queryset, user):
            queryset = super(IncomingsTickets, self).filter_queryset(queryset, user)
            # the queryset is already filtered for this user
            queryset.filter(state__in=(Ticket.STATE.NEW, Ticket.STATE.NEEDS_REPLY))
            return queryset

    class SupportTicketFeedback(ObjectListDashBoard):
        verbose_name = 'feedback'

        def queryset(self):
            queryset = super(IncomingsTickets, self).filter_queryset(queryset, user)
            return queryset.filter(state=Ticket.STATE.REQUEST_FOR_FEEDBACK).feedback_set.all()

    dashboard = (IncomingTickets, SupportTicketFeedback)

    def filter_queryset(self, queryset, user):
         queryset = super(LocalShopSupport, self).filter_queryset(queryset, user)
         return queryset.filter(shop=user.shop)

How subclasses of Feature are pulled together is more simple to explain, when the application boots it discover somehow all the Feature classes, gather the needed permissions for each features, given that informations it will match features to regroup them based on groups' permissions. If a feature is a subset of another, the feature that has more permissions (and probably features) will be associated with the group. At the end all features should be associated with groups if it's not the case then the remaining features are again matched against the groups but only considering the subset of remaining features. This process continue until there is no more lonely features or features can not be matched with a group. In this case, the feature will only be available in superuser tab.

« Experienced programmers don't use design patterns. Design patterns use them. » Lunivore in Design patterns - do you use them?

User features

2013-01-05T00:00:00+01:00

This doen't look like a widespread idea to say the least. I'm not sure the wording express my feeling, a definition might help:

An user feature is a particular ability an user has in his or her use of a software

Examples of user features follow:

Ability to explore, filter, navigate the objects related to a particular task
Ability to edit objects in the context of a particular task
Ability to have and give feedback about a particular action or task or group of tasks
Ability to have an overview regarding the state of a project
Ability to move to another task quickly
Ability to customise UX/UI
Ability to not loose sight of the current task

Those are a bit generic and grow out of my reflexion about django admin. I will try something else:

Ability to pay for something

I'm stuck! Maybe this is not an interesting concept. The thing that looks to be the most related is user stories. Here is the abstract definition from Wikipedia:

In software development and product management, a user story is one or more sentences in the everyday or business language of the end user or user of a system that capture what a user does or needs to do as part of his or her job function. User stories are used with Agile software development methodologies as the basis for defining the functions a business system must provide, and to facilitate requirements management. It captures the 'who', 'what' and 'why' of a requirement in a simple, concise way, often limited in detail by what can be hand-written on a small paper notecard. User stories are written by or for the business user as that user's primary way to influence the functionality of the system being developed. User stories may also be written by developers to express non-functional requirements (security, performance, quality, etc.),[1] though primarily it is the task of a product manager to ensure user stories are captured.

From User story article from Wikipedia

Emphase is mine.

Example user stories are:

I can browser stuff listing and click buy to add it to the shopping cart
I can browser recent comments and click delete

Based on this definition, user features are generic stories or abstract stories. which allows to think on a higher level and probably find other patterns and code abstractions.

Django AdminNext

2013-01-01T00:00:00+01:00

In an attempt to chime into Django developement but first because I was interested by the project and that I think I have a solution for Django Admin. It may seem very pretencious and arogant to say that, when I first posted my message on django-developper I didn't knew the revamp was a subject of interest in the Django community until watching some djangocon videos and writring this article.

So instead of trying to build something and then see if it works, I decided to start it all over again, starting by reading Django's guide to contribute and «Advice for new contributors» in particular those got my attention:

Analyze tickets’ context and history
Start small: It’s easier to get feedback on a little issue than on a big one.
If you’re going to engage in a big task, make sure that your idea has support first
Be bold! Leave feedback!

So, that is what follows.

Context

History

Current admin project named new-forms-admin was built primarly to make use of new-forms and add extra customization hooks.

So new-forms-admin was part of the deprecation of old-forms which, I think, was part of the refactoring of models and queryset. I think they most notably decoupled form fields from their html representation, widgets, but I'm not sure.

User features

Django was created in a newspaper and I think that it was primarly meant to be a CMS framework and since the beggining the admin was one of the most powerful feature.

But the web evolved, most notably the web uses evolved, the needs evolved, even «CMS» application evolved, so one can not think about the admin without thinking about Django uses and the web in general.

First there is a lot of applications type CRM, ERP, e-commerce, e-administration (or e-governement), any other business specific application, all this can be summed up in terms:

objects inspections: search, filtering, navigation
input data: crud operations and ordering things
workflows: multi user interactions, wait for X event to happen
permissions: different set of groups for each every operations

But also there is another space more related to the application on its own:

configuration
supervision
reporting

This is different from developpers and sysadmin work, it's more a webmaster work.

Last but not least, «User features» stress the fact that this is related to the user. So if a software wants to offer a framework for building an UI/UX that can be further refined to deal with user matters, a specific business, I think it must be expressed in the terms of users.

Code feature

In what follows I try to gather all dos and doings of Django admin and also what people would like to see in the admin. Except if you are interested by the subject of customizing the admin you can skip it, it is used as a reference for what follows.

Have a look at CMS@DjangoPackages. Have a look at AdminInterface@DjangoPackages.

Admin apps

Grappelli
- Collapsibles
- Inline Sortables
- Sortable Excludes
- Related Lookups
- Autocomplete Lookups
- Using TinyMCE
- Changelist Templates
- Changelist Filters
django-adminplus, easily add custom views to the Django admin cf. #18665
django-admin-tools, see also pops.
- a full featured and customizable dashboard
- a customizable menu bar
- tools to make admin theming easier.
django-djam
- Make it easy to add new workflows (or similar custom admin sections).
nexus

Extensions

Django FileBrowser media-management with Grappelli
django-massadmin bulk edits
django-mptt-treechangelist
django-admin-tabs tabs for model forms
django-sneak custom objects in change list (filter and actions support)
django-ckeditor CKEditor integration
django-guardian implementation of per object permissions
django-memcache-status displays statistics about your memcached instances
django-admin-histograms histograms
django-admin-sortable generic drag-and-drop ordering for objects and tabular inlines
django-mobileadmin
django-admin-filtrate custom filter framework

Skins

Other interesting app

django-tables2 build tables directly from queryset
django-reversion version control
django-appconf
django-dynamiq-search-form create dynamic search forms that generate Q objects.
django-locking it has admin integration
django-forms-builder
django-sitetree

Feature requests

«system for administrators to be notified about specific model instances about which they need to take action» in» Admin notifications

From «Call for ideas: Admin Improvements» 3/31/09, those are a bit old but maybe still current:

Ajax/UI/UX: Autocomplete widget, Drag & Drop, inplace edit
Dynamic filters based on models
Finer Permissions limit the ability of a user right down to the created object
Dynamic add/remove columns for models
Foreign Key filtering
History improvements with undo functionality (this is already done with django-reversion but is an excellent addon)
Mass file/photo upload
Add and remove instances to/from formsets with a button
Photo Management - crop, rotate (with ability to add more filters)
Make it easier for people to create custom themes and styles for the admin and change between them
Allow models to be added twice with two modeladmins serving a different process this is possible with ModelProxy

Hooks

Easily link up custom admin views to admin:index #18665
Include reverse()-able links and off site URLs to admin:index on a per app basis #18665
Object tool customization #18914
Admin Changelist: add app-model_name class to <body> tag #13629
Use short description with properties in the admin #17308
Add more hooks in the ModelAdmin (get_extra, get_max_num) #18388
Allowing inline fields to be specified in the fields tuple #4848
Support nested fieldsets in administration #10590
Support lookup separators in ModelAdmin.list_display #10743

UI/UX

Add support for custom app_label and verbose_name» #3591
Include blocks in admin:index to customize the displaying of each app block #18665
Ability to define the sort order of the apps manually. #18665
Flexible filters for django admin #18736
Add next/previous buttons when editing a model in the admin site #18957
Admin actions displayed as buttons instead of drop-down list #19235
Add option to disable Admin log (LogEntry) #19302
Link to object in message after creating/updating object in django admin #19361
SelectBox.js with grouping (optgroup elements) #13883
Adding support for Autocomplete in contrib.admin #14370
Model description on administrative interface #14587
Admin search should be pluggable #15961
Provide keyboard shortcuts in admin #16521
Improve multiple sort UX #16212
Admin history view should also show history of super-object(s) #16465
Use short description with properties in the admin #17308
Filtering interface on ForeignKey <select> boxes #25
Fix usability issue with limit_choices_to and "Add another" in admin #29
Bi-Directional ManyToMany in Admin #897
Multi-select admin filter for RelatedFields #1873
Adding to admin nice selector for editing ForeignKey fields in case of relatively large number of items #2651
Better OneToOneField handling in Admin Interface desired #2948
Ability to disable admin pagination #4065
Easier way to overwite the admin welcome message #7467
ModelAdmin hook for customising the "show on site" button #8261
Allow overriding of default adminsite. #8500
Generic Foreign Keys should have a nice widget #9976
Show admin actions on the edit pages too #12090
"Tooltips" in admin list view #12701
Add has_module_permission to ModelAdmin #6327
Year navigation in admin's date widgets' calendar #9388

Other

Dogfood class-based views in contrib.admin #17208
Logout link should be protected #15619
FilterSpec implementation limits querying functionality #16142
View collections in the generic views #16213
Enable admin log display to be restricted to a specific site #2901
Some admin Javascript cleanups #4045
Reusable templatetags for admin list #6159
Add databrowse-like functionality to the admin #8936
Nested Inline Support in Admin #9025
Some admin Javascript fixes and cleanups #4045
Add admin.site._registry manipulation methods #9602
ModelAdmin.queryset() is missing a mechanism for specifying different querysets for changelist and change object views #10761
Admin object deletion confirmation page causes server out of memory error #10919
Admin action templates cannot be overriden on a per-app basis #12566
Admin validation doesn't reflect ModelAdmin's extensibility #12674
API for simpler (permission or any) checks for generic view classes #15215

Among many others.

Sum up

Permissions django-guardian, #6327
More object navigation and filtering django-admin-filtrate, django-dynamiq-search-form, #18736, #18957, #19302, #25, #1873, #9388, #16142, #10761
Better support of complex data in ModelAdmin forms django-admin-tabs, #10590, #897, #2651, #2948
Custom objects in change list Django-FileBrowser, django-mptt-treechangelsit, django-sneak, #9976
Geneirc UX/UI, and ajax features django-admin-sortable, edit inplace, #13883, #16521, #16212, #16212, #16465, #17308, #29, #4065, #7467, #12090, #12701
More hooks and configuration django-adminplus, #18665, #18914, #17308, #18388, #4848, #8261, #2901, #6159, #12566
Customizable dashboard and menu bar most CMS, grappelli, django-memcache-status, django-admin-tools

AdminNext

If you did not read the above basicaly that's what I took into account to come up with an idea for AdminNext. I think that the most important tickets are the following:

Dogfood class-based views in contrib.admin #17208
Making it easier to customize Django Admin #18665
View collections #16213
API for simpler (permission or any) checks for generic view classes #15215

Talking about CBV, I just can't find the ticket but there is CBV that wants to talk with several form. I like the idea but it breaks OOP principles. Inheritance is «is a» relation, where composition, done through a property, is «has a» relation, so which one sound better:

ModelAdmin has several forms
ModelAdmin is a form aggregation

The second sound akward and I tried my best not to use the word «composition».

I should have probably entitled this paragraph «cbv-admin» but then it wouldn't have been a surprise to say that it is an almost solved problem. I say almost because it's quiet a bit of code but there is a lot of tests to have it checked for both backward compatibility and progress.

I also say almost because it's only supposed that CBV will solve the extensibility problem, I personnaly have no hard evidence of that, except maybe Mixins.

What was proposed in Splitting out admin functionality is really interesting too, but this will not benefit the admin, per se.

The problem is that I try to tackle it bottom-up, instead of up-bottom. Like I said ealier code-feature vs. user-feature. The user doesn't care about the feature of the code, he or she wants to talk in terms of dos: «I do that then I do this, then when X event, I do that, If Y do those». A software doesn't speak this language, a language does, the framework should be in between.

At work, I had a problem, this was minor but I think it's symptomatic of how the admin works and give a hint about what is wrong. We had two applications, one of the applicatins works standalone but the other relate to the first. A model for the latter had M2M relation to a model of the former. The ChangeList refered to those via a count column. When you clicked on it, you were suddenly teleported to the filtered ChangeList of another model in another application for further inspection. This is what is wrong, basically the user is in a context then a different context with the possibility of getting lost without hitting «back» browser button. You might have guessed that what I find wrong is that the admin is model centric instead of procesus centric. While I definitly need to read some process definitions, I think what match best the ideas behind it, is «groups», both «groups of processus» and «groups of users». That's why I think AdminNext should be groups centric.

How ? I don't know yet but the admin certainly needs to me more pluggable and customizable (aka. extensible), so what is above still holds. To make the admin or its features pluggable those must be able to live outside of the admin. When those components are pluggable would it make sens to ship something that use it to implement something else than the admin which takes some assumptions regarding application ergonomy that isn't default in most applications ?

Where do I go from here ?

I'd like to know how does django-composite tackle the pluggability of the elements of the admin. I'm pretty sure it does but I'd like to see it.
- If I'm satisfied I'll propose the concept to be included in Django
- If not template tags will do.
Then, I'll do some Django bugs smashing :)
I will also dive into admin backends and ergonomy to see what people do

How can you help? Pick a subject that you are interested in and do something with it. And like Russel Keith-Magee said «Look for the general problem, not the specific. If you can solve the general problem, the net gain is much larger.» [django-developers]

Meta framework: Part one

2013-01-01T00:00:00+01:00

What ?

This title sound obnoxious, each of the words taken alone are already the source of a lot of flame wars. Guess what this an already solved problem and it has Zope in its name!

Maybe you read my #2012pythonmeme then you know that I would like a «a full stack framework to do web, dekstop, mobile and unhosted applications». This is not just me surfing on top of the buzz several other pythonista expressed similar wish. Moreover this kind of things already exists, like GWT and pyjs (formely pyjamas), meteor and probably others. In other families of software they are most kernels, GUI toolkits and anything that deals with multi-plateform and has to deal with the absence of standard or cross-standards or just backward compatibility.

This is definitly a common pattern in software developpement. Summaries of the following Structural design patterns prooves that this very common:

Adapter or Wrapper or Translator: Convert the interface of a class into another interface clients expect. An adapter lets classes work together that could not otherwise because of incompatible interfaces. The enterprise integration pattern equivalent is the translator
Bridge: Decouple an abstraction from its implementation allowing the two to vary independent
Facade: Provide a unified interface to a set of interfaces in a subsystem. Facade defines a higher-level interface that makes the subsystem easier to use.
Flyweight: Use sharing to support large numbers of similar objects efficiently.
Module: Group several related elements, such as classes, singletons, methods, globally used, into a single conceptual entity.
Proxy: Provide a surrogate or placeholder for another object to control access to it.

Other patterns that I think are related:

Abstract factory: Provide an interface for creating families of related or dependent objects without specifying their concrete classes.
Mediator: Define an object that encapsulates how a set of objects interact. Mediator promotes loose coupling by keeping objects from referring to each other explicitly, and it lets you vary their interaction independently.
Strategy: Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

I don't only want to sound érudit, I back up my intentions with proofs. So, this a doing and hence probably doable.

Framework for frameworks

What is meta-framework or framework for frameworks.

It's not related to meta-programming, per se, but can offer some facility to ease the creation of APIs that use such programming technics pervasive in application frameworks.
It offers a canvas for creating pluggable applications
It offers a canvas for creating extensible applications
It can offer a canvas for programming

How ?

Solved problem

At least the meta-framework part or its base component is I think solved by Zope Component Architecture (cf. Defending Pyramid’s Design), even if it's not refered as «meta-framework». But I think it is, since it offers what is needed to plug and extend any pieces in a software, thus the framework would be to specify these pieces and probably providing default implementation which I think is the purpose of softwares like Pyramid.

I won't refer to ZCA any further to avoid any confusion.

Moar solutions

Callable class

Presentation

First I'm fascinated by callable class, I have no particular interest or knowledge in functionnal programming, I just find them beautiful. A callable class in Python is a class that defines a __call__ method:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

class CallMe(object):

    def __init__(self, name):
        self.name = name

    def __call__(self):
        print(u'Héllo %s' % self.name)

It used this way:

>>> call_me = CallMe('callable class')
>>> call_me()
Héllo callable class

Neat isn't it ?

Except the fact that you have to instantiate it first, it produce code similar to function call but the function is configurable as such a callable class can be seen as function factory. The above example is an example of that. But it doesn't take «function arguments», let's correct this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

class CallMe(object):

    def __init__(self, name):
        self.name = name

    def __call__(self, adj):
        print(u'Héllo %s %s' % (adj, self.name))

if __name__ == '__main__':
    call_me = CallMe('configurable function')
    call_me('awesome')

This will print Héllo awesome configurable function.

This can also be achieved with a function, but you will have to repeat for every call of call_me the name argument.

Another pattern is to build the call_me function from another function:

def call_me_factory(name):
    def call_me(adj):
         print(u'Héllo %s %s' % (adj, name)
    return call_me

It works similarly to the above CallMe class except it is less readable.

Another pattern is the ability to factorize some inner working of the callable in a method:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

class CallMe(object):

    def __init__(self, name, lang):
        self.name = name
        self.lang = lang

    def translate(self, msg):
        return msg

    def __call__(self, adj):
        hello = self.translate(u'Héllo')
        print('%s %s %s' % (Hello, adj, self.name))

This can be done for readability or because some function is recursive or needs to be called several times the same function. But this can also be achived without class just using a module and functions.

Another interesting aspect is the fact it is a class, thus it can be subclassed and its behaviour changed from outside:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

class BabelCallMe(CallMe):

    def translate(self, msg):
        return babel.translate(msg, self.lang)

Of course, you still can use a factory function...

Callable-class all the things

I'm not 100% sure this will work, but at least in my current experience of Python developpement it's rare to mutate an object from outside, most of the time you instantiate a class once or it is already instantiated and use it then bye. More over most of the time, since you call the class only one time, the entry point is a method instead of __call__ to allow for readable code. But it should be possible to name the class or class instance in a way that makes it obvious what it does when it is called like a function. If a class has several purpose, then it is possible to wrap it in another class so that each class has only one method to call.

In the case needs to have several call methods, it can the following pattern:

class MethodDispatch(object):

    def __init__(self):
        self.method_one = CallableClass(self)
        self.method_two = CallableClass(self)

So that when you want to use MethodDispatch, you do the following:

method_dispatch = MethodDispatch()
method_dispatch.method_one()

It looks like a regular method call but it isn't.

While this might not be a good pattern, since it multiply the number of objects and class, you can also use a dispatch argument when you call a callable object, using a switch-like statement:

class ArgumentDispatch(object):

    def __call__(self, algorithm):
        if algorithm == 'trolilol':
            self.trolilol()
        elif algorithm == 'spam&egg inc.':
            self.spam_and_egg()
        else:
            raise NotImplementedError()

This is kind of evil meta programming technics, that I've exposed here for the sake of completness since in my vision the end user should never have to deal with dispatching directly, the meta-framework should deal with this kind of things.

The purpose of this is to avoid complicated classes that said it doesn't mean that a class can't be complex, it can still have several methods but those will be for internal use only.

Semantic based object names

In an application there is several components, the application way to name the libraries its use is by their name, for instance SQLAlchemy, PySide and werkzeug which can be respectively named orm, gui and wsgi or data, ui and gateway at the framework level depending on the level of abstraction. So you deal with data this way:

from framework.data import Model
from framework.data import field
from framework.data import persistence

from app import config


class Stub(Model):
    title = field.String()
    body = field.String()
    creation = field.DateTime(auto_now_add=True)

hello = Stub(title='hello', 'I am going well and you ?')
db = persistence.build_from_config(**config)
db.save(hello)

persistence.build_from_config is vague at best and use a global variable as configuration which is strictly forbidden! Configuration is the purpose of the next section.

Configuration

Configuration should be auto-discoverable, customizable and allow to override the application inner working from outside.

Auto-discoverable means that given a context of execution it knows what it should do. For instance, given a callable that takes a user as argument, depending on the configuration user.notify will be implemented differently depending of the running program:

CLI based program will use print()
Daemon-like programs can use OS logging facility or send a mail
GUI based programs will rely on some widget

End

This is the end! More meta-framework programming will come. I will try not to document myself more about ZCA and framework developpement so that this keeps it primary purpose: an exercice of creativity and problem solving.

Colophon

2012-12-30T00:00:00+01:00

You can definitly have a look at the source, but since you are here you are looking for more.

This is an original theme by me for pelican, built using:

foundation css framework to ease the pain and learn the basics which are not very different from twitter bootstrap and for which there is actually a documentation with live demos.
The font used for the titles is the awesome open source League Gothic, I used it for all my slides so far.
I also use animate.css.
slabText is used to render the big titles
mansonry to render the index pages
the background is from subtle patterns
And github to host all the things.

Like Tupac says «Keep your head up!»

New Year's Python Meme 2012

2012-12-30T00:00:00+01:00

Tarek Ziadé is doing a meme each year about Python, a good occasion to share.

1. What’s the coolest Python application, framework or library you have discovered in 2012 ?

It's hard to tell, since it's both my first year as a professionnal which also happens to have been related to Python and Django to be precise and the first year of me getting involved in the Python and FOSS community by contributing code, actively trying to stay in the loop and blogging: I learned a lot. And thus discovered a lot of the Python eco-system. If the question was about the most awesome techno I discovered this year it would have been graph databases.

I will list all the things I discovered this year in Python that I liked very much and then I will try to choose one:

PyZMQ
pyjnius I didn't knew about JNI, hence didn't know one could call Java code from Python!
appengine-ndb-experiement code is really interesting
Django, even if I knew about it, I know it way better now

I will stop here because it's of course Django and it's indeed pretty cool.

Since ZeroMQ is getting quite a bit of publicity, I will add that the multiprocessing module is pretty cool and that it can be used when you only need to communicate over a network without all the patterns that ZeroMQ offers.

2. What new programming technique did you learn in 2012?

I don't know if it's a programming technique but it's a pratical sens of what I knew about Python to build APIs, like «here let's use a class property since it's shared by all class» or that we can actually avoid declarative syntax to achieve a similar purpose without meta-classes.

I don't understand it fully yet but Zope Component Architecture is an impressive software by its features, it changed the way I think about extensibility and plugability making them a bit «automatic» and a bit magical too.

3. Which open source project did you contribute to the most in 2012? What did you do?

Except Django for which I created several open source generic applications at work and my own projects, pyjnius. Actually Cython is not difficult, even if I have to consider to create Python bindings of a C library in the future I will probably use cffi which is supported by PyPy.

Talking about pyjnius I find the Java method call resolution quite interesting since it overrides the default Python method resolution which does not support what I think is called method polymorphism.

4. What was the Python blog or website you read the most in 2012?

I don't read specific Python blog and website, most of my readings comes from hacker news even if I am quite desappointed so I'm looking for something else.

Regarding Python it is cafepy that I used as a reference to understand Python type/class system, metaclasses and descriptors while coding Graphiti and Subscript.

5. What are the three top things you want to learn in 2013?

Hu! That is a though question since there are too many.

It's not about learning and more about doing: better commits, and release code that actually work in one time. It basicaly boils down to be more rigourous and most importantly being patient.
Learn actual machine learning and probably learn some Math along the way with scikit.learn.
Learn or do more frontend developpement probably in Python but also in javascript.

6. What is the top software, application or library you wish someone would write in 2013?

Maybe those already exists and maybe those are quite non-sensical or impossible but here is the full list of softwares I wish existed all in Python except otherwise noted and the last one can be in any language:

a desktop environnement probably in Kivy and espeacially a browser
a (meta) web app that will allow me to a) keep my data near b) that I can hack d) enjoy freedom.
A graph database, probably in Go, that can run Python queries cf. PrintempsDB.
an integrated (web?) developpement environnement so that I can learn, improve my skills and ask for feedback all in one place.
a full stack framework to do web, dekstop, mobile and unhosted applications
Javascript in the browser, it's looks like brython is solving that.
A psychohistoir machine so that I can learn how to bring world peace and inter galatic space missions

Like Bob Marley says «Time will tell» ;)

Want to do your own list ?

Here's how:

Copy-paste the questions and answer to them in your blog
Tweet it with the #2012pythonmeme hashtag

Nouveau Premier

2012-12-29T00:00:00+01:00

Héllo! C'est Amirouche à l'autre bout du clavier de l'autre coté de l'écran qui vous présente son nouveau blog.

Je suis lasse de blogspot et de son editeur tout-pourri où il est difficile d'intégrer des bouts de code. En fait je suis resté pour le thème, les poissons et les envoies par mails qui me semblaient indispensables... Pas de miracle et c'est bien avec des articles interessants et originaux qu'on arrive à toucher, même si ils ne commentent pas, ça fait monter le kikimeter, flatte l'égo même si par ailleurs déprime. Apparement c'est comme ça et pas autrement, je l'ai accepté. Ceci dit philosophiquement c'était un peu dure de ne pas avoir un système de commentaire sur un blog donc il y a un de nouveau sur celui-ci.

Donc voilà ma nouvelle maison, un nouveau thème, un presque nouveau nom, une nouvelle formule et surement pleins d'articles et peut être du code, j'espère que ça vous plaira.

J'ai un peu honte de n'avoir jamais fini aucune solution de publication qui me satisfasse mais tanpis cela me donnera peut-être l'occasion de contribuer à pelican, une solution de blogue qui jusqu'à maintenant me satisfait dans un contexte technique original et cocasse pour les fonctions que je veux ajouter.

Si vous ne me connaissez pas c'est pas grave, «watchez» le dépôt sur github ou abonnez-vous au flux et bientôt vous me connaitrez mieux.

Porter les billets de l'ancien blog n'est pas prioritaire, il y a quelques brouillons que je veux publier avant.

Aux portes de l'année 2013, j'ai pleins de nouveaux projets et d'opportunités qui se presentent, encore pleins de choses que j'aimerai faire, mais stop au teasing, comme dirait Oxmo Puccino «Tu déconnes avec tes tonnes de projets, ils vont encore s'entasser avec ceux de l'année prochaine»...