Planet Debian

Julian Andres Klode: Memory allocators

Thu, 31 May 2012 09:43:00 +0000

In the past days, I looked at various memory allocators in a quest to improve performance in my multi-threaded test cases of a reference counting language runtime / object allocator (a fun project).

It turns out the glibc’s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options.

The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object.

This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let’s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register).

I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc’s 1.33 or glibc’s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608).

Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that’s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable.

All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation.

Should I try replacing Python’s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?

Filed under: General

Richard Hartmann: Potpourri II

Thu, 31 May 2012 09:25:34 +0000

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon.

Kicking things along

This is coming along nicely

RAID 4

Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5.

Don't use RAID 4. Seriously.

Tom Tom vs OSM

In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV.

There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great.

These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice".

But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course.

Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason.

As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey.

Windows

100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this.

UEFI

It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc.

Wrapping up

And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

Marc 'Zugschlus' Haber: atop in unstable

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Thu, 31 May 2012 07:13:06 +0000

Eight days ago, I uploaded atop 1.26-1 to DELAYED/8, listing me as new maintainer. This means that the package has in the mean time appeared in unstable, and I hope that it’ll swiftly migrate to testing.

Matthew Garrett: Implementing UEFI Secure Boot in Fedora

Thu, 31 May 2012 05:41:15 +0000

(Brief disclaimer - while I work for Red Hat, I'm only going to be talking about Fedora here. Anything written below represents only my opinions and my work on Fedora, not Red Hat's opinions or future plans)

Fedora 17 was released this week. It's both useful and free, and serves as a welcome addition to any family gathering. Do give it a go. But it's also noteworthy for another reason - it's the last Fedora release in the pre-UEFI secure boot era. Fedora 18 will be released at around the same time as Windows 8, and as previously discussed all Windows 8 hardware will be shipping with secure boot enabled by default. While Microsoft have modified their original position and all x86 Windows machines will be required to have a firmware option to disable this or to permit users to enrol their own keys, it's not really an option to force all our users to play with hard to find firmware settings before they can run Fedora. We've been working on a plan for dealing with this. It's not ideal, but of all the approaches we've examined we feel that this one offers the best balance between letting users install Fedora while still permitting user freedom.

Getting the machine booted

Most hardware you'll be able to buy towards the end of the year will be Windows 8 certified. That means that it'll be carrying a set of secure boot keys, and if it comes with Windows 8 pre-installed then secure boot will be enabled by default. This set of keys isn't absolutely fixed and will probably vary between manufacturers, but anything with a Windows logo will carry the Microsoft key[1].

We explored the possibility of producing a Fedora key and encouraging hardware vendors to incorporate it, but turned it down for a couple of reasons. First, while we had a surprisingly positive response from the vendors, there was no realistic chance that we could get all of them to carry it. That would mean going back to the bad old days of scouring compatibility lists before buying hardware, and that's fundamentally user-hostile. Secondly, it would put Fedora in a privileged position. As one of the larger distributions, we have more opportunity to talk to hardware manufacturers than most distributions do. Systems with a Fedora key would boot Fedora fine, but would they boot Mandriva? Arch? Mint? Mepis? Adopting a distribution-specific key and encouraging hardware companies to adopt it would have been hostile to other distributions. We want to compete on merit, not because we have better links to OEMs.

An alternative was producing some sort of overall Linux key. It turns out that this is also difficult, since it would mean finding an entity who was willing to take responsibility for managing signing or key distribution. That means having the ability to keep the root key absolutely secure and perform adequate validation of people asking for signing. That's expensive. Like millions of dollars expensive. It would also take a lot of time to set up, and that's not really time we had. And, finally, nobody was jumping at the opportunity to volunteer. So no generic Linux key.

The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key.

Bootloaders

We've decided to take a multi-layer approach to our signing for a fairly simple reason. Signing through the Microsoft signing service is a manual process, and that's a pain. We don't want to have bootloader updates delayed because someone needs to find a copy of Internet Explorer and a smartcard and build packages by hand. Instead we're writing a very simple bootloader[2]. This will do nothing other than load a real bootloader (grub 2), validate that it's signed with a Fedora signing key and then execute it. Using the Fedora signing key there means that we can build grub updates in our existing build infrastructure and sign them ourselves. The first stage bootloader should change very rarely, and we don't envisage updating it more than once per release cycle. It shouldn't be much of a burden on release management.

What about grub? We've already switched Fedora 18 over to using grub 2 by default on EFI systems, but it still needs some work before it's ready for secure boot. The first thing is that we'll be disabling the module loading. Right now you can load arbitrary code into grub 2 at runtime, and that defeats the point of secure boot. So that'll be disabled. Next we'll be adding support for verifying that the kernel it's about to boot is signed with a trusted key. And finally we'll be sanitising the kernel command line to avoid certain bits of functionality that would permit an attacker to cause even a signed kernel to launch arbitrary code. These restrictions will all vanish if secure boot is disabled.

Kernel

Secure boot is built on the idea that all code that can touch the hardware directly is trusted, and any untrusted code must go through the trusted code. This can be circumvented if users can execute arbitrary code in the kernel. So, we'll be moving to requiring signed kernel modules and locking down certain aspects of kernel functionality. The most obvious example is that it won't be possible to access PCI regions directly from userspace, which means all graphics cards will need kernel drivers. Userspace modesetting will be a thing of the past. Again, disabling secure boot will disable these restrictions.

Signed modules are obviously troubling from a user perspective. We'll be signing all the drivers that we ship, but what about out of tree drivers? We don't have a good answer for that yet. As before, we don't want any kind of solution that works for us but doesn't work for other distributions. Fedora-only or Ubuntu-only drivers are the last thing anyone wants, and this really needs to be handled in a cross-distribution way.

Wait signed what

Secure boot is designed to protect against malware code running before the operating system. This isn't a hypothetical threat. Pre-boot malware exists in the wild, and some of it is nastier than you expect. So obviously bootloaders need to be signed, since otherwise you'd just replace the signed bootloader with an unsigned one that installed malware and booted your OS.

But what about the kernel? The kernel is just code. If I take a signed Linux bootloader and then use it to boot something that looks like an unsigned Linux kernel, I've instead potentially just booted a piece of malware. And if that malware can attack Windows then the signed Linux bootloader is no longer just a signed Linux bootloader, it's a signed Windows malware launcher and that's the kind of thing that results in that bootloader being added to the list of blacklisted binaries and suddenly your signed Linux bootloader isn't even a signed Linux bootloader. So kernels need to be signed.

And modules? Again, modules are just code. It's a little trickier, but if your signed kernel loads an unsigned module then that unsigned module can set up a fake UEFI environment and chain into a compromised OS bootloader. Now the attacker just has to include a signed kernel and a minimal initramfs that loads their malware module. It'd slow down boot by a couple of seconds, but other than that it'd be undetectable. X? If you can access the registers on a GPU then you can get the GPU to DMA over the kernel and execute arbitrary code. Trickier again, but still achievable - and if you've locked down every other avenue of attack, it's even attractive.

If we produce signed code that can be used to attack other operating systems then those other operating systems are justified in blacklisting us. That doesn't seem like a good outcome.

Customisation

A lot of our users want to build their own kernels. Some even want to build their own distributions. Signing our bootloader and kernel is an impediment to that. We'll be providing all the tools we use for signing our binaries, but for obvious reasons we can't hand out our keys. There's three approaches here. The first is for a user to generate their own key and enrol it in their system firmware. We'll trust anything that's signed with a key that's present in the firmware. The second is to rebuild the shim loader with their own key installed and then pay $99 and sign that with Microsoft. That means that they'll be able to give copies to anyone else and let them install it without any fiddling. The third is to just disable secure boot entirely, at which point the machine should return to granting the same set of freedoms as it currently does.

But I don't trust Microsoft

A system in custom mode should allow you to delete all existing keys and replace them with your own. After that it's just a matter of re-signing the Fedora bootloader (like I said, we'll be providing tools and documentation for that) and you'll have a computer that will boot Fedora but which will refuse to boot any Microsoft code. It may be a little more awkward for desktops because you may have to handle the Microsoft-signed UEFI drivers on your graphics and network cards, but this is also solvable. I'm looking at ways to implement a tool to allow you to automatically whitelist the installed drivers. Barring firmware backdoors, it's possible to configure secure boot such that your computer will only run software you trust. Freedom means being allowed to run the software you want to run, but it also means being able to choose the software you don't want to run.

You've sold out

We've been working on this for months. This isn't an attractive solution, but it is a workable one. We came to the conclusion that every other approach was unworkable. The cause of free software isn't furthered by making it difficult or impossible for unskilled users to run Linux, and while this approach does have its downsides it does also avoid us ending up where we were in the 90s. Users will retain the freedom to run modified software and we wouldn't have accepted any solution that made that impossible.

But is this a compromise? Of course. There's already inequalities between Fedora and users - trademarks prevent the distribution of the Fedora artwork with modified distributions, and much of the Fedora infrastructure is licensed such that some people have more power than others. This adds to that inequality. It's not the ideal outcome for anyone, and I'm genuinely sorry that we weren't able to come up with a solution that was better. This isn't as bad as I feared it would be, but nor is it as good as I hoped it would be.

What about ARM

Microsoft's certification requirements for ARM machines forbid vendors from offering the ability to disable secure boot or enrol user keys. While we could support secure boot in the same way as we plan to on x86, it would prevent users from running modified software unless they paid money for a signing key. We don't find that acceptable and so have no plans to support it.

Thankfully this shouldn't be anywhere near as much of a problem as it would be in the x86 world. Microsoft have far less influence over the ARM market, and the only machines affected by this will be the ones explicitly designed to support Windows. If you want to run Linux on ARM then there'll be no shortage of hardware available to you.

Is this all set in stone?

No. We've spent some time thinking about all of this and are happy that we can implement it in the Fedora 18 timescale, but there's always the possibility that we've missed something or that a new idea will come up. If we can increase user freedom without making awful compromises somewhere else then we'll do it.

[1] In fact, chances are that everything will carry the Microsoft key. Secure boot requires that UEFI drivers also be signed. The signing format only permits a single signature per binary. For compatibility, approximately all add-on hardware shipped will be signed with Microsoft's key, and that means that all system vendors have to recognise Microsoft's key in order to permit that hardware to run on their systems.
[2] Current source is here. It relies on a port of the UEFI crypto library and OpenSSL that I have building with some handholding, and which I'll upload as soon as possible.

comments

Paul Tagliamonte: my sbuilder setup (sb)

Thu, 31 May 2012 00:00:27 +0000

I’ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly.

It’s gone from one small shell script to a set of 8, and growing. Quickly.

Until I spin these off into their own repo, they’ll be in my dot-bin

Before getting too far, please relize this is only how I’m using sbuild. There are many other (more) sane configurations that you can (and should) use.

This was not made to be useful for anyone other then me :)

I’ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot.

Firstly, forgive me for my sins. This is all a huge hack, but it’s use isn’t the worst thing in the world.

As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable.

Now — the big question — how do you use this. Rather, this is more of a how do I use sbuilder :)

Firstly, I’ve created a file for each schroot set that I’m using. Here’s my /etc/schroot/chroot.d/sid.

[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config

It’s very similar to the one that sbuilder-createchroot creates.

If you’re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf.

Next, create a dist.d directory in /var/sbuild — such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists.

The arguments for all the sb* commands is always in the order of variant, dist.

variant defaults to “generic”, which I’ve set as an alias to the generic chroot, and the dist defaults to “sid”. At some point, I’ll change the default to be the last UNRELEASED entry in the changelog.

So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable.

After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay.

The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it’s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory — this is to prevent mistakes, not malicious scripts).

By the way, for those interested, there’s a small issue with what I’m describing it’s very subtle. Basically, there’s a few assumptions made by how schroot operates, and handles “sessions”. Since it’ll keep stuff mounted if the session is still active, it makes purging all this a real … pain.

The magic behind the scripts is in the sb-endsessions routine — which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn’t have a situation where this kills off another build.

The code (which might have issues) looks like:

SESSIONS=`schroot --all-sessions -l | grep "^session:$DIST-"`
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done

By backing of schroot’s mounts, we can do our mounting / removing without worrying about klobbering schroot.

Now, there’s also the case of wanting to interactively log into a chroot. For that, there’s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user.

Eventually, I’ll be adding some scripts to upgrade all the build chroots for a dist at once.

Jaldhar Vyas: Dovecot 2.1.7 Uploaded to Unstable

Wed, 30 May 2012 22:27:53 +0000

This is the latest upstream version and will, barring any unforseen circumstances, be the version in wheezy, the upcoming stable release of Debian GNU/Linux. In order to ensure it is the high quality our users have come to expect from Debian stable, I need your help.

First and foremost, please test the hell out of it. Upgrades during a stable release are in general not allowed unless the problem is security related so we need to shake all the bugs out now. I am particularly interested in knowing if there are any issues with upgrades from squeeze.

I waved my hands ineffectually at the bug list but it needs more attention. Could some kind person go through them and let me know which ones are no longer valid, need further attention etc.?

I attempted to add hardening flags but lintian still complains a lot. I think the warnings are probably false positives but I'm not a 100% sure so I would appreciate some advice from those who know more about such things.

Those are the issues that come to mind at the moment but feel free to let me know if there's something I've missed.

Lior Kaplan: Debian packaging for beginners @ FOSDEM – the video

Wed, 30 May 2012 20:20:40 +0000

I recently got a question by email about the lecture I presented at FOSDEM, and by that mail I discovered that the video of the lecture was uploaded to YouTube (FOSDEM 2012 – Debian packaging for beginners). While I thank the person who did the upload, he/she did not keep the right license for the video which is CC-BY-SA 3.0 (as noted at FOSDEM archive). You can also grab the webm file directly.

Well, the next task will be to listen to myself in order to improve for FOSDEM 2013… See you there.

Filed under: Debian GNU/Linux, FOSDEM

Thomas Goirand: New release of MLMMJ (version 1.2.18.0) uploaded in Debian

Wed, 30 May 2012 19:59:22 +0000

MLMMJ stands for Mailing List Manager Made Joyful. To me, it’s the best mailing list manager available. Not only its written in C (no slow interpreted language here…), but also it is easy and very convenient to setup. No hack is needed to make it run on a multi-domain environment, and no need to run a special subdomain for your lists (yes, I’m thinking about you, silly Mailman…). If you didn’t know, MLMMJ is used to handle relatively high traffic lists: all mailing lists for Gentoo are using MLMMJ for example.

MLMMJ 1.2.18.0 has just been release upstream, few days ago (on the 29 of may). It’s with a great pleasure that I packaged (and uploaded to SID) the latest version, and wrote the changelog, where I wrote that 6 Debian specific patches (out of 8) could be removed! One of the funny changes are the renaming of the “mlmmj-recieve” into “mlmmj-receive” as it always should have been (with a symlink to keep backward compatibility), and the removal of the .sh extension to the mlmmj-make-ml (which was patched in Debian to be kept policy compliant: that’s one of the removed patches…).

Even if I did a few quick functional tests before uploading, I’d be happy to get some feedback before Wheezy is out, so please test and eventually bug-report!

Sorina Sandu: Initial commit

noreply@blogger.com (Sorina) — Wed, 30 May 2012 14:57:36 +0000

Hello!

I'm Sorina and this summer I will be working on Debian Installer network setup during Google Summer of Code. I am a second year bachelor's student at Politehnica University of Bucharest, I am studying Computer Science. I very much enjoy what I am doing even if sometimes I feel that the amount of stuff I want/have to learn/do/work on is infinite and I don't really know where to start.

I am at the beginning of the project, I believe now I am accustomed to the way things work and I am waiting forward to finishing the exam period so that I could really start working.

The hardest part for me in what I've done so far is testing what I am doing. It took me a while to figure out at least a part of how the installer functions and to solve the problems that occurred while building. Luckily, there is documentation and internet and even more important, I had help and support from my mentors.

So far I have only written two patches for netcfg, for bugs #610752 and #636211. For the first one I had a version for a while now, but I was unable to test it (unfortunately, it's likely that you will hear this a lot in the future). The idea was to make the link detection timeout preseedable and for this I have added two new templates, asking the user to provide a value for that matter.

The second one involved a missing "Go Back" button. This seemed to be the result of not letting debconf know that the program is capable of handling the user pressing a back button. That could be solved using deb_capb.

As I said before, testing was my main problem. Eventually, I built the image, wrote it to an USB and rebooted my laptop. This served its purpose, since I had access to anything I needed such as the wireless interface and physical access to the network cable. Running it through a virtual machine did not help much in this case, because I was unable to provide those.

Another thing I considered to be hard was blog posting, but now that I have just finished my first post it seems that I overreacted.

That's all for now, have a nice day!

Michal Čihař: Four counferences in Prague this October

michal@cihar.com (Michal Čihař) — Wed, 30 May 2012 11:00:00 +0000

I'm participating on organizing this year's openSUSE conference and as we've finally finalized place and time and CFP is open, it's time to share some information.

This will be special - there is not a single conference, but four of them are sharing same space and the motto - Bootstrapping awesome!!!. The 4th openSUSE conference will be held together with LinuxDays, Czech conference following tradition of canceled LinuxExpo, what should be the biggest free software event in Czech republic. In addition there will be also 12th SUSE Labs conference (so you can meet quite a lot of kernel hackers and other strange guys) and first Gentoo mini summit (the website is empty so far).

All that will happen on weekend from 20th October to 21st October, the SUSE conferences will then continue for 22nd and 23rd October.

I believe it will be great mixture of conferences and I hope to meet lot of people there.

Filed under: English Gentoo Suse | 0 comments | Flattr this!

Lucas Nussbaum: Debian archive rebuilds on Amazon Web Services

Wed, 30 May 2012 06:52:30 +0000

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, …). It is also a good way to stress-test all packages that are involved in building other packages.

Since 2007, I had been running Debian archive rebuilds on the Grid’5000 testbed, a research infrastructure for performing experiments on distributed systems – HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process.

Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With Sébastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS.

On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds.

Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

Raphael Geissert: ldebdiff: what local change did I make to a package?

noreply@blogger.com (Raphael Geissert) — Wed, 30 May 2012 04:03:55 +0000

When fixing or modifying a package I at times change the installed files directly. However, once the changes are okay and it is time to prepare a diff, having to keep track and manually diff'ing the installed files is time consuming.

That's when ldebdiff does its thing: it runs diff against the original files of a given package and the ones installed on a system.
For example:

$ ldebdiff acpi-fakekey
--- unpacked/etc/init.d/acpi-fakekey    2012-04-05 05:14:21.000000000 -0500
+++ /etc/init.d/acpi-fakekey    2012-04-12 12:10:55.000000000 -0500
@@ -4,8 +4,8 @@
 
 ### BEGIN INIT INFO
 # Provides:          acpi-fakekey
-# Required-Start:    $local_fs $remote_fs
-# Required-Stop:     $local_fs $remote_fs
+# Required-Start:    $remote_fs
+# Required-Stop:     $remote_fs
 # Default-Start:     2 3 4 5
 # Default-Stop:      
 # Short-Description: Start acpi_fakekey daemon

($remote_fs implies $local_fs)

Bartosz Feński: dibbler 0.8.2

Wed, 30 May 2012 03:40:17 +0000

To whom it may concern.

I’ve just uploaded new upstream version of dibbler (portable DHCPv6 client/server) to unstable.
The resolvconf patch needed some changes to apply on that version so whoever actually uses dibbler, please test it ASAP so we could address any issues.

This version amongst other things fixes nasty Debian bug #659476.

Christian Perrier: 2012 update 24 for Debian Installer localization

Wed, 30 May 2012 02:46:00 +0000

The "Save the baltic languages" week..:-)
Latvian translations completed for level 1. Latvian won't be deactivated!
Also progress in levels 2 and 3 for Latvian
Lithuanian progress for level 1. Language won't be deactivated!
Estonian was already safe..:-)

Status for D-I level 1 (core D-I files):

51 languages 100%: ar ast be bg bn bs ca cs da de el eo es et eu fa fi fr gu he hi id it ja kk km kn ko lo lv mr nb nl pa pl pt pt_BR ru si sk sl sr sv te th tr ug uk vi zh_CN zh_TW
1 languages 99%: ta
2 languages 97%: ga hu
5 languages 95%: dz is lt mk ro
1 language 92%: ml
others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

31 languages 100%: ast be bg ca cs da de eo es fa fr gu he id is it ja kk km mr nl pl pt ru si sk sl sv tr uk zh_CN
8 languages 99%: eu fi hr lv nb th vi zh_TW
3 languages 98%: dz pt_BR ro
1 language 97%: ta
1 language 96%: ug
3 languages 94%: ar el gl
1 language 92%: sr
5 languages 92%: bn bs hu ko ne
others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

38 languages 100%: ast be bg bs ca cs da de el eo es fa fi fr ga gl he hr id is it ja kk km nb nl pl pt ru sk sl sr sv th tr vi zh_CN zh_TW
3 languages 98%: hu ug uk
1 language 95%: lv
others are 90% or below

Full 100% completeness (hall of fame) for 26 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Danish, German, Esperanto, Spanish, Persian, French, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Turkish, Simplified Chinese

Christian Perrier: 12 languages to be deactivated in Debian Installer

Wed, 30 May 2012 02:43:00 +0000

Latvian is now completed.
Lithuanian is now completed.
I got an offer for Welsh

At this very moment, it means that I would deactivate 14 languages:

Amharic (5f17u in sublevel 2)
Welsh (39f82u, 32f47u)
Galician (7f10u, 6f9u)
Croatian (16f19u, 6f9u)
Georgian (17f47u, 10f21u)
Kurdish (12f25u, 6f22u)
Malayalam (5u in sublevel 2 only!)
Nepali (16f54u, 10f21u)
Norwegian Nynorsk (13f34u, 13f23u)
Albanian (19f50u, 13f23u)
Tamil (2f1u, 1f)
Tagalog (18f54u, 24f39u)

Paul Tagliamonte: Goofing off with sbuild "variants"

Wed, 30 May 2012 02:41:00 +0000

I’ve recently found myself in need of more then one sbuild chroot, each one set up slightly differently then the next. For a while, providing different behavior with unstable and sid was all well and good, but I figure, like any good computer scientist, I should solve the general problem.

As a result, I’ve written an sbuild wrapper to help maintain sanity. It uses a series of symbolic links to hide everything out of the way, and some bash hackery.

Basically, I’ve set up a /var/sbuild directory, full of a few different files. For each distribution, there is one folder, one symbolic link and up to two other files. There’s the dist.d file (such as /var/sbuild/sid.d) that contains all the “variants”, dist symbolic link (such as /var/sbuild/sid) which points to the currently active chroot, and up to two lock files.

The script will default to sid-default (that is to say, exactly, /var/sbuild/sid.d/sid-default - which is a symbolc link to the “default” chroot), but may take two arguments, in the order of variant, then dist.

Currently, I have two chroots, one for gcc, and one for clang (set up like Sylvestre used for cland.d.n) — but I may add some more with different setups.

Currently the behavior is to have the second wait kindly if you kick off two sbuilds at once, and fail out a third.

There are a bunch of bugs, and I’m not sure that this will work very well, but it’s working.

The next step is to combine this with unionfs overlays to help protect accidentally trashing the chroots.

You can find the script on my GitHub, pasted below.

#!/bin/bash

DIST="sid"
VARIANT="default"
SBUILD=/var/sbuild

if [ "x$1" != "x" ]; then
    VARIANT=$1
fi
if [ "x$2" != "x" ]; then
    DIST=$2
fi

if [ -e $SBUILD/$DIST.lock ]; then
    PID=`cat $SBUILD/$DIST.lock`
    echo "W: We *must* wait for the finish of PID $PID"
    echo "W: Blocking execution until this file is removed."
    if [ -e $SBUILD/$DIST.wait ]; then
        echo "E: Someone else is waiting. Bombing out."
        exit 2
    fi
    echo $$ > $SBUILD/$DIST.wait
    while [ -e $SBUILD/$DIST.lock ]; do
        sleep 1
    done
    echo "I: Dropping out of lock. Purging wait."
    rm $SBUILD/$DIST.wait
fi

echo "I: Aquiring lock"
echo $$ > $SBUILD/$DIST.lock

CURRENT=$(basename `ls -xl $SBUILD/$DIST | awk '{print $11}'`)

# We set up the name like:
#   dist-variant
# such as "sid-clang" is cool.
#echo "I: We're currently $CURRENT."

if [ "x$CURRENT" == "x$DIST-$VARIANT" ]; then
    echo "I: This matches the desired setup. No action."
else
    if [ -e $SBUILD/$DIST.d/$DIST-$VARIANT ]; then
        echo "I: Re-setting $DIST symlink."
        rm $SBUILD/$DIST
        ln -s $SBUILD/$DIST.d/$DIST-$VARIANT $SBUILD/$DIST
    else
        echo "E: No such dist! Looking in $SBUILD/$DIST.d/$DIST-$VARIANT."
        exit 1
    fi
fi

echo "I: "
echo "I: Dist:    $DIST"
echo "I: Variant: $VARIANT"
echo "I: "
# echo "I: Before we go farther, we're updating."
# sbuild-update -d $DIST --upgrade

sbuild -d $DIST

echo "I: Releasing lock"
rm $SBUILD/$DIST.lock

Paul Tagliamonte: My first foray into non-compliant Debian packages

Wed, 30 May 2012 00:40:00 +0000

I’ve decided to start putting together packages to rebuild the bare bones install I use for most of my systems, since rebuilding everything by hand is starting to get annoying. I call it hairy candy.

Because my setup relies on some chem interesting chem hacks, none of this is fit — in any way — for the archive.

If the vim team was to catch wind of what I was doing, they’d be none too happy :)

While these packages work great for me, I can’t stress enough how badly they can break something.

Using Zack’s guide, I was able to get an archive up and running, no problem. Taking his advice, I even got signing working. Simple.

mini-dinstall rocks — a nice alternative to a PPA, for sure. At least, for one arch :)

Now, the one spin I took on Zack’s work was to add a “landing” page — a listing of what’s in the archive.

I did this using a bit of Python & Jinja2. The script follows.

#!/usr/bin/env python
# Copyright (c) Paul Tagliamonte, under the terms of the Expat license.

from jinja2 import Template
import sys

def parse_packages(fd):
    ret = []
    cur = {}
    key = None
    for line in fd.readlines():
    if line.strip() == "":
        ret.append(cur)
        cur = {}
        continue

    if line[0] == " ":
        cur[key] += "\n" + line.strip()
    else:
        key, val = line.split(":", 1)
        cur[key] = val.strip()
    return ret

pkgs = parse_packages(open(sys.argv[1], 'r'))
context = {
    "pkgs": pkgs
}

t = Template(open(sys.argv[2], 'r').read())
print t.render(**context)

Invocation looks something like:

render.py $ARCHIVE/debian/wicked/Packages $DATA/index.html

I kick this script off in the singing script, since mini-dinstall doesn’t have a exit hook.

Works great, thanks everyone!

Daniel Kahn Gillmor: KVM, Windows XP, and Stop Error Code 0x0000007B

Tue, 29 May 2012 22:57:34 +0000

i dislike having to run Windows as much as the next free software developer, but like many sysadmins, i am occasionally asked to maintain some legacy systems.

A nice way to keep these systems available (while not having to physically maintain them) is to put them in a virtual sandbox using a tool like kvm. While kvm makes it relatively straightforward to install WinXP from a CD (as long as you have the proper licensing key), it is more challenging to transition a pre-existing hardware windows XP installation into a virtual instance, due to Windows only wanting to boot to ide chipsets that it remembers being installed to.

In particular, booting a disk image pulled from a soon-to-be-discarded physical disk can produce a Blue Screen of Death (BSOD) with the message:

Stop error code 0x0000007B

(INACCESSABLE_BOOT_DEVICE)

This seems like it's roughly the equivalent (in a standard debian GNU/Linux environment) of specifying MODULES=dep in /etc/initramfs-tools/initramfs.conf, and then trying to swap out all the hardware.

At first blush, Microsoft's knowledge base suggests doing an in-place upgrade or full repartition and reinstall, which are both fairly drastic measures -- you might as well just start from scratch, which is exactly what you don't want to have to do for a nursed-along legacy system which no one who originally set it up is even with the organization any more.

Fortunately, a bit more digging in the Knowledge Base turned up an unsupported set of steps that appears to be the equivalent of setting MODULES=most (at least for the IDE chipsets). Running this on the old hardware before imaging the disk worked for me, though i did need to re-validate Windows XP after the reboot by typing in the long magic code again. i guess they're keying it to the hardware, which clearly changed in this instance.

Such silliness to spend time working around, really, when i'd rather be spending my time working on free software. :/

Jon Dowland: Debian Day

Tue, 29 May 2012 22:18:54 +0000

After a relatively long period of flux, I've managed to settle into a routine where I've got all the time I can dedicate to open source stuff bunched together: every Tuesday night. I've started calling it "Debian Day", although Debian won't get all that time exclusively.

One evening a week doesn't sound like much, but that's unfortunately a realistic amount given my other commitments. Still, having a firm idea of how much resource I have makes planning what I can and can't do much easier. It also means I need to be quite selective about what I spend my time on.

This also impacts the type of interactions I can use. I can respond to email, bugs, etc., but I cannot effectively interact with folks on IRC, for example: the period between windows is too large for it to work effectively. Therefore, I can't work reliably with any team or area of Debian that relies on IRC to get things done.

Tonight, then. I have recently been ripping some CDs, a task for which I use grip, despite it long since having been dropped by Debian. It still works very well, whereas Sound Juicer, which still pops up whenever I insert an audio CD, does not.

I've been interested in helping out with Debian GNOME packaging for a while, and it seems the issue has been fixed upstream, but after asking around in #debian-gnome, the root cause is that the newer musicbrainz API packages have not been uploaded to Debian. (the existing package uses the old API).

Two separate people appear to have been working on a new Music Brainz package, but neither filed an ITP bug. I've spent the largest chunk of my Debian time just trying to figure out what has been going on, sadly, but at least there's the likelyhood this will be fixed for wheezy.

There's also the tail end of a musicbrainz-2.1 transition finishing up.

Another thing I considered this evening was the future of debgtd in Debian stable. On reflection, and after sitting on the fence for several months, I've decided I don't have the time to fix it up enough to have it ready for wheezy, so with regret I've filed a removal bug. With any luck, at some point in the wheezy cycle I'll manage to put more time towards this, because I think it addresses a genuine need and could be useful to people.

Finally, whenever I sit down and look at Debian stuff to finish up, I have a strong temptation to fix some issues belonging to the games team. In particular, Gustavo Panizzo has put a lot of work into vavoom packages, but sadly his requests for sponsorship are falling on deaf ears. There's also Johey Shmit's work on packaging zdoom which needs some attention to finish removing some embedded libraries. I have to keep reminding myself I left the team for a reason, to focus on something else, and finding that something else should be the priority.

So all in all not the most uplifting evening: I'm starting to think I simply couldn't work effectively with the Debian GNOME team without being able to dedicate more time to use IRC effectively; I spent more time that could be necessary figuring out a bunch of other folks have potentially wasted each other's time by working on packages and not filing ITPs; and I've removed one of my own packages from the distribution altogether.

It's not all doom and gloom, though: last week I packaged Simon Howard's lhasa, a library and tool for decompressing LHA files, as used by various old DOS games. (See? I just can't escape that stuff…) I also managed to 'resolve' two bup bugs, one RC. I also finished transitioning from my old PGP key to my new one.

Bartosz Feński: burp 1.3.6

Tue, 29 May 2012 20:13:43 +0000

Enjoy burp 1.3.6 in unstable!

Steve Kemp: A heady mixture of photography and programming

Tue, 29 May 2012 15:06:59 +0000

The past few weeks have consisted of a heady mixture of taking interesting pictures of cute people, and writing code.

I spent a while getting to grips with seccomp filters, using the facilities present in recent GNU/Linux Kernels to filter system calls binaries are allowed to make.

My initial test was to patch GNU Less to only allow it to open, read, and close files. The side-effect of this was that the built in shell-escape was closed, thus allowing me to test it.

After that I toyed around with interfacing with spidermonkey|seamonkey. In this regard I was less succesful, but I did manage to write code in C that would invoke javascript functions loaded dynamically. Similarly I could call from my (loaded) javascript code into functions defined in C.

I don't have a use for a Javascript to/from C bridge, but I'm sure that time will come.

Photography has been a constant distraction, I took some fun shots of the Edinburgh Marathon, and then distracted myself with a volunteer to take an abstract fishnet photograph. We went on to do some more fun shots begin careful to stay on the safe side of the NSFW limit. I think this is borderline NSFW, but we were both clear exactly what we wanted and we got it perfectly there.

Next week will be quieter, but providing we don't have another mini-heatwave in Edinburgh I'll be cheerful regardless.

ObQuote: One, two, Freddy's coming for you. - A Nightmare on Elm Street (original)

Niels Thykier: Lintian 2.5.7 and 2.5.8

Tue, 29 May 2012 12:52:49 +0000

The new version of Lintian (2.5.8) can pretty much be summed up as:

Its like 2.5.7, only with less false positives and no FTBFS.

Especially people annoyed by the hardening flags will hopefully find that 2.5.8 greatly reduces the number of false positives. I believe this is best demonstrated with an example:

$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb | wc -l 
2.5.7
94
[...]
$ lintian --print-version && lintian -q -C binaries amarok_2.5.0-1_i386.deb | wc -l
2.5.8
4

However, nothing comes for free. We dropped hardening-no-stackprotector from the default profile (and demoted it to an “I” tag). For hardening-no-fortify-functions we made a “false positive -> false negative” trade-off by ignoring binaries if their only unprotected function is memcpy. For more information, please refer to #673112.

hardening-no-stackprotector is still available and can be used via the debian/extra-hardening profile (or via the –tags argument). For the 2.5.7 behaviour of hardening-no-fortify-functions, you have to use hardening-check directly.

But 2.5.7 had other changes besides the myriad of false-positives:

Around 19 redundant tags were removed.

A consequence of this is that we no longer warn if you use things like “dpkg –assert-working-epoch” or your postinst creates a “usr/doc transition symlink”.

The last of the Lintian’s (perl) modules have been put under the Lintian name space.

With all the Lintian modules under the Lintian namespace, we can install them in the standard perl @INC path. Admittedly, I am not certain we are ready to commit to the current API in these modules, which is one of the reasons why they are still installed in /usr/share/lintian. But in a couple of releases things may look differently.

Lazy loading of data files

Lintain 2.5.7 ships over 50 data files of various kinds (usually white- or blacklists of some kind). Before 2.5.7, all data files with a few exceptions would be loaded eagerly (i.e. as soon as the check was run for the first time). A few data files had been special cased with “manual laziness”.

In 2.5.7, Lintian::Data was updated to lazily load the data. So depending on the packages being checked, Lintian may now load fewer data files.

Create proper data files for tables embedded within checks.

While this is not something exclusive to 2.5.7, we have separated quite a few checks from their data tables by now. My personal favorites are the table of known interpreters and (from 2.5.8) the table of known versioned interpreters.

Support for Vendor specific data files.

I am certainly biased here. But this is probably the most awesome feature in Lintian 2.5.7. It is now possible for vendors to extend or simply “shadow” the core Lintian data files. Allow me to demonstrate how this can be used:

$ lintian --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc 
E: a2ps source: build-depends-on-obsolete-package build-depends: dpatch
$ cat ~/.lintian/profiles/local/main.profile 
Profile: local/main
Extends: debian/main
$ cat  ~/.lintian/vendors/local/main/data/fields/obsolete-packages 
@include-parent
@delete dpatch
bison
$ lintian --profile local/main --tags build-depends-on-obsolete-package a2ps_4.14-1.1.dsc
E: a2ps source: build-depends-on-obsolete-package build-depends: bison
$

It is a toy example, but I believe it is a good demonstration of the feature.

The full documentation of Vendor specific data files can be found in the Lintian User manual (in lintian/2.5.7 or newer). It will also be on lintian.debian.org when we find time to update lintian there.

Michal Čihař: Android support in Gammu

michal@cihar.com (Michal Čihař) — Tue, 29 May 2012 10:00:00 +0000

Android support in Gammu is one of frequently requested features. I don't have neither time to motivation to do that, so unless somebody else will step up, it won't be done.

Recently, I got offer from MyPhoneExplorer author to share it's applet for communicating with Android phones. It basically provides protocol similar to Sony-Ericsson phones Gammu already supports, just over TCP/IP. So all what would have to be done is glue layer for using TCP/IP and possibly some minor adjustments to Sony-Ericsson code. Both of these should be quite easy tasks and I'm willing to guide you through this process.

In case anybody would be interested in this challenge, just let me know :-).

Filed under: English Gammu | 0 comments | Flattr this!

Martin-Éric Racine: OpenWRT: accepting Router Adverts from the ISP in White Russian?

noreply@blogger.com (Martin-Éric) — Tue, 29 May 2012 07:05:56 +0000

As we are getting closer to the World IPv6 Day, I thought that I'd see if I can upgrade my WRT54GL's OpenWRT (White Russian) installation to accept Router Adverts from the ISP to acquire an IPv6 address (rather than via a 4-to-6 tunnel broker, which seems to be the most documented case), and to propagate its IPv6 address space to my local network via my WRT45GL's radvd. Unfortunately, most of the instructions out there concern newer OpenWRT releases to which I cannot upgrade to because their software base takes up significantly more space than White Russian does. Still, I figure that someone in the community must have already succeeded at configuring their WRT54GL for this and might be able to help. Anyone? :)

C.J. Adams-Collier: Ohai! It looks like I’m back online!

Tue, 29 May 2012 02:30:29 +0000

I’m working on my disk array today. The plan is to take the data off of a 6-disk RAID-6 and put it onto three 4-disk RAID-4s.

Jonathan McDowell: Go go gadget multiarch

Mon, 28 May 2012 19:58:58 +0000

Multiarch has been coming RSN for an extremely lengthy meaning of the word "soon". I remember watching Tollef give a presentation about it at DebConf4 and I'm pretty sure it's been talked about at every DebConf since then as well. Deemed the "correct" answer to the issue of running i386 binaries on x86_64 machines, or old ARM ABI programs on more modern hardware, it's always seemed to be at least another Debian release away.

Not so anymore. Through foolishness I ended up buying a Brother HL3040CN when I first moved to the US. It was a cheap networked laser printer and it touted Linux support. Quality wise it's been fine. I don't use it a lot, but unlike an inkjet I don't have to worry about not using it for a month and then needing to print something in a hurry and having to clean print heads etc. Where it falls down is that I failed to check that "Linux support" involved source. No. Instead it involves an i386 binary (at least packaged as a .deb, but in a horrible fashion). Up until now I've mostly been printing from my laptop, so all the drivers are installed there. I've got some guests this week and they needed to print their boarding passes, so I decided it was time to make the house server act as a print server too. It's an AMD64 box and before now I haven't had any need to run i386 code on it, so when I installed the driver deb it failed to work. Normally I'd just install ia32-libs, but this time I decided to try multiarch. So I did:

# dpkg --add-architecture i386
# apt-get update
# apt-get install libc6:i386

and magically I was now able to run the printer driver binary. I know there's a lot more work still to be done (I need to check if I can ditch ia32-libs on my laptop which runs a few more i386 only apps), but this is pretty cool - thanks to all those involved in making it happen!

Update: I tried to install all the multiarch bits required for Skype on my laptop but hit an issue with libqtgui4:i386 which ends up pulling in liblcms1:i386 which isn't yet multiarch enabled. There was already a bug, #637732 filed by vorlon, and mhy did the appropriate NMU a week ago, so it should hopefully hit testing in the next week. Thanks guys.

Vincent Bernat: Integration of NetSNMP into an event loop

Mon, 28 May 2012 18:54:50 +0000

NetSNMP comes with its own event loop based on the select() system call. While you can build your program around it, you may prefer to use an existing event loop, either a custom one or something like libevent, libev or Twisted.

Own custom event loop

Let’s start with the easiest case: you have written your own event loop. This means you make a call to select(), poll(), epoll(), kqueue() or something similar. All those functions take a set of file descriptors and wait for any of them to become available in a specified time frame.

Here is a typical use of select() (adapted from Quagga)

readfd = m->readfd;
writefd = m->writefd;
exceptfd = m->exceptfd;
timer_wait = thread_timer_wait (&m->timer);
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)
  {
    if (errno == EINTR)
      continue; /* signal received - process it */
    zlog_warn ("select() error: %s",
               safe_strerror (errno));
    return NULL;
  }
if (num == 0)
  {
    /* Timeout handling */
    thread_timer_process (&m->timer);
  }
if (num > 0)
  {
    thread_process_fd (&readfd);
    thread_process_fd (&writefd);
  }

thread_process_fd (fds) function iterates on each file descriptor fd and executes the appropriate action if FD_ISSET (fds, fd) is true.

Integrating NetSNMP into such an event loop is easy: NetSNMP provides the snmp_select_info() function which alters a set of file descriptors to insert its own. Here is the new code:

#if defined HAVE_SNMP
struct timeval snmp_timer_wait;
int snmpblock = 0;
int fdsetsize;
#endif

/* ... */

#if defined HAVE_SNMP
fdsetsize = FD_SETSIZE;
snmpblock = 1;
if (timer_wait)
  {
    snmpblock = 0;
    memcpy(&snmp_timer_wait, timer_wait,
           sizeof(struct timeval));
  }
snmp_select_info(&fdsetsize, &readfd,
                 &snmp_timer_wait, &snmpblock);
if (snmpblock == 0)
  timer_wait = &snmp_timer_wait;
#endif

num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0) { /* ... */ }

#if defined HAVE_SNMP
if (num > 0)
  snmp_read(&readfd);
else if (num == 0)
  {
    snmp_timeout();
    run_alarms();
  }
netsnmp_check_outstanding_agent_requests();
#endif

/* ... */

snmp_select_info() may modify the provided set of file descriptors (and therefore, its size). It may also modify the provided timer in case it needs to schedule an action before the original timeout.

snmpblock is a tricky variable. While select() can be called with a timeout set to NULL, this is not the case of snmp_select_info(): you have to pass a valid pointer. snmpblock is set to 0 if the provided timeout must be considered or to 1 otherwise. snmp_select_info() will set snmpblock to 0 if and only if it alters the timeout.

Here are a two examples of integration. Both are for a subagent but the code for a manager is the same:

Third-party event loop

With a third party event loop, you don’t have access to the select() system call anymore. Therefore, you cannot alter it with snmp_select_info(). Instead, at each iteration of the loop, a list of SNMP related file descriptors is kept up-to-date with the help of snmp_select_info(): new ones are added and old ones are removed.

libevent

Let’s assume that snmp_fds is the list of current SNMP related events¹. A function levent_snmp_update() calls snmp_select_info() to update this list. Here is a partial implementation (error handling removed and some declarations omitted, look at the complete version from lldpd):

static void
levent_snmp_update()
{
    int maxfd = 0;
    int block = 1;

    FD_ZERO(&fdset);
    snmp_select_info(&maxfd, &fdset, &timeout, &block);

    /* We need to untrack any event whose FD is not in `fdset`
       anymore */
    for (snmpfd = TAILQ_FIRST(snmp_fds);
         snmpfd;
         snmpfd = snmpfd_next) {
        snmpfd_next = TAILQ_NEXT(snmpfd, next);
        if (event_get_fd(snmpfd->ev) >= maxfd ||
            (!FD_ISSET(event_get_fd(snmpfd->ev), &fdset))) {
            event_free(snmpfd->ev);
            TAILQ_REMOVE(snmp_fds, snmpfd, next);
            free(snmpfd);
        } else
            FD_CLR(event_get_fd(snmpfd->ev), &fdset);
    }

    /* Invariant: FD in `fdset` are not in list of FD */
    for (int fd = 0; fd < maxfd; fd++)
        if (FD_ISSET(fd, &fdset)) {
            snmpfd->ev = event_new(base, fd,
                EV_READ | EV_PERSIST,
                levent_snmp_read,
                NULL);
            event_add(snmpfd->ev, NULL);
            TAILQ_INSERT_TAIL(snmp_fds, snmpfd, next);
        }

    /* If needed, handle timeout */
    evtimer_add(snmp_timeout, block?NULL:&timeout);
}

Then, replace the main loop (usually, a call to event_base_dispatch()) with this:

do {
    if (event_base_got_break(base) ||
        event_base_got_exit(base))
        break;
    netsnmp_check_outstanding_agent_requests();
    levent_snmp_update();
} while (event_base_loop(base, EVLOOP_ONCE) == 0);

Here is how are defined the two callbacks:

static void
levent_snmp_read(evutil_socket_t fd, short what, void *arg)
{
    FD_ZERO(&fdset);
    FD_SET(fd, &fdset);
    snmp_read(&fdset);
    levent_snmp_update();
}

static void
levent_snmp_timeout(evutil_socket_t fd, short what, void *arg)
{
    snmp_timeout();
    run_alarms();
    levent_snmp_update();
}

Twisted reactor

As a second example, here is how to integrate NetSNMP into Twisted reactor. Twisted is an event-driven network programming framework written in Python. It does not come with support for SNMP². Because of the language mismatch, the integration of NetSNMP in Twisted needs to be done using a C extension or the ctypes module.

Twisted event loop is called a reactor. Like for libevent, events need to be registered. It is possible to register file descriptor-like objects using a class implementing a handful of methods. Here is the implementation of such a class (adapted from PyNetSNMP, a subproject of Zenoss using the ctypes module):

class SnmpReader:
    "Respond to input events"
    implements(IReadDescriptor)

    def __init__(self, fd):
        self.fd = fd

    def doRead(self):
        netsnmp.snmp_read(self.fd)

    def fileno(self):
        return self.fd

Like for libevent, we have a function to update the list of SNMP related events (stored as a mapping between file descriptors and SnmpReader instances):

class Timer(object):
    callLater = None
timer = Timer()
fdMap = {}

def updateReactor():
    "Add/remove event handlers for SNMP file descriptors and timers"

    fds, t = netsnmp.snmp_select_info()
    for fd in fds:
        if fd not in fdMap:
            reader = SnmpReader(fd)
            fdMap[fd] = reader
            reactor.addReader(reader)
    current = Set(fdMap.keys())
    need = Set(fds)
    doomed = current - need
    for d in doomed:
        reactor.removeReader(fdMap[d])
        del fdMap[d]
    if timer.callLater:
        timer.callLater.cancel()
        timer.callLater = None
    if t is not None:
        timer.callLater = reactor.callLater(t, checkTimeouts)

Contrary to libevent, we cannot alter the main loop to call updateReactor() at each iteration. Therefore, it must be called after each SNMP related function.

For another example, have a look at my equivalent implementation as a C extension.

Miscellaneous

Lack of asynchronicity

Integrating NetSNMP into an event-based program is not risk-free. I have written a fairly comprehensive article on the lack of asynchronicity in NetSNMP AgentX protocol implementation. I urge you to read it if you want to integrate a SNMP subagent into an existing program.

On the manager side, you will get similar drawbacks when using SNMPv3. To retrieve or manipulate management information using SNMPv3, it is necessary to know the identifier of the remote SNMP protocol engine. This identifier can be configured directly on each manager but it is usually discovered by querying SNMP-FRAMEWORK-MIB::‌snmpEngineID, as described in RFC 5343.

Unfortunately, this discovery is done synchronously by NetSNMP. There is a bug report for this issue but it seems difficult to fix. I have not been able to come up with a proper patch but I have described a workaround around snmp_sess_async_send().

There is no such problem with SNMPv2.

Limitation of file descriptor sets

snmp_select_info() and snmp_read() uses the fd_set type to handle a set of file descriptors. This type should be manipulated with FD_CLR(), FD_ISSET(), FD_SET() and FD_ZERO(). Those may be defined as functions but they usually are macros. Moreover, no file descriptor greater than FD_SETSIZE (which is usually set to 1024) can be handled with the fd_set type. This means that if you have a file descriptor greater than 1024, you won’t be able to use snmp_select_info() with it.

Starting from NetSNMP 5.5, you can use snmp_select_info2() and snmp_read2() instead of snmp_select_info() and snmp_read(). They use the netsnmp_large_fd_set.

If you want to keep compatibility with NetSNMP 5.4, here is another twist. The fd_set type is usually a fixed-size array of long integers. FD_CLR(), FD_ISSET() and FD_SET() are size-independant. The size only matters for FD_ZERO() which is not used by either snmp_select_info() or snmp_read(). You can therefore allocate a larger fd_set. A common way is to compile your program with -D__FD_SETSIZE=4096. You can still use FD_ZERO() yourself. However, this is not portable (GNU C Library only). Another way is to allocate your own array of long integers and cast it to fd_set. You’ll have to redefine FD_ZERO():

typedef struct {
   long int fds_bits[4096/sizeof(long int)];
} my_fdset;

#undef FD_ZERO
#define FD_ZERO(fdset) memset((fdset), 0, sizeof(my_fdset))

{
   /* ... */
   my_fdset fdset;
   FD_ZERO(&fdset);
   /* ... */
   rc = snmp_select_info(&n, (fd_set *)&fdset, &timeout, &block);
   /* ... */
}

Threads

NetSNMP provides two API:

the traditional session API which is not thread-safe and
the single session API.

In the above examples, I have used the traditional session API. If you are using threads, you need to ensure that all SNMP operations are done in a single thread. Otherwise, you need to adapt the examples to use the single session API. snmp_select_info() is part of the traditional session API. You should replace it with snmp_sess_select_info() and keep a list of SNMP sessions.

This API is not available for the agent side of NetSNMP.

The event list is global but some code could be added to bind a different list for each base in case you use different bases. This is not needed in the case of lldpd which uses only one event base. ↩
TwistedSNMP is an SNMP protocol implementation for Twisted based on PySNMP but is not maintained anymore. PySNMP comes with examples on how to integrate with Twisted. ↩

Thibaut Girka: Multiarch Cross-toolchains

Mon, 28 May 2012 14:24:18 +0000

Hi! I have been selected to work on multiarch cross-toolchains as part of the Google summer of Code. Although coding officially started last week, I have been busy with exams and other uni stuff at the time.

Anyway, let's talk about the project itself, that is, multiarch cross-toolchains.

What is multiarch?

In order to understand what the project is about, one need to know what multiarch is. I'll quickly describe it, and I invite anyone interested to read the spec (really, it's interesting, and not that long). Basically, multiarch lets you install binary packages for multiple architectures on one system, and let package managers resolve dependencies as one would expect.

This works by identifying four sets of packages:

Packages that don't need to be aware of multi-arch. Such packages can only be installed for one architecture at a time, and can only be used to resolve dependencies of packages having the same architecture.

Those packages are “Multi-Arch: none” or “multiarch-unaware” packages.
Packages that can be installed for multiple architectures at a time, and can only be used to resolve dependencies of packages having the same architecture.

Those packages are “Multi-Arch: same” packages. They typically are libs.
Packages that can only be installed for one architecture at a time but can satisfy dependencies of any package regardless of its architecture.

Those packages are “Multi-Arch: foreign” packages. They typically are tools working with architecture-independent interfaces. (for instance, gpg, zenity or tar).
Packages that can only be installed for one architecture at a time but can satisfy any dependency annotated with “:any”, regardless of the architecture.

Those packages are “Multi-Arch: allowed” packages. They are packages providing both arch-dependent and arch-independent interfaces. They are typically things like python, which can be used to interpret arch-indep files, but providing arch-dependent interfaces for C extensions.

Anyway, multiarch is an amazing thing that may be of use in various cases, such as running i386-only software on an amd64 system, running armel software on an armhf system, running i386 software on an armhf system using qemu-user-static and binfmt, cross-grading (switching, for instance, from i386 to amd64 on a running system), and, in our case, easing cross-compilation.

State of cross-toolchains in Debian

There aren't really cross-toolchains available in Debian (although some can be built from source packages), but they are available in emdebian's repositories.

However, those packages predate multiarch, and don't make use of it. Instead, they depend on special packages converted using dpkg-cross. Such packages are merely arch: all versions of foreign packages, with files moved to arch-qualified paths to prevent conflicts with “normal” packages.

Such packages thus have to be generated from real packages, either manually or using some sort of automation. In addition, this additional step brings several issues, like keeping in sync converted packages with real packages from the archive.

With multiarch, such packages are rendered useless, and may in fact bring additional problems, since “converted” and multiarch packages may share the same files, and thus may not be co-installable.

Switching from dpkg-cross to cross-arch deps

Cross-toolchains packages currently depend on a few packages converted using dpkg-cross (ex: libc6-$ARCH-cross and libc6-dev-$ARCH-cross for some packages). They need to depend on the “real” package with the right arch (ex: libc6:$ARCH, libc6-dev:$ARCH).

The obvious way to do so is by annotating the dependency with a specific arch qualifier (ex: “Depends: libc6:armhf”). This is, however, not covered by the multiarch spec.

Note that some “lib*-$ARCH-cross” packages will remain (for instance, libgcc1-armhf-cross). Indeed, those packages are different from the packages mentioned above in that they are of the architecture of the build system and actually part of the toolchain itself.

Likewise, cross-toolchains may build-depend on foreign packages (ex: libc6-dev). Thus, specific arch qualifiers must be handled in the “Build-Depends” source package field too.

Changing dpkg and apt to support that should be fairly straightforward. In fact, I already have a few patches, but I need to refresh them, maybe clean them up a little, and push them a little harder to get them accepted!

Pietro Abate: managing puppet manifest with gitolite

Mon, 28 May 2012 10:18:35 +0000

Managing the puppet manifest using a vcs is a best practice and there is a lot of material on the web. The easier way to do it, is to use git directly in the directory /etc/puppet and use a simple synchronization strategy with an external repo, either to publish your work, or simply to keep a backup somewhere.

Things are a bit more complicated when you would like to co-administer the machine with multiple people. Setting up user accounts, permission and everything can be a pain in the neck. Moreover working from your desktop is always more comfortable then logging in as root on a remote system and make changes there ...

The solution I've chosen to make my life a bit easier is to use gitolite, that is a simple git gateway that works using ssh public keys for authentication and does not require the creating of local users on the server machine. Gitolite is available in debian and installing it is super easy : apt-get install gitolite .

If you use puppet already you might be tempted to use puppet to manage your gitolite installation. This is all good, but I don't advice you to use modules like this one http://forge.puppetlabs.com/gwmngilfen/gitolite/1.0.0 as it's going to install gitolite from source and on debian it's not necessary ... For my purposes, I didn't find necessary to manage gitolite with puppet as all the default config options where good enough for me.

Once your debian package is installed, in order to initialize your repo, you just need to pass to gitolite the admin public key, that is your .ssh/id_rsa.pub key and then run this command:

sudo -H -u gitolite gl-setup /tmp/youruser.pub

This will create the admin and testing repo in /var/lib/gitolite/repositories and setup few other things. At this point you are ready to test you gitolite installation by cloning the admin repo :

git clone gitolite@example.org:gitolite-admin.git

Gitolite is engineered to use only the gitolite use to manage all your repositories. To add more repositories and users you should have a look at the documentation and then editing the file conf/gitolite.conf to add your new puppet repository.

At this point, you can go two ways. If you use git to manage your puppet directory, you can just make a copy of it somewhere and then add gitolite as a remote

git remote add origin gitolite@example.org:puppet.git

If you didn't use git before, you can just copy the manifest in your new git repository, make a first commit and push it on the server.

git push origin master

Every authorized users can now use git to clone your puppet repository, hack, commit, push ...

git clone gitolite@example.org:puppet

One last step is to add a small post-receive hook on the server to synchronize your gitolite repository with the puppet directory in /etc : This will sync your main puppet directory and trigger changes on the nodes for the next puppetd run. First I created a small shell script in /usr/local/bin/puppet-post-receive-hook.sh :

#!/bin/bash

umask 0022

cd /etc/puppet

git pull -q origin master

This script presuppose that your git repo in /etc/puppet has the gitolite repo as origin ... Then I added a simple hook in the gitolite git repo that calls this script using sudo :

sudo /usr/local/bin/puppet-post-receive-hook.sh

And since you are at it you should also add a pre-commit hook to check the manifest syntax. This will save you a lot of useless commits.

If you use a more complicated puppet setup using environments (I'm not there yet, and I don't think my setup will evolve in that direction in the near future), you can use puppet-sync that seems a neat script to do the job.

For the moment this setup works pretty well. I'm tempted to explore mcollective to trigger puppet runs on my nodes, but I'm there yet...

Bastian Blank: Installing Wheezy on Thinkpad X130e

Sun, 27 May 2012 20:55:00 +0000

I installed a Lenovo Thinkpad X130e with Wheezy.

UEFI

The X130e does not want to boot via BIOS emulation. The Wheezy D-I installs only BIOS Grub. It does not provide the selection to use EFI at all. So the initial installation was BIOS only and EFI refused to boot it without an error. BIOS emulation simply don't want to work on the X130e.

Partman does not align partitions in GPT. For some reasons the partitions created by partman are not aligned in the partition table at all. This is bad for performance. Also it makes it impossible to create an EFI partition on it that is recognizable by Grub.

EFI Grub needs the EFI partition. This partition needs to be properly aligned and formatted with FAT16. It must be mounted on /boot/efi. Grub gets installed on this with the following commands:

apt-get install grub-efi-amd64
grub-install --bootloader-id=GRUB --removable

Grub

Grub fails to load EFI console stuff. Grub needs extra modules to allow the kernel to write on the console. Otherwise the kernel just hangs without output. This can be done via /boot/grub/custom.cfg to load them:

insmod efi_gop
insmod efi_uga

Radeon

The Radeon card needs a firmware. It just produces random output. To get a usable output, Radeon modeset must be disabled on the kernel command line:

radeon.modeset=0

Now the firmware can be installed:

apt-get install firmware-linux

Wireless card

The Broadcom wireless card needs a firmware:

apt-get install firmware-brcm80211

Colin Watson: OpenSSH 6.0p1

Sun, 27 May 2012 19:14:00 +0000

OpenSSH 6.0p1 was released a little while back; this weekend I belatedly got round to uploading packages of it to Debian unstable and Ubuntu quantal.

I was a bit delayed by needing to put together an improvement to privsep sandbox selection that particularly matters in the context of distributions. One of the experts on seccomp_filter has commented favourably on it, but I haven't yet had a comment from upstream themselves, so I may need to refine this depending on what they say.

(This is a good example of how it matters that software is often not built on the system that it's going to run on, and in particular that the kernel version is rather likely to be different. Where possible it's always best to detect kernel capabilities at run-time rather than at build-time.)

I didn't make it very clear in the changelog, but using the new seccomp_filter sandbox currently requires UsePrivilegeSeparation sandbox in sshd_config as well as a capable kernel. I won't change the default here in advance of upstream, who still consider privsep sandboxing experimental.

Gregor Herrmann: RC bugs 2012/21

Sun, 27 May 2012 19:04:55 +0000

new week, new report about my work on RC bugs. again, mostly gcc 4.7 related, & again, mostly using patches prepared by others.

~~#667111~~ – bcov: "bcov: ftbfs with GCC-4.7"
add patch from Cyril Brulebois, upload to DELAYED/2
~~#667308~~ – oggvideotools: "oggvideotools: ftbfs with GCC-4.7"
add patch from Paul Tagliamonte, upload to DELAYED/2
~~#667322~~ – pianobooster: "pianobooster: ftbfs with GCC-4.7"
add patch to fix Windows linker flags, upload to DELAYED/2
~~#667329~~ – powertop: "powertop: ftbfs with GCC-4.7"
add patch from Cyril Brulebois, upload to DELAYED/2
~~#667361~~ – rtorrent: "rtorrent: ftbfs with GCC-4.7"
add patch from Cyril Brulebois, upload to DELAYED/2, later fixed in maintainer upload
~~#667362~~ – rubberband: "rubberband: ftbfs with GCC-4.7"
add patch from Cyril Brulebois, upload to DELAYED/2
~~#667364~~ – rutilt: "rutilt: ftbfs with GCC-4.7"
add patch from Cyril Brulebois, upload to DELAYED/2
~~#667388~~ – tagcoll2: "tagcoll2: ftbfs with GCC-4.7"
manual close
~~#667403~~ – uisp: "uisp: ftbfs with GCC-4.7"
add compiler flag, upload to DELAYED/2
~~#667419~~ – xdiskusage: "xdiskusage: ftbfs with GCC-4.7"
add patch from Cyril Brulebois, upload to DELAYED/2
~~#667866~~ – src:cenon.app: "cenon.app: FTBFS with GCC-4.7"
add patch from Yavor Doganov, upload to DELAYED/2
~~#667873~~ – src:gridlock.app: "gridlock.app: FTBFS with GCC-4.7"
apply patch from Yavor Doganov, upload to DELAYED/2
~~#667874~~ – src:gworkspace: "gworkspace: FTBFS with GCC-4.7"
add patch from Yavor Doganov, upload to DELAYED/2
~~#672003~~ – src:dibbler: "dibbler: FTBFS: ClntIfaceIface.cpp:34:49: error: 'unlink' was not declared in this scope"
add patch from Paul Tagliamonte, upload to DELAYED/2
~~#672025~~ – src:mmpong: "mmpong: FTBFS: client.cpp:276:71: error: 'usleep' was not declared in this scope"
add patch from Cyril Roelandt, upload to DELAYED/2
~~#672042~~ – src:l2tp-ipsec-vpn-daemon: "l2tp-ipsec-vpn-daemon: FTBFS: src/VpnClientConnection.cpp:167:60: error: '::chown' has not been declared"
add patch from Paul Tagliamonte, upload to DELAYED/2
~~#672089~~ – src:icewm: "icewm: FTBFS: Xlib.h:1694:1: error: 'deprecated' was not declared in this scope"
add patch from Ubuntu / Andreas Moog, upload to DELAYED/2, then included in maintainer upload
#673286 – gem: "gem: FTBFS with gcc-4.7"
send debdiff to the BTS, later NMUd by a fellow DD
~~#674301~~ – src:libplack-middleware-reverseproxy-perl: "libplack-middleware-reverseproxy-perl: FTBFS: tests failed"
upload new upstream release (pkg-perl)
~~#674304~~ – src:clusterssh: "clusterssh: FTBFS: tests failed"
add missing build dependency
~~#674395~~ – src:libdbd-mysql-perl: "libdbd-mysql-perl: FTBFS: ld: cannot find -lssl"
manual close (pkg-perl)

Petter Reinholdtsen: Debian Edu interview: Ralf Gesellensetter

Sun, 27 May 2012 15:15:00 +0000

In 2003, a German teacher showed up on the Debian Edu and Skolelinux mailing list with interesting problems and reports proving he setting up Linux for a (for us at the time) lot of pupils. His name was Ralf Gesellensetter, and he has been an important tester and contributor since then, helping to make sure the Debian Edu Squeeze release became as good as it is..

Who are you, and how do you spend your days?

I am a teacher from Germany, and my subjects are Geography, Mathematics, and Computer Science ("Informatik"). During the past 12 years (since 2000), I have been working for a comprehensive (and soon, also inclusive) school leading to all kind of general levels, such as O- or A-level ("Abitur"). For quite as long, I've been taking care of our computer network.

Now, in my early 40s, I enjoy the privilege of spending a lot of my spare time together with my wife, our son (3 years) and our daughter (4 months).

How did you get in contact with the Skolelinux/Debian Edu project?

We had tried different Linux based school servers, when members of my local Linux User Group (LUG OWL) detected Skolelinux. I remember very well, being part of a party celebrating the Linux New Media Award ("Best Newcomer Distribution", also nominated: Ubuntu) that was given to Skolelinux at Linux World Exposition in Frankfurt, 2005 (IIRC). Few months later, I had the chance to join a developer meeting in Ulsrud (Oslo) and to hand out the award to Knut Yrvin and others. For more than 7 years, Skolelinux is part of our schools infrastructure, namely our main server (tjener), one LTSP (today without thin clients), and approximately 50 work stations. Most of these have the option to boot a locally installed Skolelinux image. As a consequence, I joined quite a few events dealing with free software or Linux, and met many Debian (Edu) developers. All of them seemed quite nice and competent to me, one more reason to stick to Skolelinux.

What do you see as the advantages of Skolelinux/Debian Edu?

Debian driven, you are given all the advantages of a community project including well maintained updates. Once, you are familiar with the network layout, you can easily roll out an entire educational computer infrastructure, from just one installation media. As only free software (FOSS) is used, that supports even elderly hardware, up-sizing your IT equipment is only limited by space (i.e. available labs). Especially if you run a LTSP thin client server, your administration costs tend towards zero.

What do you see as the disadvantages of Skolelinux/Debian Edu?

While Debian's stability has loads of advantages for servers, this might be different in some cases for clients: Schools with unlimited budget might buy new hardware with components that are not yet supported by Debian stable, or wish to use more recent versions of office packages or desktop environments. These schools have the option to run Debian testing or other distributions - if they have the capacity to do so. Another issue is that Debian release cycles include a wide range of changes; therefor a high percentage of human power seems to be absorbed by just keeping the features of Skolelinux within the new setting of the version to come. During this process, the cogs of Debian Edu are getting more and more professional, i.e. harder to understand for novices.

Which free software do you use daily?

LibreOffice, Wikipedia, Openstreetmap, Iceweasel (Mozilla Firefox), KMail, Gimp, Inkscape - and of course the Linux Kernel (not only on PC, Laptop, Mobile, but also our SAT receiver)

Which strategy do you believe is the right one to use to get schools to use free software?

Support computer science as regular subject in schools to make people really "own" their hardware, to make them understand the difference between proprietary software products, and free software developing.
Make budget baskets corresponding: In Germany's public schools there are more or less fixed budgets for IT equipment (including licenses), so schools won't benefit from any savings here. This privilege is left to private schools which have consequently a large share among German Skolelinux schools.
Get free software in the seminars where would-be teachers are trained. In many cases, teachers' software customs are respected by decision makers rather than the expertise of any IT experts.
Don't limit ourself to free software run natively. Everybody uses free software or free licenses (for instance Wikipedia), and this general concept should get expanded to free educational content to be shared world wide (school books e.g.).
Make clear where ever you can that the market share of free (libre) office suites is much above 20 p.c. today, and that you pupils don't need to know the "ribbon menu" in order to get employed.
Talk about the difference between freeware and free software.
Spread free software, or even collections of portable free apps for USB pen drives. Endorse students to get a legal copy of Libreoffice rather than accepting them to use illegal serials. And keep sending documents in ODF formats.

Andrew Pollock: [life] Dear World, Sarah's Twitter account is okay

Sun, 27 May 2012 13:57:00 +0000

I've received two separate emails now about a spammy looking tweet from Sarah's Twitter account, so I thought I'd attempt to head off any more.

Yes, it appears her account got hacked (weak password), but surprisingly the password didn't get changed. Sarah has changed her password to a stronger one and deleted the offending tweet. All is right with the world.

Ingo Juergensmann: DaviCal and Addressbook Sync

Sun, 27 May 2012 12:31:37 +0000

After exchanging my rusted Nokia N97 against an iPhone I was in need to setup calendar and addressbook syncing again. Addressbook syncing wasn't possible with N97 anyways, or I haven't found out how to do it. Previously I synced my N97 by using iSync, but iSync doesn't sync anymore with iPhone, although iPhone now syncs with iTunes. Weird? Yes. But that's how it works. The iPhone syncs now via WLAN instead of Bluetooth, which is an improvement, but I don't really want to fire up iTunes everytime I want to sync my calendar or addressbook. And using iCloud is really not an option as well, because of privacy concerns. I'm a big fan of selfhosting and already have a running DaviCal instance running on my server. DaviCal is a great piece of software from Debian maintainer Andrew McMillan, who is doing a survey on Davical, so there's, of course, a Debian package for it.

Anyway, one problem with OSX and addressbook sync via carddav is that it is not working out of the box with Addressbook.app on OSX, although the documentation in the DaviCal wiki is quite useful. When you try to enter a new account in Addressbook.app the sync will not work. The solution can be found on the private blog of Harald Nikolisin, which is in German. He writes (German, English translation follows)

Mac OS X Adressbuch anschliessen
Oh ja – wenn man mittels SSL drauzugreift, dann gibts Probleme.
Im der Applikation Adressbuch kann man zwar ein CardDAV Account anlegen bei dem man die Authorisierungsdaten und den kompletten Serverpfad (s.o.) eingeben kann, man läuft aber immer auf eine Fehlermeldung hinaus.
Die Lösung ist, zweimal “Create” anzuklicken um den fehlerhaften Account anzulegen.

Dann editiert man manuell folgende Datei:

~/Library/Application Support/AddressBook/Sources/UNIQUE-ID/Configuration.plist
Dort trägt man unter Server String die komplette URL ein.
https://SERVERNAME/davical/caldav.php/USERNAME/contacts
Am besten modifiziert man noch das Feld HaveWriteAccess auf den Wert auf “1″
English translation: 

Connecting Mac OS X addressbook
Oh, yes - there are problems when accessing via SSL.
In Addressbook.app you can add a CardDAV account where you can define authentication and 
server path, but you'll always get an error message.
The solution is to click twice on "Create" in order to create the faulty entry.

Then you can edit the following file:

~/Library/Application Support/AddressBook/Sources/UNIQUE-ID/Configuration.plist

There you enter your complete URL under Server String.
https://SERVERNAME/davical/caldav.php/USERNAME/contacts 
It's best to modify the field HaveWriteAccess to the value "1"

After following this advice my Addressbook.app did successfully stored the contacts into DaviCals CardDAV from where I can sync with my iPhone. Maybe Andrew want to include this to the DaviCal wiki, maybe I'll do this myself by registering in the Wiki for that purpose...

Oh, and I forgot: Using the Roundcube plugin from graviox is working nice as well with DaviCals CardDAV!

Kategorie:

Tags:

Lars Wirzenius: Obnam 0.29 (backup software)

Sun, 27 May 2012 10:51:05 +0000

I've just pushed out Obnam 0.29, my backup program. NEWS snippet below.

This is a RELEASE CANDIDATE for 1.0, since there are no known bugs that would block a 1.0 release. I'm not entirely happy with Obnam's performance over sftp with small files, but it's not something I am prepared to let block 1.0. I am going to try a few things to improve things, but I want this release out first.

Please test and report any problems via this mailing list or as bugs on the website or via IRC. I am on IRC only intermittently until 1.0 is released, so e-mail and bug reports are best options.

"obnam backup" now writes performance statistics at the end of a backup run. Search the log for "Backup performance statistics" (INFO level).
"obnam verify" now continues past the first error. Thanks to Rafał Gwiazda for requesting this.
Add an obnam-viewprof utility to translate Python profiling output into human readable text form.
Bug fix: If a file's extended attributes have changed in any way, the change is now backed up.
"obnam fsck" is now a bit faster.
The shared directories in the repository are now locked only during updates, allowing more efficient concurrent backups between several computers.
Obnam now gives a better error message when a backup root is not a directory. Thanks to Edward Allcutt for reporting the error (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654211).
The output format of "obnam ls" has changed. It now has one line per file, and includes the full pathname of the file, rather mimicking the output of "ls -lAR". Thanks to Edward Allcutt for the suggestion (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=655095).
A few optimizations to sftp speed. Small files are still slow.

Lars Wirzenius: Larch 1.20120527 (Python B-tree library)

Sun, 27 May 2012 10:04:12 +0000

It has been a long time since I announced a version of Larch on my blog: about 14 months. A fair bit has happened since then: see the NEWS file for details. Below is the snippet for today's release.

This is an implementation of particular kind of B-tree, based on research by Ohad Rodeh. See "B-trees, Shadowing, and Clones" (copied here with permission of author) for details on the data structure. This is the same data structure that btrfs uses. Note that my implementation is independent from the btrfs one, and might differ from what the paper describes.

The distinctive feature of this B-tree is that a node is never modified (sort-of). Instead, all updates are done by copy-on-write. Among other things, this makes it easy to clone a tree, and modify only the clone, while other processes access the original tree. This is utterly wonderful for my backup application, and that's the reason I wrote larch in the first place.

See the home page and README for more information; the API docs may be amusingly sparse.

Version 1.20120527, released 2012-05-27

New version scheme. Thank you, Joey Hess.
The on-disk data structures and file formats are now declared frozen. An automatic test has been added to verify that things do not break.

Andrew Cater: Raspberry Pi - its's here in volume

noreply@blogger.com (Andrew Cater) — Sun, 27 May 2012 09:19:16 +0000

The Raspberry Pi's are now shipping in volume and are gradually spreading. The site at raspberrypi.org has links to a map showing the spread worldwide. This post is being edited from a Pi on my desk and using Midori as a lightweight browser.

After all the hype: it works, it runs Debian Squeeze well and can update from Debian apt repositories. Graphics drivers are non-free but I'd hope there would be a way to work together with the development team to get them put into the non-free repositories.

Blogger doesn't like the browser and suggests an update to Google Chrome but that's life, I suppose :(

Bartosz Feński: cowbuilder with sid under Ubuntu/Mint

Sun, 27 May 2012 07:12:39 +0000

Just for reference cause I had to figured it out again after upgrade from Mint Lisa to Mint Maya (which in Mint world means reinstalling whole system).

This is the way to create cowbuilder environment with Debian Sid/unstable under Linux Mint Maya or any Ubuntu based system:

sudo sed -ie 's/MIRRORSITE.*/MIRRORSITE=http://ftp.debian.org/debian/g' /etc/pbuilderrc sudo apt-get install debian-archive-keyring sudo cowbuilder --create --distribution sid --debootstrapopts --keyring=/usr/share/keyrings/debian-archive-keyring.gpg

Moray Allan: DebConf9.0.1

Sat, 26 May 2012 19:31:52 +0000

Vincent Sanders: Each morning sees some task begun, each evening sees it close; Something attempted, something done, has earned a night's repose.

noreply@blogger.com (Vincent Sanders) — Sat, 26 May 2012 18:20:30 +0000

Thursday saw all the Collabora employees at Cambridge office go out and socialise at the beer festival. They seemed to have selected a wonderful day for it, the sun was shining and it was warm and blue sky day.

Alas, I had to attend some customer conference calls and work on some time sensitive research so I could not go to the ball as it were. At about eight my brain had run out of steam so I decided to call it a day and go and meetup with people at the festival for an hour or two.

The queue when I arrived dissuaded me from that notion. I asked one of the stewards and they indicated it would take at least an hour from where the queue finished.

So I decided to wend my way home along the bank of the Cam. I proceeded slowly along and to my utter surprise bumped into Ben Hutchings and his Solarflare work colleagues having their own soiree. I was immediately invited to sit and converse. Pretty quickly I was inveigled into accepting a glass of wine by John Aspden from his floating bar (AKA houseboat).

From here on my evening was a pleasant one of amusing new people, easy conversation and a definite pondering if the host would be discovering the delights of Cam swimming as he became progressively inebriated!

So although I missed the festival I did manage to have an enjoyable time. A big thanks to the solarflare guys and especially John who was the consummate host and provided me with far too much alcohol.

Petter Reinholdtsen: The cost of ODF and OOXML

Sat, 26 May 2012 16:00:00 +0000

I just come across a blog post from Glyn Moody reporting the claimed cost from Microsoft on requiring ODF to be used by the UK government. I just sent him an email to let him know that his assumption are most likely wrong. Sharing it here in case some of my blog readers have seem the same numbers float around in the UK.

Hi. I just noted your http://blogs.computerworlduk.com/open-enterprise/2012/04/does-microsoft-office-lock-in-cost-the-uk-government-500-million/index.htm comment:

"They're all in Danish, not unreasonably, but even with the help of Google Translate I can't find any figures about the savings of "moving to a flexible two standard" as claimed by the Microsoft email. But I assume it is backed up somewhere, so let's take it, and the £500 million figure for the UK, on trust."

I can tell you that the Danish reports are inflated. I believe it is the same reports that were used in the Norwegian debate around 2007, and Gisle Hannemyr (a well known IT commentator in Norway) had a look at the content. In short, the reason it is claimed that using ODF will be so costly, is based on the assumption that this mean every existing document need to be converted from one of the MS Office formats to ODF, transferred to the receiver, and converted back from ODF to one of the MS Office formats, and that the conversion will cost 10 minutes of work time for both the sender and the receiver. In reality the sender would have a tool capable of saving to ODF, and the receiver would have a tool capable of reading it, and the time spent would at most be a few seconds for saving and loading, not 20 minutes of wasted effort.

Microsoft claimed all these costs were saved by allowing people to transfer the original files from MS Office instead of spending 10 minutes converting to ODF. :)

See http://hannemyr.com/no/ms12_vl02.php and http://hannemyr.com/no/ms12.php for background information. Norwegian only, sorry. :)

Tanguy Ortolo: Fun with spammers

Sat, 26 May 2012 12:56:00 +0000

Email spam is a plague, but sometimes there is at least on category of spammers it is easy to have fun with: naive manual spammers. People that try to do more or less legal business, using Internet in an illegitimate way, for instance (this is a real example, including the spammer's address: spamming bots, please use it ;-) !):

From: Emma Perroni <emmaperroni@gmail.com>
Subject: Hello

Hello,

Do you have a Facebook page? If you do, I can offer you to add fans by
hunders/thouthands in a single day using an original (and confidential)
technique. We can also add subscribers to Twitter, Youtube or Google+1.

Prices: 10€ for 100 fans, -20% for more than 1000 fans ordered.

Regards

Reporting

Well, in that case it may or may not be a scam, but even if it is not a criminal activity such as selling fake drugs, it is not a very honest business. Anyway, that lady is using her Google account to send her crap, so neutralizing her is quite easy:

checking that it really comes from Google Mail, using DKIM or SPF;
writing to Google, reporting that spam (and, in that specific case, the abuse they are offering to commit on Google +1);
optional, do not do that with scammers: replying to the spammer explaining that this is spamming, that you reported it and that she can expect to loose her Google account and everything on it.

That should work with any good email provider, that is, any provider that listens to complaints at <abuse@> and takes action on them. For what I have seen, Google and Yahoo! are, but fossils such as traditional national telephonists are not. If the message headers allow you to identify it, you can also report the abuse to the spammer's network operator if it is a good one (this is rarely the case).

Consequences

With a bit of luck, that can seriously damage the business of the spammer, and if he was really careless (e.g. storing important documents on the same Google account with no local backup), cause serious trouble in his personal stuff. If you replied to the spammer, you can get fun answers such as (these are taken from real examples I got):

Ha ha ha, why not calling the FBI? That's it, laugh while you can, do you think your provider does not care? Too bad, he does, so you may change of mood when you see the effects loosing one's online account.
We are doing legal business, you will stop your harrassment or we will file a complain. Yes, please do so if you dare, but may I remember you that you are the one at fault here? (in fact, such a reaction is a very good sign, since it shows that the spammer is actually afraid, probably for his business)
Why did you do that? I promise, I shall not do it again, please… Too late, lamer, you should have thought of that before sending your crap…

Conclusion

This is too complicated to be done for each spam of course, but it is worth doing for spam messages that are not catch by one's antispam. And the rules for email marketing are well known: only write to people that explicitly allowed you to do so, full stop, so there is no reason to show any mercy for a spammer.

Craig Small: JFFNMS 0.9.3

Sat, 26 May 2012 06:44:26 +0000

jffnms version 0.9.3 has been released today. This is a vast improvement over the 0.9.x releases and anyone using that train is strongly recommended to upgrade.So what changed? What didn’t change! A nice summary would be fixing a lot of things that were broken or needed some tweaking. A really, really big thanks to Marek for all the testing and bug reports and also patient “just run this and tell me what it says” tests he did too. If something wasn’t right before and works now, it is quite likely it is working because Marek told me how it broke.

A brief overview of what has changed:

TFTP transfers work again
A lot of the wierd polling effects due to caching fixed
Lots of the selects in sub-tables now work
The PHP string-to-float brokeness in SLAs worked-around
Even more SNMP library cruft removed or escaped
HostMIB apps match properly
Interface autodiscovery delete and update fields back working

You can download the file off sourceforge at

https://sourceforge.net/projects/jffnms/files/JFFNMS%20Releases/

PHP floats and locales (enc.com.au)

Joey Hess: the rocket

Sat, 26 May 2012 03:29:40 +0000

The best thing about rockets is when they go up successfully, and don't come back down.

My git-annex Kickstarter reached escape velocity 24 hours after launch, and is now past 200% funded.

This is great news, because it gives me more time to spend hacking on git-annex, and will let me add more features and polish it better. I had set the goal at a minimal amount because I'd have hated not to get funded at all if this hadn't taken off. But the current funding is much more comfortable, and the further up it goes, the more scope the project can have.

At the point shown above, the project had started to be highlighted as popular, based largely on kind and generous readers of this blog and git-annex users who chipped in. For a while, I personally knew around 1/3rd of contributors.

That was enough to get it noticed by Kickstarter staff, which in turn has led to more growth. I love the juxtaposition of geeky tech project with other stuff here.

I have 17 days remaining until the Kickstarter is finished. I really had worried it might take that long to get funded. As it is, I can't wait to see where the rocket's trajectory takes it in that time!

Charles Plessy: Cloud-init uploaded to Debian.

Sat, 26 May 2012 02:12:45 +0000

Each instance (virtual machine) started from the same image is configured identically. The same account and the same procedure will be used to connect to it. If the machine image is public and if the instance is open to the Internet, which is common when using commercial providers such as Amazon, an attacker could connect randomly to known IPs from the cloud, and eventually access to an instance before its owner.

To individualise the connection to each instance, one can pass informations at startup, called instance metadata, which will be available to this instance only at a fixed HTTP URL. If the owner provides a public SSH key, this can be used to ensure that only the owner of the private key can connect.

After the next update of network-console, this method of connection will be available for the Debian Installer. Likewise for the Debian images produced, by installing the cloud-init packages, that I uploaded to the experimental section of our archive. This package is far from ready. It lacks init scripts, it is not translated, and to be honest, I have not tested it. But to have it in Debian makes me benefit from the package and bugs tracking systems. Do not hesitate to report some, or better, to participate to the maintenance of this package.

Cloud-init has many other interesting functionalities. I would like to thank Scott Moser, the upstream developer and maintainer of the Ubuntu package, for his cooperation during the creation of the Debian package, as I made large modifications, in particular the removal of the part managing the menu.lst for pv-grub, which will have its own package or will be transferred somewhere else.

Alexander Reichle-Schmehl: Release Critical Bug report for Week 21

alexander@schmehl.info (Alexander Reichle-Schmehl) — Fri, 25 May 2012 13:17:27 +0000

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:	1534
Affecting Wheezy:	1095
Wheezy only:	238
Remaining to be fixed in Wheezy:	857

Of these 857 bugs, the following tags are set:

Pending in Wheezy:	82
Patched in Wheezy:	175
Duplicates in Wheezy:	62
Can be fixed in a security Update:	27
Contrib or non-free in Wheezy:	11
Claimed in Wheezy:	2
Delayed in Wheezy:	18
Otherwise fixed in Wheezy:	86

Ignoring all the above (multiple tags possible) 500 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released.

However, with the view of the Release Managers, 975 need to be dealt with for the release to happen.

Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Johannes Schauer: setting up mailman, postfix, lighttpd

Fri, 25 May 2012 07:42:00 +0000

I was worried about having to learn hundreds of configuration options to properly set up mailman, postfix and lighttpd on Debian Squeeze. Turned out, that except for lighttpd it all works out of the box.

apt-get install postfix

When asked by debconf, I specified lists.mister-muffin.de as the fully qualified domain name.

apt-get install mailman
newlist mailman

The newlist command reminds me that I have to add its output to /etc/aliases. After doing so, I have to run:

newaliases

From now on, I can add any mailinglist by running newlist, editing /etc/aliases and running newaliases.

Mailinglists can also be added through the mailman webinterface but one still has to put the according entries into /etc/aliases.

Following is a working lighttpd configuration that works out of the box with the default settings of mailman on Debian squeeze.

This was the only part that caused me some headaches.

server.modules += ("mod_alias", "mod_cgi", "mod_accesslog")

$HTTP["host"] == "lists.mister-muffin.de" { accesslog.filename =
    accesslog.filename = "/var/log/lighttpd/lists-access-log"

    alias.url += (
        "/cgi-bin/mailman/private/" => "/var/lib/mailman/archives/private/",
        "/cgi-bin/mailman/public/" => "/var/lib/mailman/archives/public/",
        "/pipermail/" => "/var/lib/mailman/archives/public/",
        "/cgi-bin/mailman/"=> "/var/lib/mailman/cgi-bin/",
        "/images/mailman/" => "/usr/share/images/mailman/",
    )

    cgi.assign = (
        "/admin" => "",
        "/admindb" => "",
        "/confirm" => "",
        "/create" => "",
        "/edithtml" => "",
        "/listinfo" => "",
        "/options" => "",
        "/private" => "",
        "/rmlist" => "",
        "/roster" => "",
        "/subscribe" => "")
}

server.document-root        = "/var/www"
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"
index-file.names            = ( "index.html" )
server.dir-listing          = "disable"
include_shell "/usr/share/lighttpd/create-mime.assign.pl"

As a bonus, I wanted to import my existing email exchange with my GSoC mentors into the mailinglist. First I was planning on manually sending the email messages to the list, but a much easier option is to just import them in mbox format.

To extract all email messages, I first wrote the following python snippet:

import mailbox, itertools
box = mailbox.mbox('~/out')
for message in itertools.chain(mailbox.mbox('~/sent'), mailbox.Maildir('~/Mail/Web/', factory=None)):
    if (("wookey" in message.get('to', "").lower()
      or "wookey" in message.get('cc', "").lower()
      or "wookey" in message.get('from', "").lower()
      or "abate" in message.get('to', "").lower()
      or "abate" in message.get('cc', "").lower()
      or "abate" in message.get('from', "").lower())
      and not message['subject'][0] == '['
      and not message['subject'] == "multistrap"):
        box.add(message)
box.close()

It iterates through messages in my mbox and maildir mailboxes, filters them for emails by wookey or pietro, strips away some messages I found to not be relevant and then saves the filtered result into the mbox mailbox ~/out.

It is important to specify factory=None for the Maildir parser, because it otherwise defaults to rfc822.Message instead of MaildirMessage.

Also do not forget to call box.close(). I initially forgot to do so and ended up with missing messages in ~/out.

I then copy the archive in its place:

scp out lists.mister-muffin.de:/var/lib/mailman/archives/private/debian-bootstrap.mbox/debian-bootstrap.mbox

Another thing that initially caused me trouble, was that the mbox didnt have the correct permissions due to the scp. Fixing them:

chown -R list:www-data /var/lib/mailman/archives/private/
chmod 664 /var/lib/mailman/archives/private/debian-bootstrap.mbox/debian-bootstrap.mbox

And update the mailman archive like this:

sudo -u list /usr/lib/mailman/bin/arch debian-bootstrap /var/lib/mailman/archives/private/debian-bootstrap.mbox/debian-bootstrap.mbox

Initially I was running the above command as root which screws up permissions as well.

Matthew Garrett: Fedora 17 and Mac support

Fri, 25 May 2012 03:42:53 +0000

Fedora 17 will be shipping next week. It's got a bunch of new features, none of which I contributed to in the slightest. What I did work on was improving our support for installation on x86 Apple hardware. There's still a few shortcomings in this so it's not an announced or supported feature, but it's sufficient progress that it's worth writing about.

There's a few ways that Apple platforms differs from normal PC hardware. The first is that Apples are very much intended to be EFI first and BIOS second, while that's still not really true for commodity PC hardware. Apple launched Boot Camp shortly after they started shipping x86 Macs (and shortly after I'd got Ubuntu running on one for the first time[1]) but the focus of Boot Camp has always been to be good enough to boot Windows and not much else[2]. The other is the integration with the boot environment. You can boot Linux easily enough by setting the standard EFI boot variables, but if you ever boot back into OS X it's easy to trigger deletion of those. There's then no straightforward way of resetting them, and you're left having to recover your installation.

The other difficult bit has been actually starting the installer at all. If you're happy to use BIOS emulation than this can be done without too much misery, but you're then left with dealing with Apple's custom synchronised GPT/MBR. More modern Macs will boot via the standard EFI mechanisms, but there's a range of problems that can be caused that way. Fedora 17 is the first Linux distribution to ship install media that will EFI boot a Mac when either written to optical media or a USB stick[3]. Put the install media in your system and then either select it in the OS X Startup Disk preferences menu and reboot, or reboot and hold down the alt key. In the latter case there'll be a nice Fedora logo. Click on it and it'll boot.

The biggest difference between Apple hardware and anyone else is that we'd normally put the bootloader in a FAT partition that's shared with any other installed operating systems. That does actually work on Apples, but then you run into the earlier point I mentioned. The Apple startup picker that you get by holding down the alt key doesn't pay any attention to the standard EFI boot variables, so it won't show a FAT partition merely because it's got an EFI bootloader on it. The same is true of the OS X Startup Disk preferences. The best way to get a drive to appear in the menu is to make it look like an OS X drive.

This was problematic in a few ways, due to the requirement for an OS X drive to be HFS+. The first was that we didn't have any support for that in our installer, so work had to be done there. The second was that the state of HFS+ was woefully poor in Linux[4]. Linux will refuse to write to an HFS+ filesystem if it hasn't been cleanly unmounted or fscked, and since we can't rely on everyone always shutting their machine down cleanly that means a working fsck.hfsplus. In theory we were shipping one of those. In practice, it reliably segfaulted on 64-bit systems[5]. So I had to package the latest version of Apple's code, and doing that involved dealing with the fact that Apple now use blocks almost as much as Grub 2 uses nested functions, and that meant using Clang and that meant dealing with a bug where Clang segfaulted if it had been built with gcc 4.7[6], but finally once all of those yaks had been shaven we could trust our filesystem.

Of course, more problems existed. You can't just drop a bootloader onto an HFS+ partition and expect it to work. You need to write its inode value into the superblock. Trivial, except that if you do this from userspace then the kernel kindly overwrites your update when you unmount the disk and it updates the superblock. That required a mechanism for updating that data via the kernel, which meant adding an ioctl. This would have been fine, except the OS X Startup Disk preference looks for a bootloader in a location other than where Fedora installs it. Changing our bootloader install path wasn't a great option, so the easiest thing to do was just to link them together. Except HFS+ doesn't really support hardlinks. When you hardlink something the original file gets moved into a magic directory in the filesystem root and the links are special files that refer to it. And it turns out that it's not actually the inode that the firmware wants when finding the bootloader, it's the catalogue ID, and these aren't the same when hardlinks are involved. So, another kernel patch.

All that was left for the boot partition then was adding an icon and a drive label. We already had code for generating the icon, and the drive label wasn't that difficult. Hurrah!

Ok, so we can construct a filesystem that (a) appears in the bootpicker, and (b) appears in the OS X Startup Disk preferences. We can even put a bootloader on it. This led to the next problem. The bootloader started without any trouble. But the bootloader couldn't read any files off the boot partition, including its configuration. This should have been obvious with hindsight - the problem was simply that grub legacy[7] doesn't support HFS+. Writing an HFS+ driver seemed like an excessive amount of work, especially since the firmware already supported reading HFS+, so instead I wrote a simple grub filesystem driver that uses the UEFI interfaces to read files.

A working bootloader! One more problem there. grub looks in its startup directory to find its configuration file. We have two "copies" of grub hardlinked to each other, each looking in a different directory for configuration. Hardlinking the config files together worked fine, except that most tools that update config files have this irritating habit of writing to a temporary file and moving that over the original, thus breaking the hardlink. Easily solved by using a symlink instead, except that the UEFI filesystem semantics don't really support symlinks because UEFI only really envisaged using FAT. If you use Apple's UEFI HFS+ driver to open a symlink, you get a short file containing the target path of the symlink. Rather less than ideal, and a rather gross hack to cope with it.

All set now, other than a bug in some older Mac firmware where read would return BUFFER_TOO_SMALL without returning the real buffersize and the odd way that Apple treat CDs. Oh, and a bug in Apple's Broadcom wifi driver that could overwrite the kernel, and case-sensitivity, something that obviously isn't something you often hit on UEFI when all you usually see is FAT.

Kernel-side, we had a couple of things. Obviously there the bugs I'd mentioned in the past, but those had already been dealt with. New issues included the fact that we were frequently picking the wrong framebuffer address, especially on machines with multiple GPUs. Fixed now, along with another issue where we'd sometimes get confused by models with different GPU configurations. And at that point, most of the implementation work was done.

Of course, there was also the work of integrating this into the tools that we use to generate our install media, and I'd like to thank Will Woods, Brian Lane and Dennis Gilmore for the work they put into that, along with anyone I've forgotten. Tom Calloway handled getting the device label graphics generated right before deadline. A cast of thousands helped with testing.

So why did I mention shortcomings? First, this only works on machines with 64-bit firmware. Early 64-bit Apples still had 32-bit firmware. In theory we can support that case - in practice it's a moderate amount of writing, including some mildly awkward kernel code. But anything later than mid-2007 should have 64-bit firmware, so it's not a massive concern. Second, we seem to have fairly significant trouble with Radeon hardware on a lot of Macs. We fixed one bug there, but something's still going wrong in setup and you'll frequently end up with a black screen. And thirdly, I'm sure there's some other machines that still don't work for (as yet) undetermined reasons.

With luck we'll get these things dealt with by Fedora 18. But even with these flaws, Fedora 17 will work fine on a lot of Apple hardware, and testing it is as simple as downloading and booting a live image. Bugs go in the usual place.

[1] Note my charming belief that the Ubuntu installer would fully support this hardware via EFI in the near future. 6 years ago. It still doesn't.
[2] This can be fairly easily seen from little things like the name "Windows" being hardcoded into every UI that sees a Boot Camp drive.
[3] I cover this in more depth here
[4] To be fair, it mostly still is - there's plenty of work to be done there.
[5] Approximately nobody had noticed this, which is all kinds of unreassuring.
[6] Thanks to Kalev Lember for dealing with that
[7] Still the EFI bootloader in Fedora 17. We've already swapped Fedora 18 over to grub 2.

comments

Paul Tagliamonte: playing with non-lintian lintian checks

Fri, 25 May 2012 00:25:03 +0000

Crazy title, I know.

Over the past $WHILE, I’ve found myself checking for the same things. Some of these are fit for inclusion into lintian it’s self — others, not so much.

Here’s a rundown on what my lintian wrapper (tragically also called “lintian” and just higher up on my $PATH. I know it’s a hack.)

Since I only use lintian on .changes files - it pre-processes for some information that is stuffed in the changes.

The first step is to get all the bugs that the package closes. It then takes these bugs, and queries the BTS to see what package they’re filed against. If the bug is filed against a different bug then the package’s source, it’ll emit a warning. It has a greylist for stuff like “WNPP”. This will fail if something’s filed against it’s binary package rather then it’s source :)

Next, the wrapper will get “other” info from the changes. It checks if the suite is in the blacklist (“UNRELEASED”, and “testing”, (stable for me as well, since I don’t usually deal with SRUs often)).

It’ll also check for the changed-by and maintainer, ensuring they’re the same. This will fail for co-maint’d packages.

If you’ve not guessed by now, this system relies on false-positives :)

Nextly, if there are .debs in the Files, it’ll go through and check the control of each. It’ll print out a stanza with the short-description inside a template, to see if the description “works”.

After that, it’ll attempt to make HTTP requests to the Homepage (and eventually URLs in the description and VCS-Browse)

If the package has “python” in the source name anywhere, or in any of it’s binary packages, it’ll trigger jwilk’s lintian4python over the source as well.

After this, it passes through to regular lintian.

Oh, and it colorizes all the output :)

Here is a (mostly lint-free) package:

As you can see, it’s very (very) verbose, but i’d rather cross stuff off a checklist then worry about something being off.

Anyway, it’s a work in progress. Not even a serious one, at that. It’s a bunch of hacks put together, so I can’t really endorse anyone else using this stuff. If you feel crazy enough to do so, all the tools are found on my dot-bin repo on GitHub

Vincent Sanders: Interrupt Service Routines

noreply@blogger.com (Vincent Sanders) — Thu, 24 May 2012 22:44:21 +0000

Something a little low level for this post. I have been asked recently how to "test" for the maximum duration of an Interrupt Service Routine (ISR) in Linux

To do this I probably ought to explain what the heck an ISR is!

A CPU executes one instruction after another and runs your programs. However early in the history of the electronic computer it soon became apparent that sometimes there were events happening, generally caused by a hardware peripheral, that required some other code to be executed without having to wait for the running program to check for the event.

This could have been solved by having a second processor to look after those exceptional events but that would have been expensive, difficult to synchronize and the designers took the view that there was a perfectly good processor already sat there just running some users program. This Interruption in the code flow became known as, well, an Interrupt (and the other approach as polling).

The hardware for supporting interrupts started out very simply, the processor would complete execution of the current instruction and when the Program Counter (PC) was about to be incremented if an Interrupt ReQest (IRQ) was pending the PC would be stored somewhere (often a "special" IRQ stack or register) and then execution started at some fixed address.

The interrupting event would be dealt with by some code and execution returned to the original program without it ever knowing the CPU just wandered off to do something else. The code that deals with the interrupt is known as the Interrupt Service Routine (ISR).

Now I have glossed over a lot of issues here (sufficient to say there are a huge number of details in which to hide the devil) but the model is good enough for my purpose. A modern CPU has a extraordinarily complex system of IRQ controllers to deal with numerous peripherals requesting the CPU stop what its doing and look after something else.

This system of controllers will ultimately cause program execution to be delivered to an ISR for that device. If we were living in the old single thread of execution world we could measure how long execution remains within an ISR, perhaps by using a physical I/O line as a semaphore and an external oscilloscope to monitor the line.

You may well ask "Why measure this?" well historically while the ISR was running nothing else could interrupt it executing which meant even if there was an event that was more important it would not get the CPU until the first ISR was complete. This was known as IRQ latency which was undesirable if you were doing something that required an IRQ to be serviced in a timely manner (like playing audio)

This is no longer how things are done while the top half runs with IRQ disabled many are threaded interrupt handlers and are preemptable (I.e. can be interrupted themselves) which leads to the first issue with measuring ISR time in that the ISR may be executed in multiple chunks if something more important interrupts. Indeed it may appear an ISR has taken many times longer one time than another because the CPU has been off servicing multiple other IRQ.

Then we have the issue that Linux kernel drivers often do as little as possible within their ISR, often only as much as is required to clear the physical interrupt line. Processing is then continued in a "bottom half" handler this leads to ISR which take practically no time to execute but processing is still being caused elsewhere in the system.

The next issue is the world is not uniprocessor any more, how many processors does a machine have these days? even a small ARM SoC can often have two or even four cores. This makes our timing harder because it is now possible to be servicing multiple interrupts from a single peripheral on separate cores at the same time!

In summary measuring ISR execution time is not terribly enlightening and almost certainly not what you are interested in. The actual question is much more likely that you really want to be examining something that the ISR time was an historical proxy for like IRQ latency or system overheads in locking.

Mark Brown: Anne Brown (1946-2012)

Thu, 24 May 2012 22:13:37 +0000

BROWN Anne After a long illness bravely fought, Anne Margaret (née Wilson), wife of Fred and mother to Mark, enthusiastic Scottish country dancer. Service to be held at Borders Crematorium, on Friday, May 25, 2012, at 11 am. Donations, if desired, to Marie Curie Cancer Support. Family flowers only.

Junichi Uekawa: A few weeks ago, my sony VAIO type P that I have been using for the past 2 years have died.

Thu, 24 May 2012 22:07:17 +0000

A few weeks ago, my sony VAIO type P that I have been using for the past 2 years have died. I've now got my MacBook Air and back to being productive again. One thing is that I probably lost your mail since I recovered from my home directory backup from 5 May. Not that I read mail all that often, but your mail will certainly be lost if you've sent me a mail in the meanwhile.

Mark Brown: regulator updates in 3.4

Thu, 24 May 2012 18:08:44 +0000

This has been a fairly quiet release from a regulator point of view, the only real framework features added were devm support and a convenience helper for setting up fixed voltage regulators. Much more coming next time, though! The most noticeable thing in the changelog is that Axel Lin continued his relentless and generally awesome stream of fixes and cleanups.

Managed (devm) versions of regulator_get() and regulator_bulk_get() simplifying error handling and resource management for drivers.
Added a convenience interface for setting up fixed voltage regulators.
Device tree support for TWL4030.
New drivers for Freescale i.MX Anatop, Samsung S5M8767, TI TPS62360 and TPS65271 regulators.
Removed BQ24022 in favour of the more generic gpio-regulator driver.

Mark Brown: regmap updates in 3.4

Thu, 24 May 2012 18:07:16 +0000

Things are really quieting down with the regmap API, while we’re still seeing a trickle of new features coming in they’re getting much smaller than they were.

Support for padding between the register and the value when interacting with the device. This is required by some devices with high speed control interfaces in order to give the device time to get the values ready.
Support for applying register updates to the device when restoring the register state. This is intended to be used to apply updates supplied by manufacturers for tuning the performance of the device (many of which are to undocumented registers which aren’t otherwise covered). For want of a better term this feature is known as a patch.
Support for two bit address, six bit value registers contributed by Wolfram Sang.
Support for multi-register operations on cached registers contributed by Laxman Dewangan.
Some additional diagnostics available via debugfs for the cache state.
Support for syncing only part of the register cache, useful for devices with power domains or sub devices which can be reset independently.
Stubs and parameter query functions intended to make it easier for other subsystems to build infrastructure on top of the regmap API.

plus the general bug fixes and tweaks you’d expect.

Sylvestre Ledru: llvm & clang 3.1 in Debian unstable + GSoC

Thu, 24 May 2012 16:13:37 +0000

Released a few days ago, I am happy to announce that LLVM and clang 3.1 are already available in the Debian archive in Unstable. RC versions were already available for the last few weeks in Debian experimental. They are very likely to be part of wheezy.
We will soon try a new rebuild of the Debian archive with the version 3.1.

Talking about clang, a couple of months ago, I proposed a GSoC project to extend the Debian infrastructure in order to use clang instead gcc to build Debian packages. The goal is not to replace gcc but it is more about building packages with a new compiler.

With Paul Tagliamonte as co-mentor, Alexander Pashaliyski is going to work during the summer on this subject.

The final deliverables should be:

Build services modified to be able to use clang instead of gcc/g++
Integration into the Debian infrastructure (buildd.d.o, and similar services)
An easy way to switch of compiler between gcc/g++ and clang
A way to build a package with a different compiler (gcc, clang, intel compiler, etc)
Time permitting, a fully clang-built Debian installation

For now, Alexander is working on an installation of the wanna-build and buildd services in order to have a dedicated testing environment.

Richard Hartmann: Public service announcement

Thu, 24 May 2012 15:06:32 +0000

This will be a repost for anyone subscribed to Planet Debian, sorry.

In case you have not heard about git-annex yet and you care about syncing, tracking, and managing your private or public files, go click the link; it's OK, I will wait. The use cases sound intriguing, don't they? But unless you are comfortable with a shell and git, don't bother using it; pain is inevitable.

joeyh announced a kickstarter project to make git-annex more user friendly, support automated adding and syncing of files, and a web front-end. Notwithstanding the cut that kickstarter and amazon payments will take and the inherent questions that are being raised by paying for FLOSS, this is a very worthwhile endeavour to support.

PS: Yes, this feels a tad circle-jerky inasmuch joeyh's post links to my blog; I would have linked to his post anyway, though. If anything, this made me think twice about posting this here.

PPS: Yes, I will continue writing my Trans-Siberian series even though I am home already. Air China decided to keep me in China longer than planned, real life is busy, and, truth be told, Diablo 3 came out. The second Mongolian post is half-way finished, at least.

Vincent Sanders: Linux kernel presentation

noreply@blogger.com (Vincent Sanders) — Thu, 24 May 2012 10:41:37 +0000

Recently I was asked to present a short introduction to the Linux kernel for our project managers. I put together a short slide deck for the presentation which I have decided to share.

I feel its important to note that I had a lot more to say about each section and the slides were more an aid for my memory to cover the important points. Of special note would be the diagram showing the "hierarchy" of contributors, this is of course nowhere near as well stratified as portrayed.

Ingo Juergensmann: About Gallery3 in Debian and MySQL

Thu, 24 May 2012 05:49:40 +0000

Years ago, when I started using a cheap Kodak DX3600 digital camera to make some digital photos, I used Gallery from Menalto to collect these pictures within a gallery. Gallery (version 1) was using plain text files to keep its information about galleries and photos and as the number of photos I put into the gallery the more it got slower and slower. Then Gallery2 was released which used a database, either MySQL or PostgreSQL, and was a huge improvement in speed. My main galleries do have about 10-20.000 pictures each. But Gallery2 is aged nowadays and the next logical step would be to migrate to Gallery3. But what a mess!

Gallery3 has some drawbacks:

there is currently no gallery3 package in Debian, although it's been released upstream for some time now.
there is actually a bug open (#511715), stating that there are some license issues with Gallery3 and some SWF files
it's been said that Gallery3 doesn't support per picture permissions anymore, only permission per album is now possible, which is giving me headache as I changed permissions of some personal pictures in the past for privacy reasons and which would either lead to completely unavailable albums to the public or the need of splitting album into a public and a private section, which breaks the timely order of the pictures
whereas G2 supported both MySQL and PostgreSQL as database backends, G3 only supports MySQL. That's a real pitty because I prefer PostgreSQL because of its stability and easiness over MySQL. It already happened several times that MySQL databases were gone after a kernel crash or something. Even mysql.user table was gone more than once, whereas PostgreSQL never ever has shown such behaviour to me. It just works.

I'm really upset about the last point! Why is there such a strong believe in MySQL. In my eyes, MySQL is utter crap. It's more like MS Access than a real thing when talking DBMS/SQL. And my impression is that, after Oracle bought MySQL, Oracle did a good job to scare their customers off. PostgreSQL on the other hand gained a good momentum in usage since the Oracle/MySQL deal. So, it's a total mystery to me how a big software package with dozens of developers can decide not to support PostgreSQL or even drop the support of PostgreSQL for a new release! It's driving me nuts, again and again.

Other big software packages like Drupal are doing the right thing: while PostgreSQL support in drupal6 was weak and buggy because of all these awful MySQLisms around, Drupal now uses a database abstraction layer in drupal7 to allow even sqlite or Oracle to be used as underlying database. That's the way to go, but it's totally awkward to drop PostgreSQL support as Gallery3 did.

So, is there a way out of my dilemma? Will gallery3 be a package within Debian soon (no offense to Michael Schultheiss, I think he's doing a great job and needs assistance from upstream in this case)? Is there any good replacement for Gallery3 that can deal with tenthousends of images and dozens of users and supports PostgreSQL and has some kind of import tool from gallery2?

Kategorie:

Tags:

Gintautas Miliauskas: HP LaserJet 1018 on Ubuntu

noreply@blogger.com (Gintautas Miliauskas) — Wed, 23 May 2012 18:39:37 +0000

An offtopic post for an issue that I just ran into for the n-th time: after upgrading to Ubuntu Precise, my HP LaserJet 1018 stopped working again, the issue being that the printer firmware was not being uploaded to the printer properly. For posterity, (one) way to fix this is to uninstall the printer-driver-foo2zjs package, remove the printer itself from the list of printers, and then use HP utilities to set up the printer:

sudo apt-get remove printer-driver-foo2zjs

sudo apt-get install hplip-gui

sudo hp-setup

You may need to turn the printer off and on after the procedure to trigger the firmware uploader.

Planet Debian

Julian Andres Klode: Memory allocators

Richard Hartmann: Potpourri II

Kicking things along

RAID 4

Tom Tom vs OSM

Windows

UEFI

Wrapping up

Marc 'Zugschlus' Haber: atop in unstable

Matthew Garrett: Implementing UEFI Secure Boot in Fedora

Getting the machine booted

Bootloaders

Kernel

Wait signed what

Customisation

But I don't trust Microsoft

You've sold out

What about ARM

Is this all set in stone?

Paul Tagliamonte: my sbuilder setup (sb)

Jaldhar Vyas: Dovecot 2.1.7 Uploaded to Unstable

Lior Kaplan: Debian packaging for beginners @ FOSDEM – the video

Thomas Goirand: New release of MLMMJ (version 1.2.18.0) uploaded in Debian

Sorina Sandu: Initial commit

Michal Čihař: Four counferences in Prague this October

Lucas Nussbaum: Debian archive rebuilds on Amazon Web Services

Raphael Geissert: ldebdiff: what local change did I make to a package?

Bartosz Feński: dibbler 0.8.2

Christian Perrier: 2012 update 24 for Debian Installer localization

Christian Perrier: 12 languages to be deactivated in Debian Installer

Paul Tagliamonte: Goofing off with sbuild "variants"

Paul Tagliamonte: My first foray into non-compliant Debian packages

Daniel Kahn Gillmor: KVM, Windows XP, and Stop Error Code 0x0000007B

Jon Dowland: Debian Day

Bartosz Feński: burp 1.3.6

Steve Kemp: A heady mixture of photography and programming

Niels Thykier: Lintian 2.5.7 and 2.5.8

Michal Čihař: Android support in Gammu

Martin-Éric Racine: OpenWRT: accepting Router Adverts from the ISP in White Russian?

C.J. Adams-Collier: Ohai! It looks like I’m back online!

Jonathan McDowell: Go go gadget multiarch

Vincent Bernat: Integration of NetSNMP into an event loop

Own custom event loop

Third-party event loop

libevent

Twisted reactor

Miscellaneous

Lack of asynchronicity

Limitation of file descriptor sets

Threads

Thibaut Girka: Multiarch Cross-toolchains

What is multiarch?

State of cross-toolchains in Debian

Switching from dpkg-cross to cross-arch deps

Pietro Abate: managing puppet manifest with gitolite

Bastian Blank: Installing Wheezy on Thinkpad X130e

UEFI

Grub

Radeon

Wireless card

Colin Watson: OpenSSH 6.0p1

Gregor Herrmann: RC bugs 2012/21

Petter Reinholdtsen: Debian Edu interview: Ralf Gesellensetter

Andrew Pollock: [life] Dear World, Sarah's Twitter account is okay

Ingo Juergensmann: DaviCal and Addressbook Sync

Lars Wirzenius: Obnam 0.29 (backup software)

Lars Wirzenius: Larch 1.20120527 (Python B-tree library)

Version 1.20120527, released 2012-05-27

Andrew Cater: Raspberry Pi - its's here in volume

Bartosz Feński: cowbuilder with sid under Ubuntu/Mint

Moray Allan: DebConf9.0.1

Vincent Sanders: Each morning sees some task begun, each evening sees it close; Something attempted, something done, has earned a night's repose.

Petter Reinholdtsen: The cost of ODF and OOXML

Tanguy Ortolo: Fun with spammers

Reporting

Consequences

Conclusion

Craig Small: JFFNMS 0.9.3

Related articles