1. after your buggy program was slowly eating your memory and swap, and you escaped to a console to kill it using Alt+SysRq+R and then Ctrl+Alt+F1, when you get back to your X-session you can make it grab the keyboard again by typing “kbd_mode -s” in an xterm (found here after trying all sorts of incantations in DuckDuckGo)
2. if that was on a laptop keyboard with “AltGr dead keys”, the SysRq combo might need to be Fn+SysRq+AltGr+R (yes, try that with one hand!), where AltGr is the right Alt key (found after lots of failed attempts – very frustrating if your swap runs out before you get to the console)
3. in the end, you wouldn’t typically have needed any of the above if only you’d known that “ulimit -v 1000000” limits any process in your bash shell to 1GB of memory, so that your buggy program will die superquick without locking up the system for 10 minutes (found in no time through a most obvious query – it just took a long time before I thought of looking for it at all…)

Ksplice Uptrack is a new service that lets you effortlessly keep your systems up to date and secure, without rebooting.

Once you’ve completed the easy installation process, your system will be set up to receive rebootless updates instead of traditional, disruptive updates.  […]

Ksplice, Inc. is proud to make this service freely available for the latest version of the world’s most popular desktop Linux distribution: Ubuntu 9.04 Jaunty Jackalope.

No more reboots, and still applying security patches as soon as they become available. That’s worth the dist-upgrade hassle.

For now, all I did was running a quick test. I had a USB stick with Ubuntu Netbook Remix 9.04 lying around, so I booted from that, hooked up the wifi (man, connecting is fast with NetworkManager 0.7-something – another reason to upgrade…), downloaded ksplice-uptrack.deb, and installed it on the Live system (you also need network connectivity to fetch some dependencies from the Ubuntu repository). This is what you get:

There’s a little tray-icon (the one resembling a “K”…) informing you that kernel updates are available, and clicking it opens an update window. Nothing exciting to see here, actually.

Still not very exciting. The whole thing is very understated, almost disappointingly so – I mean, something this cool should look cool, shouldn’t it?

…. and everything still works after this. In fact, I’m typing this post from the Live system with the (supposedly) updated kernel. I tried shutting the lid on my D630, and it nicely went into ACPI suspend. And came back up.

Wicked.

(Small disappointment: it seems Firefox crashed between suspend and resume. Did it a second time, and again Firefox died. Third time: no problems. Not sure if this has anything to do with anything, so for now pretend I didn’t mention it.)

Cool stuff, seriously. This will be in 10.04 by default, I’ve no doubt. In case you’re looking, here’s one guy eager to work on that!

One more thing: in their FAQ they suggest a little test to demonstrate that the thing actually does something. I tried their suggestion and ran their test-thing a couple of times. But I’m off to bed now, so here’s the output, and I’ll leave calculating whether the difference before/after updates is statistically significant to you…

ubuntu@ubuntu:~$ wget -O demo.c http://www.ksplice.com/uptrack/2009-06-demo.c
ubuntu@ubuntu:~$ gcc demo.c -o demo
ubuntu@ubuntu:~$ sudo cpufreq-selector -c 0 -g performance
ubuntu@ubuntu:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
ubuntu@ubuntu:~$ sudo cpufreq-selector -c 1 -g performance
ubuntu@ubuntu:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz
stepping        : 6
cpu MHz         : 2101.000
cache size      : 3072 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 4189.64
clflush size    : 64
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz
stepping        : 6
cpu MHz         : 2101.000
cache size      : 3072 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 4189.57
clflush size    : 64
power management:

ubuntu@ubuntu:~$ ./demo
time to write 100 lines is 6(msec)
# ...hmmm, wait, this is a Live system...
ubuntu@ubuntu:~$ sudo mount /dev/sda3 /mnt
ubuntu@ubuntu:~$ cd /mnt/
ubuntu@ubuntu:/mnt$ sudo mkdir test
ubuntu@ubuntu:/mnt$ sudo chmod a+rwx test
ubuntu@ubuntu:/mnt$ cd test/
ubuntu@ubuntu:/mnt/test$ cp /home/ubuntu/demo .
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 49(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 54(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 64(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 60(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 75(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 72(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 62(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 65(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 80(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 52(msec)
ubuntu@ubuntu:/mnt/test$ sudo uptrack-remove --all -y
The following steps will be taken:
Remove [cdoprpi1] Performance regression in filesystem buffer code.
Remove [9xoc5qmo] Possible erroneous memory overcommit in program start.
Remove [ll9q1ymc] Multiple bugs in filesystem core.
Remove [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Remove [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Remove [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Remove [xgqc9vy4] VGA console corrupts non-ASCII characters.
Remove [pdfrn6qa] Denial of service by evading CPU time limits.
Remove [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
Removing [cdoprpi1] Performance regression in filesystem buffer code.
Removing [9xoc5qmo] Possible erroneous memory overcommit in program start.
Removing [ll9q1ymc] Multiple bugs in filesystem core.
Removing [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Removing [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Removing [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Removing [xgqc9vy4] VGA console corrupts non-ASCII characters.
Removing [pdfrn6qa] Denial of service by evading CPU time limits.
Removing [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 816(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 805(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 793(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 786(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 785(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 787(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 791(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 787(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 786(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 785(msec)
ubuntu@ubuntu:/mnt/test$ sudo uptrack-upgrade -y
The following steps will be taken:
Install [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
Install [pdfrn6qa] Denial of service by evading CPU time limits.
Install [xgqc9vy4] VGA console corrupts non-ASCII characters.
Install [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Install [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Install [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Install [ll9q1ymc] Multiple bugs in filesystem core.
Install [9xoc5qmo] Possible erroneous memory overcommit in program start.
Install [cdoprpi1] Performance regression in filesystem buffer code.
Installing [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
Installing [pdfrn6qa] Denial of service by evading CPU time limits.
Installing [xgqc9vy4] VGA console corrupts non-ASCII characters.
Installing [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Installing [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Installing [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Installing [ll9q1ymc] Multiple bugs in filesystem core.
Installing [9xoc5qmo] Possible erroneous memory overcommit in program start.
Installing [cdoprpi1] Performance regression in filesystem buffer code.
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 61(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 56(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 47(msec)
ubuntu@ubuntu:/mnt/test$

A quick snippet from the project page:

Ksplice enables running systems to stay secure without the disruption of rebooting.  Specifically, Ksplice creates rebootless updates that are based on traditional source code patches.  These updates are as effective as traditional updates, but they can be applied seamlessly, with no downtime.

Ksplice currently supports updating the Linux kernel, but the core technology applies to any operating system or to user space applications.

A quick search tells me even ZDNet had already heard of this project over a year ago, so I’m half ashamed that it’s news to me, but I’m too excited to keep it to myself :)

It’s a terrible idea to vest this much power with one company, even one as fun, user-centered and technologically excellent as Google. It’s too much power for a handful of companies to wield.

The question of what we can and can’t see when we go hunting for answers demands a transparent, participatory solution. […]

I completely agree with him that there’s a problem here, in fact also for at least one other reason which he didn’t mention. That reason also invalidates the solution he seems to propose – a sort of non-profit search giant under public control. Scroll down a few sections if you want to hear an alternate proposal…

Search giants slow innovation

Monopolists kill innovation even if they’re trying hard not to be evil, simply because monopolies kill innovation. There’s a specific problem with Search, in that it costs a boat-load of money to just start out doing it, let alone improve on anything. You’ll always have to index the whole internet, for example – no matter how good your algorithms, nobody will use your service if you don’t have good coverage. After Cuil, venture capitalists may hesitate to cough-up that sort of money.

Only a handful of companies have the means to put up a hundred-thousand servers and compete with Google. After more than half a decade, Microsoft now managed to produce Bing, which from my impressions so far is on par with Google Search. Read that again: half a decade – on par. What about innovation? Where’s the PageRank killer? What happened to those big leaps of progress that led to Google?

This is not Microsoft’s failure. The guy that might have had the hypothetical breakthrough-new idea just might have happened to work at another cool company, one that didn’t have the money to dive into Search. I’d say this is rather a failure of the free market (but see my About page: I’m not an economist – I have really no idea what I’m talking about :)). Every hypothetical insurgent has to overcome a multi-million dollar hurdle just to take a shot at the problem. That means there will always be too few candidates.

Paul Graham thinks it takes a different kind of investors to tackle the problem – ones that have the guts to throw money at this. I think we should better find a way to bring the cost down. But, let’s quickly shoot at the idea of a non-profit first.

A non-profit would kill innovation

As in completely, totally kill it. A public, participatory system is what you settle for when you want stability: it thus necessarily opposes innovation. You want a stable government, so you build a democracy. But you leave innovation to the free market, because innovating under parliamentary oversight would take forever.

Just imagine what would happen: we’d settle on, say, Nutch, throw a huge amount of public money at it, and then end up spending that money on endless bureaucracy – some users want this innovation, some that, others want to try something totally different instead, academics get to write papers about how it could all be better, the steering committee gets to debate it too, and then when a decision is near, there will be endless rounds of appeal…

(Doctorow realises this, as he writes “But can an ad-hoc group of net-heads marshall the server resources to store copies of the entire Internet?”)

Federation

We want to achieve two goals: the one that Doctorow outlined, which I will rephrase as “Search services that transparently serve the interests of all those who search as well as all those who want to be found” (with some legal limits to it of course), and the fast-innovation goal, which I think boils down to this: start-ups shouldn’t need to build every aspect of the search engine just to get to improve one aspect of it. The following is a rough outline of a crazy idea, and again: I have no idea what I’m talking about. Here we go…

Let’s call the people who search consumers, and the ones who want to be found providers. If you look at how the Google platform works internally, you’ll see there’s roughly a separation that reflects the presence of these two parties: there are index and document servers (let’s call them the back-end) that represent the providers, and there’s the front-end that handles a consumer’s query, talks to the index/document servers, and compiles a priority list for the consumer.

In the age of dial-up connections, you had to have all that happen within the data center. There’s a massive amount of communication between the back-end and the front-end servers. So it had to be designed the way it was. Now that there’s fat bandwidth all-over, couldn’t the front-end servers be separated from the back-end servers?

As a consumer, I’d get to deal with a front-end-providing company that would serve my interests, and my interests only. A natural choice would be my ISP, but as a more extreme solution the front-end could run on my desktop machine – the details don’t matter for now. The point is, there could be many of these front-ends, and I could switch to a different solution if I wanted more transparency (in that case I’d get an open-source solution, I guess) or if I wanted the latest and greatest.

All these front-ends would deal with many back-end servers – just like it is now, because the internet can just not be indexed on only a few machines. But they wouldn’t have to be owned by one company: there could be many. As a provider, then, I’d also have a choice of companies that would compete to serve my interests – they wouldn’t certainly not drop me from their index (as in Doctorow’s problem outline), because I’m paying them. A natural choice for this would be my hosting company, but if they do a bad job (too slow, wrong keywords, whatever), I could fire them and go somewhere else.

(Big parties like Akamai or Amazon would be at a small advantage here, having a lot of server power to handle many index queries, but small parties could cut deals with other small parties to mirror each others’ servers – heck, I’m thinking about details again!)

Note that in addition, providers are in a much better position to index their documents than search-engine crawlers currently are. They could index information that crawlers may not get to – this is the main goal of the more narrowly defined federated search that Wikipedia currently serves up for that term. What’s proposed here is bigger – all-inclusive.

So who does the PageRanking?

There’s a little problem of course, in that the above is not an accurate picture of how stuff works. At Google, the back-end servers have to also store each site’s PageRank, and the front-ends rely on that for their ordering work. In the federated model, there would be some conflict of interest there: wouldn’t the providers bribe their back-end companies to game the system?

If all the companies involved were small enough, then no. If one back-end would return dishonest rankings, it would quickly become known among the front-ends, and they would drop this back-end from their lists. That’s similar to what Google does and what Doctorow is worried about, but there’s a big difference: if your back-end company behaves in this way, and you suffer as a provider, you can leave them and find a more respectable back-end. Honest providers would not have to suffer.

What about innovation? For one scenario, let’s say I’m a new front-end company and I want to replace PageRank by my innovation called RankPage. I’d have to get all the back-end guys to give me some sort of access to their servers to get to calculate RankPage. But that should (in theory, at least) be relatively easy: they don’t stand to lose anything, except maybe some compute time and sysadmin hours. If I turn out to be onto something, I’ll become a big front-end, driving a lot of consumers to them – that is, helping me try my innovation is ultimately in the best interest of the providers they serve. Note that nobody incurs high costs in this model.

(I’m having a really hard time stopping myself from thinking about details here, but let’s say a good front-end in this federated-search world would be able to deal with heterogeneity, where some back-ends respond with PageRank, some also provide RankPage, and some do yet something else…)

(And for more irrelevant details: we would also see many more specialist front-ends appear, that serve consumers with very specific interests. Could be cool!)

Why it won’t happen anytime soon

While the front-ends and back-ends could have many different implementations, they would have to somehow be able to speak to each other in a very extensible language (we don’t want to end up with something like email – built on a hugely successful protocol, that however doesn’t even facilitate verifying the originator of a message!). That extensibility is pretty difficult to design, I imagine.

(Perhaps superfluously noted: it’s crucially important to establish a protocol, and not an implementation. If we’d settle for a federated version of Nutch, however good it may be, there’s no way to innovate afterwards.)

What’s also difficult to deal with, is the chicken-and-egg problem: no consumers will come unless all providers are on-board on this, and why would the providers participate? I could see a few big parties driving this process though – parties that want to become less dependent on Google (and Bing, and Yahoo Search).

Looking at how long it’s taken to establish OAuth (and that still has the job of conquering the world ahead of it), this might really take a while to come together.

But wouldn’t it be cool…

Two things I thought:

It’s worth mentioning that Ian Murdock said this will form the basis for “Solaris 11”.

The ARM port makes a lot of sense to me. I can imagine companies being interested in having all their employees’ smartphones becoming first-class members of their company computing ecosystem (did I just write that monster sentence??). I’m sort of thinking: MacOS will be able to offer this, but in a more closed flavour, Linux will be able to offer this, but in a more heterogeneous flavour, and Solaris could sort of offer the middle ground between those.

I know, I probably sound as “head in the clouds” as Jonathan Schwartz right now. Anyway, having multiple solutions can only be good: some companies will be looking for the flexibility of Linux offerings, but others may like the fact they can get it all from one vendor, who not only provides support but also holds the reins on development. A winner in any case will be ARM…

—

Through the Ubuntu Weblogs feed I read a post at The Open Sourcerer about a piece of software by a company called MiserWare (yeah, the name inspired the pun even before I concluded as much…). They’ve released a private beta of MicroMiser, a closed-source package that promises a substantial power-consumption reduction on x86 Linux systems. Quoting from their website:

MicroMiser typically lowers total system energy use by 10-35% even when a system is 100% utilized.

That’s interesting. Even better, Alan wrote a follow-up post at The Open Sourcerer which shows that on his Ubuntu Server system the software estimates its own power savings to be close to 65% (!). Now, assuming for the moment that it’s not a bug in the calculations in the beta (hey, I’m in a mild mood tonight), I’d say those 65% would be really worthwile.

Alan was also kind enough to send me an invitation for the beta, and so – always the curious type – I was starting to make a plan on how to test this thing: use a dummy installation (we don’t trust closed-source betas, do we?), current-monitoring at the wall-socket, that sort of thing.Then, I figured I’d download the package first to see what was in it. A kernel module? Lots of ugly scripts? Who knows…

I don’t, anyway. I never got to the download step. In the step before that I was presented with a rather restrictive license. I quote verbatim (only the change to bold-type is mine, all text is original):

5.    Confidentiality

The Software and Documentation, all related Intellectual Property, and any information learned or discovered by You about the performance of the Software in the course of use under this License constitutes proprietary trade secret information owned solely by Licensor (collectively, “Confidential Information”).  You agree that You will not, without the express prior written consent of Licensor, (1) use the Confidential Information other than to use the Software as authorized by this License Agreement; (2) disclose any Confidential Information to any third party; or (3) fail to use best efforts to safeguard the Confidential Information from unauthorized use, copying, or disclosure. You acknowledge that a breach of this Section 5 may cause the Licensor irreparable harm and damages that are difficult to ascertain.  Therefore, the Licensor, upon a disclosure or threatened disclosure of any Confidential Information, will be entitled to injunctive relief (without the requirement of posting bond), without limiting its other remedies under this License Agreement, in equity or at law.  The obligations of this Section shall survive this License Agreement without limitation in duration.  By clicking the “I Agree” button below, You consent to having any information that You provide to the Licensor processed and stored in the United States.

Well, I’m still not a lawyer, nor am I a native speaker, but I think that this basically means I would not be allowed to tell you what I would read on my current monitor. What the….

I also think that this means that the post reporting the 65% savings is violating the license.

After a long half second of thinking about it, I did not agree, so I can’t tell you anything about the software – by the way, even if I had agreed, I still wouldn’t be able to tell you anything! My advice? Don’t waste your time with bloody legalese and read some real information instead: Intel has a very informative website on the topic at LessWatts.org.

This is exactly what I usually tell people who ask why I’m running “this Linux thing”. It’s better suited to my needs on every level of the software stack. Of course, that’s usually way too vague to compel someone who had to ask to begin with. Examples please? My new favourite example is a security thing.

The other week I was catching up with a few long-overdue admin tasks on my parents’ PCs, mostly security updates. They’re (still) on MS Windows, and you should have seen the number of pop-ups when I logged on as admin: I don’t know how many apps, all reporting they’d like permission to fetch and install updates. Compare that to the elegance of the little warning star in the Gnome menu, or the output of a quick “aptitude upgrade”.

Big deal? I think so. I’m quite sure that that single interface to all software updates is not just more elegant, but that it’s also directly keeping systems more secure. Quoting from an old eWeek item:

For example, for the nine highest-profile Windows malicious code incidents as of March 2003, Microsofts patches predated major outbreaks by an average of 305 days, yet most firms hadnt applied the patches.

That is not a statement about Windows, or Linux. It’s a statement about human nature.

Here’s another item, from Ars:

Secunia cited data from Microsoft showing that third-party software vulnerabilities are the ones that are most frequently exploited, and said that its own data showed that users simply don’t update as frequently as they should.

Having a clear, simple, and non-crappy upgrade manager vastly diminishes these problems, because all people are lame, and the number of people that will not apply updates promptly will go up at least quadratically with the number of steps the update takes (and that’s a conservative estimate). That’s why distros will win from bare operating systems with apps dropped on top of them. It’s also why I’m going to press mom and dad to please let me replace their systems…

Is this benefit intrinsically tied to free software? In theory, no. I actually tried to pitch this idea to the Ideastorm crowd at some point. But maybe it would not work so well in practice: if MS would try to turn Windows into a distro, or would try to press other vendors into using their update manager, the anti-monopolistic regulators would be all over them in no time. So in practice, one might say this is a benefit of free software. Yay.

Update 2: I came across some Ubuntu-specific notes which should be useful too.

—

I was a bit surprised by Dell’s announcement just now: it seems they’ve gone through some real efforts to make Ubuntu 8.04 look all 2009ish on their latest netbook entry-level notebook. Let me try and be all 2009ish too, and include some of their Flash-terrorism YouTube footage:

Don’t worry,  I haven’t suddenly become a sucker for fancy “iconography” (apparently that has a whole different meaning “in the office of the CTO” – don’t take this badly, Dell, I am a fan!), nor have I finally grown fond of NetworkManager, but seriously: WWAN support is pretty cool, and having it neatly integrated is even better.

Actually, I’m just really pleased to see what importance Dell seem to be assigning their Ubuntu programme these days. This little cheapling has Intel’s Poulsbo chipset (just like the Mini 12), which is great for power consumption, but apparently a real chore when it comes to Linux support. As far as I’m aware, no other vendor is even trying to support it. In addition, also to keep the price low, there’s the ever troublesome Broadcom wifi chip, “official” Linux-support for which was supposedly also negotiated by Dell. I’m guessing there was no way they’d get all that working on 9.04, but to make up for that they backported the NM improvements and updated the “iconography” (sorry).

I’m just saying, that’s a lot of effort when you have a business partner called Microsoft that’s giving you Windows XP basically for free…

Ok, enough Ubuntu/Dell fanboyism for one day. Good night!

P.S.: Rik – if you happen to read this – I didn’t know they had a 6-cell battery coming (I told the guy to get a Samsung instead…)!

Maybe I’m just stoopid, but let’s assume I’m not. I hope this respin ranks a bit better for keywords, so I can help some other lost souls find the site that I found. If you want the expert story, click straight through to the source – actually, that whole site is simply excellent, and well worth browsing thoroughly if you’re looking to learn cool sysadmin stuff and more.

I learned (at least) two new things today:

• how to extract the rsync command that your locally-executed rsync sends to the remote machine’s ssh-server
• how to make the ssh-server execute that exact command, regardless what someone tries to feed it from the local machine

But first, let me explain (to myself) why I wanted to know these things.

The issue with remote backups

You want off-site backups, because, well, that’s rule #3. But, by rule #2, you also want to backup often, and there’s only one way to guarantee that that will work out: automation. There’s a problem with that though: to attain automation, your remote backups will need some unprotected authentication token, e.g. an ssh-certificate with an empty passphrase.

Obviously, you want to restrict what that dangerous key lets you do on the remote system. Simply put, you don’t want someone that managed to break into your backup client to be able to erase both your backups and the originals. The solutions I had seen so far included creating a separate backup-user on the server, and providing a restricted shell of some sort. That’s one way of doing things, but it’s not easy to set up:

• you only want to allow a select few commands, say rsync for transport, and perhaps some scripts to prepare the backup
• ideally you want to have read-only access so that the client performing the backup cannot damage files, which might even occur without malicious intent, say by a wrong string of rsync options
• but maybe you want to run some sort of hotcopy command on some database you’re using, and this does require write access
• do you create yet another user for that?
• and are you sure your shell is really as restricted as you think? No tricks to break out of it?
• aaaaaahhrghh…..

Right. I’m *that* mistrusting, and especially when it comes to my own competence. I’d definitely bork that restricted shell setup. Please give me something dead simple.

Figuring out what your local rsync needs from the remote rsync

Okay. Assume we’ll always call the server with the exact same rsync command, perhaps something like

bash$ rsync -avz remote_host:/var/backup/ /var/remote_host_backup

(On a side-note, I’m still doubting myself every time: trailing slash or no trailing slash? Terrible.)

Now, you can see what rsync command will get executed on the remote host if you add another -v:

bash$ rsync -avvnz remote_host:/var/backup/ /var/remote_host_backup

where I also added an -n to have a dry-run. The first line of output reads something like

opening connection using ssh remote_host rsync --server --sender -vvnlogDtprz . /var/backup

…which runs off the page here because I didn’t pay WordPress for my own custom CSS yet, but you can try this yourself anyway. What we’re interested in is the part that starts at “rsync”: this is what is executed on the remote host.

Using sshd with the command=”” option

Remember we’re using a passphrase-less ssh-certificate for the sake of automation. On the server, that requires an entry like this in $HOME/.ssh/authorized_keys:

ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end

The sshd manpage tells you you can insert quite a few options at the start of this line. You should really consider all of those options, but the cool one for now is the command=”” option. Between the quotes we put the result of the previous section minus the -n (or you’ll have only dry runs…).

command="rsync --server --sender -vlogDtprz . /var/backup" ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end

…that’s probably running off the page big time now. Sorry. And I didn’t even add all the other restrictive options you ought to consider.

The beauty of this is that sshd will now ignore whatever abuse you’re feeding it from the ssh client. Whenever you authenticate using this specific certificate, it will only run that exact command.

Let me put this  yet another way. The only way to successfully talk to the server with that certificate is to say what it expects you to say: you can only run the matching local rsync command or the two rsync instances will not understand eachother. All the options are fixed, client-side and server-side.

This is what you want. Or, it is what I wanted, anyway.

What about running scripts before the actual rsync?

Okay, I learned a third thing. This was in the rsync manpage: your remote rsync “can be any program, script, or command sequence you’d care to run, so long as it does not corrupt the standard-in & standard-out that rsync is using to communicate”.

In other words: you can run any database hotcopy command on the server, as long as it cares to shut up, so that to the client, it looks as if only rsync was called. Your authorized_keys entry now looks somewhat like this:

command="do_db_hotcopy >> /var/log/hotcopy.log 2>&1 ; rsync --server --sender -vlogDtprz . /var/backup" ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end

… where you’re being careful to make sure the only output sent comes from rsync. This works for me; I could imagine a long script might cause your local rsync to time-out in some way, so ymmv.

One more thing

I’ll shut up soon, too, but there was actually also a fourth thing… how do you make sure your local rsync command uses the restricted, passphraseless key under all circumstances? When I’m actually logged in myself, often ssh-agent is keeping my less-restricted key available. The problem with this is that ssh will prefer using that key, but when I use that, my fancy hotcopy (from the previous section) never gets called.

To fix this, my backup script on the client contains an extra -e option to rsync, which is self-explanatory, but that’s not enough: ssh still prefers the key held by ssh-agent. The full solution (as the ssh-agent manpage more or less documents) is thus:

#! /bin/bash
unset SSH_AUTH_SOCK
rsync -avz -e 'ssh -i .ssh/restricted_key' remote_host:/var/backup/ /var/remote_host_backup

Sometime soon I might respin this whole thing with rdiff-backup (…you want to keep multiple states of your backup, because, well, that’s rule #4 :P). I just need to figure out how client-server communication works for that.

It occurred to me that, after a patch came through for some udev vulnerability, a Debian 5.0 system that I’m running did not prompt me for a reboot, whereas Ubuntu 8.04 on my laptop did. Indeed, the related USN mentions a reboot as necessary, and there’s no such mention in the corresponding DSA. A quick look at the process IDs on the Debian system shows that the update just restarted udevd. Now, I understand too little of the intrinsics to judge if this was impossible for the Ubuntu system, so I’d like to put that out there as a question: why couldn’t we just restart udevd on Ubuntu?

In fact there was a similar issue some time ago. A NetworkManager update came through, and then I was prompted to reboot. As far as I could tell, restarting dbus was actually sufficient here to start using the updated binaries (but of course the update manager thingy kept bugging me to reboot nonetheless…). APT could have done that automatically, I’d think.

So what’s the policy with that? Just reboot to be on the safe side, avoid peculiarities with some users’ systems? User friendliness in some intractable way or another? Any thoughts?

(On a related note, the whole boot-time benchmarking obsession permeating the geek blogs lately obviously doesn’t resonate with me – if I reboot my laptop once a month at most, it may take five minutes for all I care. I’m much more interested in the kernel mode-setting advances, with its promises of more robust resume/suspend. Getting all excited here :))