Machine Cycle

Launch Playback of Multimedia Files on a SAMBA Share from Nautilus

2012-12-07T04:59:00.000-08:00

I hit this problem on a new Debian installation: if I use Nautilus to navigate to a SAMBA share (on a Windows or Linux box) I can launch playback of multimedia files by double clicking them. So far so good. This brings up Totem (the default Gnome multimedia player), which is fine, but I'd rather use VLC (from the Debian Multimedia repository).

It's easy enough to convince Nautilus to launch VLC, but VLC fails to find the multimedia files. This only happens if the SAMBA share is mounted by Nautilus. If I mount the SAMBA share manually from the command line, VLC finds the files just fine.

The problem is that Nautilus points VLC to a non-existent path that lives on the Gnome virtual file system (GVFS).

The fix is to install the GVFS FUSE daemon, which, for some reason, is not installed by default:

aptitude install gvfs-fuse

add your user account to the fuse group, and re-login.

Illegal Instruction in Code Compiled with Clang

2012-04-28T04:46:00.001-07:00

I was inspired by the effort to rebuild the Debian archive with clang, to try compiling UnDBX with clang:

CC=clang ./configure
make

I was a bit surprised when clang complained about an (legitimate) operator precedence issue, which gcc never pointed out to me. I fixed this by judicious use of parenthesis, and recompiled.

But the generated binary failed to run - I got an "Illegal instruction" error. This error persisted with optimizations turned off (i.e. -O0). I tried running the binary under valgrind and got the following:

vex x86->IR: unhandled instruction bytes: 0x66 0xF 0xEF 0xC0
==28891== valgrind: Unrecognised instruction at address 0x8048fc3.

followed by a verbose error message explaining that this may be caused either by trying to execute code in a non-code area of memory, or by a bug in valgrind.

I used objdump to find the offending instruction/address:

$ objdump -d -S undbx | grep -i 8048fc3
 8048fc3:       66 0f ef c0             pxor   %xmm0,%xmm0

I don't grok x86 assembly much, so it took me some googling to find that the combination of pxor and xmm type registers is in fact an SSE instruction. The CPU on this box is an aging 32-bit Mobile AMD Athlon XP, and the CPU flags in /proc/cpuinfo include SSE, so I'm not sure what to make of this.

The same code compiles and runs nicely on my newly upgraded laptop which sports a slightly less lame 64-bit Mobile AMD Sempron processor, running a 64-bit kernel.

In any case, I tried compiling like this:

CC='clang -mno-sse' ./configure
make

and I got a working executable.

My guess is that this is a bug in clang (maybe Debian bug #632472?), but as I said, I'm not sure.

Remote GDM3 Login with VNC

2012-04-21T13:18:00.000-07:00

I usually leave my computer at home turned on, logged in to the graphical desktop, with a VNC server running, to allow remote access to the desktop with a VNC viewer.

A few days ago, while waiting for code to compile, I logged in to my computer from work, and performed a long overdue package upgrade, which required a reboot to complete. This left me logged out of the desktop, with the GDM3 greeter waiting for someone to log in, but no VNC server to allow remote access to it.

Here's how to run a VNC server and start the desktop, in this situation:

connect to the remote machine with ssh, forwarding the default VNC port (this is the usual way I access my home PC):
```
# ssh -L 5901:localhost:5900 example.home-computer.com
```

run the following command as root to detect the path to the X authority file:

$ ps ax | grep auth
 1398 tty7     Ss+    4:30 /usr/bin/Xorg :0 -br -verbose -novtswitch -auth /var/run/gdm3/auth-for-Debian-gdm-e7ERxa/database -nolisten tcp vt7
 5053 pts/5    S+     0:00 grep --color=auto auth

launch a VNC server (as root) that allows access to your real X11 display, using the path to the X authority file (I use x11vnc):
```
$ x11vnc -xkb -ncache 0 --forever -localhost -display :0 -auth /var/run/gdm3/auth-for-Debian-gdm-e7ERxa/database
```
back on the local machine - connect to the remote machine with a VNC viewer (I use realvnc here):
```
# vncviewer -FullScreen -LowColourLevel 2 -PreferredEncoding ZRLE localhost:1
```
you should now see the remote GDM greeter and be able to login to your desktop.

Update rkhunter Database After Installing/Upgrading Packages

2012-04-13T06:20:00.000-07:00

Once upon a time I described how to automatically update the rkhunter database after installing/upgrading packages. There's an easier way of achieving this which either did not exist at the time, or I somehow missed. Just add the following line to /etc/default/rkhunter:

APT_AUTOGEN="yes"

My original method is still useful for running any custom script after installing/upgrading packages.

DVD Backup

2012-04-06T10:27:00.000-07:00

Here's how I currently backup my kids' DVDs to single CD-sized AVI files (mostly tested with PAL animation DVDs):

find the main title (usually the longest):
```
$ lsdvd
...
Title: 21, Length: 01:31:58.200 Chapters: 42, Cells: 43, Audio streams: 05, Subpictures: 15
...
```
(lsdvd is also kind enough to report the number of the longest title)
Alternatively, you can use mplayer to do this too:
```
mplayer -msglevel identify=6 -frames 0 dvd://
```
(it also lists the language audio tracks on the disc)

extract the MPEG2 stream with mplayer

mplayer -dumpstream -dumpfile temp/title.mpg dvd://21

calculate encoded video bit-rate with the script that's listed below:
```
$ calc_video_bitrate.py 1:31:58 128 700
bitrate: 929.650763
```
(the command line arguments are, in order: duration of video, audio target bitrate in kbit/sec and encoded file size in MB)

rip the MPEG2 stream into an AVI file, in two passes, using ffmpeg - note that I use the -map command line option to keep only a single audio track (my kids don't speak English, yet):

ffmpeg -i temp/title.mpg -pass 1 -ac 2 -ab 128k -ar 44100 -acodec libmp3lame -b 929.65k -s 640x360 -map 0:0 -map 0:3  temp/title.pass1.avi
ffmpeg -i temp/title.mpg -pass 2 -ac 2 -ab 128k -ar 44100 -acodec libmp3lame -b 929.65k -s 640x360 -map 0:0 -map 0:3  temp/title.avi

The code for calculating video bit-rate is based on JavaScript code found at http://quadpoint.org/projects/simplerip.

#! /usr/bin/env python

import sys

def bitrate2bytes(bitrate, length):
    # bitrate in kbit/s
    # length in seconds
    # returns number of bytes
    return (bitrate * length * 1000) / 8.0

def bytes2bitrate(size, length):
    # size in bytes
    # length in seconds
    # returns bitrate in kbit/s
    return size * 8.0 / length / 1000

def calcbr(targetsize, audiosize, length, coverhead):
    # targetsize in bytes
    # audiosize in bytes
    # length in seconds
    # coverhead in percent (container overhead)
    # returns bitrate in kbit/s
    overhead = 1.0 + (coverhead / 100.0)
    size = (targetsize - audiosize) / overhead
    return bytes2bitrate(size, length)

def main():
    if len(sys.argv) != 4:
        print >>sys.stderr, 'Usage: %s hh:mm[:ss] <audio-bitrate-kb-per-sec> <target-size-mb>\n' % sys.argv[0]
        return -1
    
    tokens = sys.argv[1].split(':')
    length = (float(tokens[0]) * 3600 + 
              float(tokens[1]) * 60 +
              (float(tokens[2]) if len(tokens) > 2 else 0))
    audiobr = float(sys.argv[2])
    audiosize = bitrate2bytes(audiobr, length)
    targetsize = float(sys.argv[3])
    tsize = targetsize * 1024 * 1024
    overhead = 0.7
    bitrate = calcbr(tsize, audiosize, length, overhead)
    print 'bitrate: %f\n' % bitrate
    return 0

if __name__ == '__main__':
    sys.exit(main())

DVD Playback: The Missing Package

2012-03-30T05:17:00.000-07:00

This took a while to figure out: I had all sorts of weird errors while trying to play a DVD on my laptop, until I did this:

aptitude install libdvdcss2

(requires the Debian Multimedia package repository).

Digging Tunnels: socat

2012-03-23T04:00:00.000-07:00

When digging tunnels, an alternative for stunnel on the client side is socat. Here's how to use it in ~/.ssh/config:

Host tunnel.example.com
  Port 443
  ProxyCommand socat - OPENSSL:%h:%p,verify=0

One Liner: Increase the Size of /tmp

2012-03-16T14:59:00.000-07:00

The temporary space on /tmp may not be enough for some programs. The following command will re-size it to 2GB, without destroying any existing files:

mount -t tmpfs tmpfs /tmp -o size=2000M,mode=1777,remount

(source: http://wiki.linuxquestions.org/wiki/Tmpfs).

If you find yourself doing this often then you have at least two options:

either add a suitable line for /tmp in /etc/fstab:
```
none /tmp tmpfs size=2000M,mode=1777 0 0
```
or avoid using tmpfs entirely, by modifying /etc/default/rcS like this:
```
RAMTMP=no
```
(not recommended: may impact both security and performance)

the changes will take effect after you reboot.

[02 Sep 2012] UPDATE: just noticed that RAMTMP=no is now the default setting, again, and that it is set in /etc/deafult/tmpfs

Install Quake and Quake3 on Debian Linux

2012-03-09T03:10:00.000-08:00

A long time ago, in a galaxy far away, I was an avid Quake, and later on Quake2 and Quake3 player and modder.

I seem to be getting old: I find myself reminiscing about all sorts of stuff, and Quake is one of my favorite subjects. In one of these nostalgia attacks I decided to relive the fun and install the games on my Linux laptop. I still have all the relevant CD-ROMs, so all that's needed is a suitable game engine. Quake and Quake3 are packaged in Debian, but Quake2 is not packaged.

Here's how to install Quake, with both mission packs:

install the game engine:
```
aptitude install quake
```
mount the game CD-ROM (e.g. on /media/cdrom0)
change directory to a temporary directory

package the game data:

game-data-packager quake -m /media/cdrom0

repeat the process for both mission packs, i.e. mount the relevant CD-ROM and then issue these commands:
```
game-data-packager quake -mp1 /media/cdrom0
```
for the SOA mission pack, and
```
game-data-packager quake -mp2 /media/cdrom0
```
for the DOE mission pack

install the generated packages (as root):

dpkg -i quake-registered_29_all.deb
dpkg -i quake-armagon_29_all.deb
dpkg -i quake-dissolution_29_all.deb

And here's how to install Quake3:

install the game engine:
```
aptitude install quake3
```

download the latest point release:

cd ~/Downloads
wget -c ftp://ftp.idsoftware.com/idstuff/quake3/linux/linuxq3apoint-1.32b-3.x86.run

mount the game CD-ROM (e.g. on /media/cdrom0)
change directory to a temporary directory

package the game data:

game-data-packager quake3 /media/cdrom0 ~/Downloads/linuxq3apoint-1.32b-3.x86.run

install the generated package (as root):
```
dpkg -i quake3-data_29_all.deb
```

Happy fragging!

Scripting a Simple Boot Time State Machine in GRUB2

2012-03-02T04:50:00.000-08:00

There's a hardware compatibility issue between my laptop's on-board nVidia nForce4 antiquated and buggy SATA interface and the modern Western Digital hard disk that I've installed in it.

The hard disk seems to disappear in mid-boot - this can happen anytime it is being re-detected: when BIOS transfers control to the boot loader (either GRUB2 or the Windows Boot Loader), and when the initial kernel image (loaded by the boot loader) re-initializes the SATA controller. If the system comes up, then any power management event may cause the disk to disappear. Sometimes, the disk just disappears for no obvious reason. The kernel's libata error handling routines may manage to reestablish connection to the disk, but sometimes it does not come up and I have to power cycle the laptop.

None of the solutions/workarounds I tried did the trick: apart for the stuff I already mentioned, I added rootdelay=10 to the kernel command line, hoping that the extra delay would give libata a better chance of re-detecting the disk; I even installed the watchdog daemon, to get my laptop to reboot when it freezes, but either it doesn't really work, or (most likely) I don't understand how to configure it.

The last ditch workaround I came up with, was to boot the laptop into GRUB2 from a USB disk-on-key, thus avoiding the internal hard disk altogether, and have GRUB2 perform the following steps:

check if boot related files on the internal hard disk are accessible
if any file is not accessible then reboot
if all files are accessible, then if this is the first success then reboot
if this is the second success then boot GRUB2 on the internal hard disk

This procedure captures the sequence of steps that seems to get my box to boot - it's Voodoo, I know, but it does work (most of the time).

The script (boot/grub/grub.cfg) below implements this procedure. It uses get_env and save_env to keep a persistent state (the variable need_reboot) between reboots. The first few lines make sure that the state is properly initialized, even if the environment variables were not saved yet (i.e. if this is the first ever boot from this device).

The fact that one can control the boot process this way is pretty neat, in my opinion. The only trouble is that even with this, my system still fails to boot - albeit less frequently than before.

I must admit defeat. The laptop is usable now, but, in my eyes, just barely. The whole experience made me itch for a new PC, but with my luck being what it is, I fear that it won't be a much easier ride. So, for now, I've resigned myself to my fate, until I gather up the courage (and cash) to tackle a new box.

set need_reboot="yes"
if [ -s $prefix/grubenv ]; then
  load_env
fi

insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
set reboot_delay=5

insmod ntfs
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus

function wait_and_reboot {
  echo "Reboot in ${reboot_delay} seconds ..."
  if sleep --interruptible ${reboot_delay} ; then
    reboot
  fi
}

function find_or_die {
  if [ -e $1 ]; then
    echo "$1 found"
  else
    echo "$1 NOT found!"
    wait_and_reboot
  fi
}

set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue

find_or_die (hd1,msdos1)/Boot/BCD
find_or_die (hd1,msdos2)/config.sys
find_or_die (hd1,msdos5)/boot/grub/grub.cfg

if [ -z "${need_reboot}" ]; then
     set need_reboot="yes"
     save_env need_reboot
  else
     set need_reboot=
     save_env need_reboot
     wait_and_reboot
  fi
fi

set timeout=30
menuentry "GRUB on internal hard disk" {
        set root='(hd1)'
        drivemap -s (hd0) (hd1)
        chainloader +1
}

Forcing SATA Speed Limit to 1.5Gbps

2012-02-24T07:19:00.000-08:00

Recap: I've upgraded the hard disk on my laptop, installed Win7 Ultimate on it from upgrade media, installed Debian GNU/Linux to a another partition on the same disk, and configured the Windows Boot Loader for dual boot. I was very pleased. It all seemed to work well, except for a small power management issue.

I got busy installing all the stuff I needed on both operating systems, setting up printers and other hardware, configuring mount points, network shares, backup, ssh, etc. All in all, this took a few days, during which the new disk withstood a lot of read/write operations (gigabytes at a time). Most of this work was rather boring.

And then, following one of many reboots, the Windows Boot Loader failed to start:

Windows failed to start. A recent hardware or software change might be the cause. To fix the problem:

    1. Insert your Windows installation disc and restart your computer.
    2. Choose your language settings, then click "Next."
    3. Click "Repair your computer."

If you do not have this disc, contact your system administrator or computer manufacturer for assistance.

    File: \Boot\BCD
    Status: 0xc000000e
    Info: An error occurred while attempting to read the boot configuration data.


ENTER=Continue

My first reaction was panic. My next reaction was to power cycle the laptop. It came up just fine: Windows seemed to work OK, and another reboot confirmed that the Linux partition was alive too.

I wasn't pleased anymore.

I searched for that error code, and found a lot of complaints, and a lot of "solutions" - but in most cases the error was permanent, and a reboot did not make it go away. This was a bad sign, but I had very little to work on, so I let it go.

Two days later Debian suddenly hard-locked on me. After power-cycling the box I was not able to find any error message in any of the log files.

And a few days after that, Debian failed to boot, dropping me into a limited shell, claiming that the boot device was missing.

After yet another Windows Boot Loader failure, I used grub-install to replace it with GRUB2, in the slim hope that it would fix the problem. Even if this wouldn't fix the problem, at least I'd be able to inspect the system when it fails, from within the GRUB2 shell.

GRUB2 did not fix anything: the laptop would still occasionally fail to boot - either GRUB2 would fail to find its own files, or the kernel would start but later on fail to find the root device.

During all that time, when my laptop was up and running, it would occasionally freeze for brief, but noticeable, periods of time. It took me a while to realize that this was not a case of a slow-to-load website, or icedove just taking its sweet time starting up - these hiccups were correlated with messages like the following being logged to /var/log/kern.log:

ata3: ATA_REG 0x40 ERR_REG 0x0
ata3: tag : dhfis dmafis sdbfis sactive 
ata3: tag 0x0: 1 1 0 1  
ata3.00: exception Emask 0x0 SAct 0x1ff SErr 0x0 action 0x6 frozen
ata3.00: failed command: WRITE FPDMA QUEUED
ata3.00: cmd 61/08:00:00:f8:4d/00:00:0c:00:00/40 tag 0 ncq 4096 out
         res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
ata3.00: status: { DRDY }
ata3.00: failed command: WRITE FPDMA QUEUED
ata3.00: cmd 61/08:08:18:f8:4d/00:00:0c:00:00/40 tag 1 ncq 4096 out
         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

...

ata3.00: status: { DRDY }
ata3: hard resetting link
ata3: nv: skipping hardreset on occupied port
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: configured for UDMA/133
ata3.00: device reported invalid CHS sector 0
ata3.00: device reported invalid CHS sector 0
ata3.00: device reported invalid CHS sector 0
ata3.00: device reported invalid CHS sector 0 
ata3.00: device reported invalid CHS sector 0 
ata3.00: device reported invalid CHS sector 0
ata3.00: device reported invalid CHS sector 0
ata3.00: device reported invalid CHS sector 0
ata3.00: device reported invalid CHS sector 0
ata3: EH complete

At first glance it seemed to me that the hard disk was failing. I subjected it to a slew of tests (fsck, smartmontools, MHDD), and in all of them the disk was found to be in excellent shape.

The next suspect in line was the on-board SATA controller. The laptop's mobo is based on the nVidia's nForce4 430 (MCP51) chipset. Wikipedia has this to say about it:

There have also been data corruption issues associated with certain SATA 3 Gbit/s hard drives.

The suggestion provided by many online, is to somehow force SATA speed to 1.5Gbps. However, as far as I can tell, there's no BIOS settings on this laptop to do that, and the disk itself cannot be forced either (no jumpers).

After some more research I found that the SATA speed can be forced on the kernel command-line:

modify /etc/default/grub:

GRUB_CMDLINE_LINUX="libata.force=1.5Gbps"

run update-grub as root
reboot

This cannot solve early boot problems, but I was hoping that it would at least prevent the lockups and hiccups.

Unfortunately, this did not work as expected - here's what I get in the logs:

ata3: FORCE: PHY spd limit set to 1.5Gbps
ata3: SATA max UDMA/133 cmd 0x30c0 ctl 0x30b4 bmdma 0x3090 irq 23

...

ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

The kernel reports its intention to force the SATA controller to 1.5Gbps, but the link comes up at 3.0Gbps regardless. Sh*t.

I still had another trick up my sleeve, but I've run out of steam here, so you'll have to wait for my next post.

Hard Disk Problems: Disabling Spindown

2012-02-17T02:00:00.000-08:00

So I managed to upgrade my laptop's hard disk, and setup a dual-boot system with Windows 7 Ultimate and Debian/testing. It all seemed to be working just fine. I was pleased, and mildly surprised. It didn't last long, though.

The first sign of trouble was that my laptop would not come out of sleep mode ("suspend") - and this happened on both Windows and Linux. My theory, at the time, was that the antiquated SATA interface on my machine, and the new hard-disk's aggressive power management features, do not play well together. The common wisdom on the Net is that one has to prevent the OS from spinning down the hard disk when it goes to sleep.

Doing that on Windows 7 is left as an exercise to the reader. Linux, on the other hand, earns its reputation fair and square as a time-waster. You should add the following incantation to /etc/hdparm.conf:

/dev/sda {
        apm = 254
        spindown_time = 0
}

(where /dev/sda is the block device pointing to the hard-disk in question), and then restart hdparm (as root):

invoke-rc.d hdparm restart

The only trouble with this, is that it only works for the first time the system goes to sleep, and on the second time the system hangs again. This is caused by a long standing wishlist-bug in hdparm (see Ubuntu bug #199094 and Debian bug #510676), with a published fix, that has not been applied for some reason. Basically, most hdparm settings are lost upon resuming from suspend.

The fix is to create the a script /etc/pm/sleep.d/20hdparm, with the following contents:

#!/bin/sh
# This script reinitializes the hard disk settings on resume.                                                                              

case $1 in
  resume|thaw)
    /usr/sbin/invoke-rc.d hdparm start >/dev/null
    ;;
  suspend|hibernate)
    # Not needed                                                                                                                           
    ;;
esac

But even with this, I ended up completely disabling sleep/hibernation because I couldn't get the box to come out of hibernation, and I had some more hard disk trouble.

Stay tuned...

Dual Boot Windows 7 and Debian GNU/Linux

2012-02-10T14:09:00.000-08:00

Previously on Machine-Cycle: your thrill-seeking host has decided to upgrade his laptop and setup a dual boot system, and has managed to install a vanilla Windows 7 Ultimate on a new clean hard disk, from upgrade media.

Onward and forward: time to install Debian and make this box useful.

I researched this step a bit and decided to use Windows' own boot manager to manage the selection of operating systems at boot time, so as to minimize the risk of somehow trashing the Windows 7 installation. I did, eventually, replace Windows boot manager with GRUB2, and it worked out just fine. I switched to GRUB2 because it allowed me to debug some hardware issues that cropped up later, but I'll leave that for a future post.

download the Debian/testing netinst ISO image and burn it to a CD
boot Debian Installer from CD
select graphical install
select manual partitioning: you should now see the list of existing disk partitions - the first, smaller, NTFS partition belongs to the Windows 7 boot manager, and should not be touched; the second, large, NTFS partition is the one we need to shrink, in order to fit in Debian
resize the second NTFS partition listed to, say, 33 percent of the hard-drive's total size (this will take a while)
select the remaining empty partition and install Debian into it (this will take a while)
when prompted to install GRUB: you can let the Debian Installer perform its magic for you, and it should just work; otherwise, in order to keep the Windows boot manager: DO NOT install GRUB on the first partition (named, most likely, /dev/sda1), install grub on the new partition (/dev/sda3)
complete the installation - note that you will not be able to boot into Debian just yet
Windows will run a disk check upon reboot, and it should then start normally
boot the system into a live CD/USB (Grml is a good choice) and copy the contents of the /dev/sda3 boot sector to a file on the second partition (mounted, for example, on /mnt/sda2):
```
dd if=/dev/sda3 of=/mnt/sda2/debian.bin bs=512 count=1
```
reboot into Windows
open Command Prompt as administrator
add a new entry to the boot manager's menu:
```
bcdedit /create /d "Debian GNU/Linux" /application BOOTSECTOR
```
this command returns the numeric ID of the new menu entry, which is used in the following steps
configure the new menu entry to boot C:\debian.bin, make it the last entry in the menu, and configure the menu to timeout after 30 seconds:
```
bcdedit /set {ID} device partition=C:
bcdedit /set {ID}  path \debian.bin
bcdedit /displayorder {ID} /addlast
bcdedit /timeout 30
```

bcdedit

this

you should now be able to select, upon reboot, to boot into either Windows 7 or Debian GNU/Linux

Going both ways ain't that simple.

Clean Install of Windows 7 with Upgrade Media

2012-02-03T08:56:00.000-08:00

A few weeks ago I made a decision. I decided to upgrade my crappy HP Pavilion dv6000 laptop. Yup, you're quite right - that was a mistake.

My plan was to replace the laptop's internal 60GB hard-drive with a shiny new 320GB Western Digital 3.5'' hard-drive, and then setup a dual boot system with Windows 7 and Debian GNU/Linux.

After some research, I concluded that the safest way to setup a dual boot system would be to first install a vanilla Windows 7 system, check that it works right, and only then complicate matters by installing Debian.

Now, about a year and half ago, my wife purchased MS Office 2010 through her workplace, at a considerable discount. It came bundled with a Windows 7 Ultimate upgrade installation media, which remained lying in some drawer ever since, gathering dust. I was really itching to use it.

But scratching that itch required me to solve two problems: a legal one (am I allowed to do this?) and a technical one: according to the installation instructions, and unlike previous versions, the Windows 7 upgrade installer seems to require a working OS to already be installed on the target system. The prospect of first installing Windows XP and then upgrading to Windows 7, did not appeal to me one bit. A quick search provided me with a way out of this - I'd still need to perform a "double-install", but of Windows 7 only:

when prompted by the Windows 7 installer, select a "custom" install - do not, at any time, enter any code, activate, or update the OS
reboot when instructed to do so
re-install from the same installation media, but this time "upgrade" the existing installation, and again, do not enter any code, activate, or update the OS
reboot when instructed to do so
activate the OS using the activation key provided with the installation media
update the OS

and it seems that it's perfectly legal, as long as you do own a full version of an older Microsoft OS.

To be continued...

Digging Tunnels: Connection Problem on Debian

2012-01-27T03:57:00.000-08:00

If you follow the instructions on my first "Digging Tunnels" post, to setup ssh tunneling over SSL with stunnel, then you may find that you can't access your Debian box from another Debian box. I got the following error:

SSL_accept: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

The fix (found on serverfault.com) is to make sure you have the following in lines in /etc/stunnel/stunnel.conf:

sslVersion = all
options = NO_SSLv2

Boot Floppy and ISO Images with GRUB2 on Debian

2012-01-20T07:09:00.000-08:00

I've once described my setup for booting floppy disk images. That setup works rather nicely, but there's a better way of doing this on Debian:

install grub-imageboot:
```
aptitude install grub-imageboot
```
put all image files you want to be able to boot in /boot/images
run (as root):
```
update-grub
```

and that's it!

grub-imageboot performs the same task as my original setup, in much the same way, but it also supports ISO images, is easier to install and much easier to uninstall.

A related tool, from the same authors (the fine Grml team) is grml-rescueboot, which lets you boot into a Grml rescue system from Grml ISO images placed in /boot/grml.

Stuttering Video Playback Due To USB Device Reset

2012-01-12T09:01:00.000-08:00

My kids don't usually complain when they're watching a movie on my PC, if video playback stutters. It seems that as long as playback resumes in a second or less, they don't mind the interruptions. I guess they've learned that if they do complain, chances are that they'll have to wait several long minutes, until their OCD inflicted dad "fixes" the problem.

I hate it when video playback stutters. So much so that I usually first download whatever video I want to watch and then watch it offline. Which is why I am a get-flash-videos fanboy.

But last time we were hit with such a problem, our lame ISP was not the culprit. This time, the video source was a backup ISO image of some DVD, stored on an external hard disk, connected to a Debian box (my old, headless laptop) over USB, accessed via SAMBA over a wireless connection, and played back on my WinXP laptop. The hiccups were rather noticeable, even by my lag-tolerant kids, to the point that they complained about it.

At first, it seemed that debugging this would be a daunting task - there are just so many factors involved. But I got lucky - I tried playing that same DVD image on my wife's laptop, over a wired network connection, and was hit by the same playback hiccups. This ruled out most of the suspects except for my old Debian laptop.

And yet, I had no idea what to look for now, especially because I can't just playback video on my old laptop - it's screen is busted... So I just browsed for any suspicious message in the output of dmesg - and found the following message, repeated several times:

"reset high speed USB device using ehci_hcd and address 3"

These messages seemed very relevant, and a quick search brought me to several blog posts, forum threads and bug reports.
I tried one the solutions/workarounds suggested on those links:

echo 128 > /sys/block/sdc/device/max_sectors

and it actually did the trick and made the hiccups go away.

The next challenge was to make this happen on system start-up. The Right Thing™ to do would have been to modify /etc/sysctl.conf, but there's always the chance that the relevant block device would be renamed, e.g. when another external disk drive is attached. So I ended up adding the following line line to /etc/init.d/bootmisc.sh:

echo 128 > /sys/block/$(readlink -f /dev/data | sed -e 's/1//g' -e 's/\/dev\///g')/device/max_sectors

where /dev/data is a symbolic link that points to this block device, which is generated with an appropriate UDEV rule.

UnDBX in Debian!

2011-10-18T13:44:00.000-07:00

UnDBX is now officially packaged in Debian!

Configuring hpodder To Handle Compressed Podcast Feeds

2011-07-29T13:28:00.000-07:00

My Nokia 2730 classic dumbphone is surprisingly smart. Unbeknownst to me at the time of purchase, was the fact that it has a 2GB MicroSD card, and can play MP3 files. But eventually I did stumble upon this feature and it wasn't too long before I started tuning in on podcasts.

I use hpodder (launched from a cron job) to follow podcast feeds and download episodes to my computer, and a semi-scripted procedure to move these files to my phone's memory card over a USB connection.

Lately, hpodder started complaining:

*** 12: Failure parsing feed: Lexical error in  file http://escapepod.org/feed/  at line 1 col 1:
  unrecognised token: ^_\213^H^@

So I tried to download the feed manually:

$ curl http://escapepod.org/feed/

and the terminal filled up with gibberish to the point that I had to blindly type reset in order to fix it.

That was weird: after all, the feed is nothing more than an XML file - a text file, which should be perfectly readable with the naked eye.

I saved the feed with

$ curl -o feed.bin http://escapepod.org/feed/

and then determined its type:

$ file -b -i feed.bin
application/x-gzip; charset=binary
$ zcat feed.bin | file -b -i - 
application/xml; charset=utf-8

I.e. a GZIP compressed XML file.

So hpodder choked on a compressed feed.

I consulted the manual and found out that hpodder delegates downloads to cURL. I skimmed through the cURL manpage, found about the --compressed command line option, tried the download again - and got the uncompressed XML contents. Hoozah!

Apparently, the server, that's serving that particular feed, is mis-configured to always compress its replies, even if not specifically asked to do it. The --compressed command line options tells cURL to request the server to compress its replies, and cURL decompresses them.

I tried downloading other feeds with the --compressed added, and it worked fine. So either this option is supported by all the other servers on my list of feeds, or that cURL does nothing when the reply is not compressed. I dunno.

All that I needed now was a way to convince hpodder to launch cURL the same way.

Turns out that hpodder is a rather nice piece of software (and rather well documented too). The hpodder manual pointed me to ~/.hpodder/curlrc:

$ echo compressed >> ~/.hpodder/curlrc

and now hpodder works like a charm again (and probably faster than before, because it always asks for compressed replies).

Cloning a GitHub GIT Repository on Ubuntu 8.04 LTS

2011-07-22T09:20:00.000-07:00

At work, we're still running Ubuntu 8.04 LTS on most PCs with Linux. Most of the time the age of the operating system isn't a problem - but sometimes it can be a pain. Case in point: cloning a GIT repository hosted on GitHub. This used to work just fine, until they switched to HTTPS:

$ git clone https://github.com/user/repo.git
Initialized empty Git repository in /current/directory/repo/.git/
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?

When this first happened, I shrugged it off as a problem with the remote end, and just downloaded a source tarball from https://github.com/user/repo/tarball/master. But a few weeks later I got this error again with a different repository, and got annoyed. I tried the same command at home (Debian/testing, GIT version 1.7.5.4):

$ git clone https://github.com/user/repo.git
Cloning into repo...
remote: Counting objects: 81, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 81 (delta 34), reused 55 (delta 8)
Unpacking objects: 100% (81/81), done.

So, this is a problem with either GIT at work, or the Net connection. I downloaded the GIT source tarball and installed it locally in my account (at ~/local):

wget -c http://kernel.org/pub/software/scm/git/git-1.7.6.tar.bz2
tar xvjf git-1.7.6.tar.bz2
cd git-1.7.6
ls
./configure --prefix=$HOME/local
make
make install

and since we're using tcsh at work (don't ask), I also had to type rehash in order to convince the shell to use the newly installed GIT.

Here's what I got this time:

$ git clone https://github.com/user/repo.git
Cloning into repo...
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://github.com/user/repo.git/info/refs

fatal: HTTP request failed

... which is useful: Google directed me to a question on Stack-Overflow. Most of the answers there deal with installing CA certificates, but the following trick works nicely with git version 1.5.4.3 on Ubuntu 8.04.4 LTS:

$ env GIT_SSL_NO_VERIFY=true git clone https://github.com/user/repo.git
Cloning into repo...
remote: Counting objects: 81, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 81 (delta 34), reused 55 (delta 8)
Unpacking objects: 100% (81/81), done.

(no need for env in bash).

And while we're at it, here's another trick that might be handy in the future:

$ env GIT_CURL_VERBOSE=1 git clone https://github.com/user/repo.git
Cloning into repo...
* Couldn't find host github.com in the .netrc file, using defaults
* About to connect() to github.com port 443 (#0)
*   Trying 207.97.227.239... * Connected to github.com (207.97.227.239) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Expire cleared
* Connection #0 to host github.com left intact
* Couldn't find host github.com in the .netrc file, using defaults
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to github.com port 443 (#0)
*   Trying 207.97.227.239... * Connected to github.com (207.97.227.239) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Expire cleared
* Connection #0 to host github.com left intact
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://github.com/user/repo.git/info/refs

fatal: HTTP request failed

Backup a Blogger Blog - Revisited

2011-04-29T09:36:00.000-07:00

I recently found out that the method I used to backup this blog automatically, has stopped working. It all hinged on the observation that one can retrieve a single web page with the full text of all the posts of a given Blogger blog, by retrieving the link

http://blogname.blogspot.com/search?max-results=N

with a large enough N, e.g. 10000. This is not true anymore - at the moment I can only retrieve the latest 42 posts on this blog, from a total of 178.

I'm not sure when this method stopped working, or, for that matter, if it ever really worked. The fact remains, however, that I have a blog that I want to backup, so I spent the better part of an evening figuring out how to properly do this.

My new Blogger blog backup script, shown below, makes use of the Google Data services API to export and download the blog archive in XML format, and then extracts from it the links of all the posts, and mirrors these pages locally, with HTTrack:

#! /bin/bash

BLOGGER_EMAIL=user@gmail.com
BLOGGER_PASSWD=password
BLOGGER_BLOGID=000000000000000000
BLOGGER_BLOG=blogname

DEST_DIR=/path/to/backup/directory/
mkdir -p ${DEST_DIR}
cd ${DEST_DIR}

eval $( \
    curl -s "https://www.google.com/accounts/ClientLogin" \
    --data-urlencode Email=$BLOGGER_EMAIL --data-urlencode Passwd=$BLOGGER_PASSWD \
    -d accountType=GOOGLE \
    -d source=MachineCycle-cURL-BlogBackup \
    -d service=blogger | grep 'Auth='
)

curl -s "http://www.blogger.com/feeds/$BLOGGER_BLOGID/archive" \
    --header "Authorization: GoogleLogin auth=$Auth" \
    --header "GData-Version: 2" \
    | xml_pp > ${BLOGGER_BLOG}.blogspot.com.archive.xml

grep -o -e '<link href="http://'$BLOGGER_BLOG'.blogspot.com/..../[^\.]*.html" rel="alternate" title=' \
    ${BLOGGER_BLOG}.blogspot.com.archive.xml | \
    sed -e 's@.link href="@@g' -e 's@" rel="alternate" title=@@g' | \
    sort -ur > ${BLOGGER_BLOG}.links

mkdir -p ${BLOGGER_BLOG}
cd ${BLOGGER_BLOG}
httrack \
    -%v0 \
    -%e0 \
    -X0 \
    --verbose \
    --update \
    -%L ../${BLOGGER_BLOG}.links \
    -"http://${BLOGGER_BLOG}.blogspot.com/" \
    -"${BLOGGER_BLOG}.blogspot.com/*widgetType=BlogArchive*" \
    -"${BLOGGER_BLOG}.blogspot.com/search*" \
    -"${BLOGGER_BLOG}.blogspot.com/*_archive.html*" \
    -"${BLOGGER_BLOG}.blogspot.com/feeds/*" \
    -"${BLOGGER_BLOG}.blogspot.com/*.html?showComment=*" \
    +"*.gif" \
    +"*.jpg" \
    +"*.png"

A few comments are in order:

the script contains the Blogger username and password - keep it safe!
the blog id is the number that appears in the URL of most links accessible from the Blogger dashboard, after the blogID= part
the XML blog archive may later be used to restore/migrate the blog
local mirroring isn't really necessary - I just like it that I can view the blog contents offline
another unnecessary step: I use xml_pp to beautify the exported XML file
currently, the script performs no error checking - I may add some checks if and when I observe failures
sources: Using cURL to interact with Google Data services, Blogger export format

Fixing Normalize Audio Feature in K3b

2011-04-15T02:34:00.000-07:00

I usually burn and copy optical media with K3b. I don't do this often, but when I do, it usually just works. Except when it doesn't.

My wife asked me to create an audio CD for her, from an assortment of audio tracks she collected from various sources. It was easy enough to accomplish this with K3b. But the resulting audio CD was annoying to listen to, because I had to change the volume setting for each track.

It's a classic noob's mistake: I should've enabled audio normalization.

So I tried it again, but found that I couldn't enable audio normalization in K3b. Turns out that K3b uses an external application (called - surprise! - normalize-audio) to perform this task, and K3b just couldn't find it - a fact that was clearly (?) stated in the programs section of the K3b settings dialog.

I tried launching normalize-audio at the command line, and it seemed to be installed alright. A quick Net search brought me to Debian bug #597155 and Ubuntu bug #45026. The root cause of the problem is that normalize-audio reports its version number as

normalize 0.7.7

while K3b expects

normalize-audio 0.7.7

This can be fixed either in K3b or in normalize-audio, and patches for both sides have already been posted. But neither has been incorporated yet. In the meantime, I've implemented a workaround, based on suggestions in those bug reports:

create (as root) an executable script named normalize-audio under /usr/local/bin/ with the following contents:

#!/bin/bash
case "$1" in    
    --version)
        /usr/bin/normalize-audio --version | sed -e 's/normalize/normalize-audio/g'
        ;;
    *)
        /usr/bin/normalize-audio "$@"
        ;;
esac

make this script executable:

chmod a+x /usr/local/bin/normalize-audio

this script is supposed to be used as a wrapper for normalize-audio, so make sure that /usr/local/bin/ appears in the PATH environment variable, and that it comes before /usr/bin
launch K3b from a new command shell - it should now detect normalize-audio and allow you to use it

Time to Burn!
[05 Nov 2011] UPDATE: fixed script to work with file names/paths that contain spaces (Thanks anonymous commenter!)

Setting Default Laptop Screen Brightness when Running on Battery Power

2011-03-18T07:48:00.000-07:00

A few weeks ago the screen of my "new" laptop (read: my wife's old WinXP laptop) started flickering.

It was almost imperceptible at first, but it got worse over time, to the point that the screen would, at random intervals, suddenly dim gradually and then brighten back, for a second or so.

I suspected that the laptop screen was failing. The prospect of having another headless laptop made me anxious. But after a while I noticed that whenever the screen brightness flickered, a battery icon would show up momentarily in the system tray.

So I came up with a theory: it may be some kind of a power supply problem that causes the laptop, for very brief periods of time, to think that it is disconnected from the mains power, causing it to switch to battery power, which, in turn, causes the OS to lower the screen brightness in order to reduce power consumption.

This theory seemed plausible, but with the warranty long-since expired, and without spare parts (power cord, power supply, battery, motherboard, etc.), my only option was to let it go. After all, other than this issue, the laptop seemed to be as functional as it can be.

The flickering, however, made me crazy, so I searched for a way to prevent Windows from automatically darkening the screen when the laptop is switched to battery power.

Well, it's easy, but it ain't obvious:

disconnect the laptop from the mains power - the screen dims
use the screen brightness controls to brighten the display back to the same brightness level when the mains power is connected
reconnect the mains power

That's it - I kid you not.

A few days ago the laptop switched to battery power and stayed there. I took a little chance and purchased a replacement AC adapter over at eBay. Surprisingly enough, that was it. Happy Happy Joy Joy.

Getting Rid of rkhunter's False Warning about Xzibit Rootkit

2011-03-04T00:52:00.000-08:00

I've installed rkhunter a long while ago, mostly because it seemed irresponsible not to install some sort of "protection". But, as is the case with any such tool, I started getting warnings, which, after I got over the induced anxiety attacks, were invariably confirmed as false positives.

It was usually rather simple to silence these warnings from the rkhunter configuration file /etc/rkhunter.conf - most of the time it was just a matter of un-commenting one or more lines, and occasionally updating rkhunter:

rkhunter --propupd

(say, for instance, after upgrading packages).

One false positive that was somewhat more complicated to disable was a warning about the Xzibit Rootkit. This warning is triggered by files containing the string hdparm - it's a known bug (see Debian bug #576680), and the workaround is to "use the RTKT_FILE_WHITELIST option to whitelist initscripts stating this string" - e.g. /etc/init.d/hdparm ...

The comments in the configuration file, suggest that the proper method of whitelisting a file is to also add it to USER_FILEPROP_FILES_DIRS and then update rkhunter. But this makes rkhunter complain that /etc/init.d/hdparm is an executable script, so I had to also add it to SCRIPTWHITELIST.

Bottom line - add the following lines to /etc/rkhunter.conf:

USER_FILEPROP_FILES_DIRS="/etc/init.d/hdparm /etc/init.d/.depend.boot"
SCRIPTWHITELIST=/etc/init.d/hdparm
RTKT_FILE_WHITELIST="/etc/init.d/hdparm /etc/init.d/.depend.boot"

and the run

rkhunter --propupd

Verify by running:

rkhunter --check

I can only hope that I won't hit any false negatives...

One Liner: Pretty Print, Syntax Highlight and Page Online XML Data

2011-02-25T12:18:00.000-08:00

Useful when leaching a bunch of video clips from YouTube:

curl -s "http://gdata.youtube.com/feeds/api/playlists/916175AB005C0FB6" | xml_pp | pygmentize -l xml | less -R

(required packages: curl, xml-twig-tools, python-pygments)

Actually, xml-pp isn't really needed with YouTube - it can do the pretty-printing for you:

curl -s "http://gdata.youtube.com/feeds/api/playlists/916175AB005C0FB6?prettyprint=true" | pygmentize -l xml | less -R