Mark Feeney

Samba: permission denied on nested directory creation

2010-01-20T14:19:00.003-05:00

I have a network setup where there is a Unbuntu (9.04, Jaunty) Samba/CIFS file server with Ubuntu and Windows clients. I had this annoying issue where, from Linux clients, recursive directory creation -- like mkdir -p /share/a/b/c would not work. In order to get /share/a/b/c you'd have to create each directory in turn: mkdir /share/a && mkdir /share/a/b && mkdir /share/a/b/c. This prevents handy commands like mkdir -p and rsync from working. Fortuantely, I finally found a work-around: turn off Samba's Unix extensions on the server. i.e.

[global]
...
unix extensions = no
...

Obviously this only helps if you don't need the Unix extensions (symlinks, hardlinks, etc on the share), but I didn't. This sounds like a bug in the CIFS client for Linux, but who knows. I'm pleased to have any work-around.

References

hexBinary Encoding

2010-01-18T23:39:00.004-05:00

Recently, I hacked together a wrapper script for reporting job statuses to Hudson. The XML API in Hudson called for "hexBinary" encoded data. I hadn't heard of this before, and couldn't find much in the way of decent examples on Teh Interwebs. From the spec, it seems to be pretty simple: for each byte in your data, write out its two character hex value. So if your byte has decimal value 223, write out its hex string: "DF". (Aside: this seems like a silly encoding, at least space-wise: why not the ubiquitous base64?) I wanted a simple shell script, so the issue was how to do this encoding without pulling in a full-out scripting language. Fortunately, hexdump has format strings. Unfortunately, its docs aren't great.

Example of hexBinary encoding using hexdump:

echo "Hello world" | hexdump -v -e '1/1 "%02x"'
48656c6c6f20776f726c640a

So what the hell is that? -v means don't suppress any duplicate data in the output, and -e is the format string. hexdump's very particular about the formatting of the -e argument; so careful with the quotes. The 1/1 means for every 1 byte encountered in the input, apply the following formatting pattern 1 time. Despite this sounding like the default behaviour in the man page, the 1/1 is not optional. /1 also works, but the 1/1 is very very slightly more readable, IMO. The "%02x" is just a standard-issue printf-style format code.

References:

XMLSchema spec that defines hexBinary
How to hexBinary encoding in Ruby (Stack Overflow)
Good details on hexdump's format strings (This finally made it clear to me)
Another example of using hexdump's format strings

Hudson External Jobs: Wrapper Script

2010-01-18T14:11:00.023-05:00

Lately, I've been using Hudson for a variety of tasks. Hudson is billed primarily as a continuous integration server, but one general-purpose cool feature it has is the ability to monitor "external" jobs. What this means is you can have some arbitrary process report status to Hudson periodically.

My first thought was to get important cron jobs to report status -- beats the automated emails that I tend to ignore. If you happen to have a full Hudson install on the server running the cron job, Hudson provides a simple Java-based wrapper you can use. I didn't want to have to have various .jar files copied to every machine that needed to post status, so instead I opted to use Hudson's XML over HTTP interface. Both the Java-based and HTTP approaches are documented to some extent here.

I wanted to make it as easy as possible to integrate any old script with Hudson, so I came up with the wrapper below. (It seems to work for me, but use at your own risk; no guarantees!)

#!/bin/sh
# Wrapper for sending the results of an arbitrary script to Hudson for
# monitoring. 
#
# Usage: 
#   hudson_wrapper <hudson_url> <job> <script>
#
#   e.g. hudson_wrapper http://hudson.myco.com:8080 testjob /path/to/script.sh
#        hudson_wrapper http://hudson.myco.com:8080 testjob 'sleep 2 && ls -la'
#
# Requires:
#   - curl
#   - bc
#
# Runs <script>, capturing its stdout, stderr, and return code, then sends all
# that info to Hudson under a Hudson job named <job>.
if [ $# -lt 3 ]; then
    echo "Not enough args!"
    echo "Usage: $0 HUDSON_URL HUDSON_JOB_NAME SCRIPT"
    exit 1
fi

HUDSON_URL=$1; shift
JOB_NAME=$1; shift
SCRIPT="$@"

OUTFILE=$(mktemp -t hudson_wrapper.XXXXXXXX)
echo "Temp file is:     $OUTFILE" >> $OUTFILE
echo "Hudson job name:  $JOB_NAME" >> $OUTFILE
echo "Script being run: $SCRIPT" >> $OUTFILE
echo "" >> $OUTFILE

### Execute the given script, capturing the result and how long it takes.

START_TIME=$(date +%s.%N)
eval $SCRIPT >> $OUTFILE 2>&1
RESULT=$?
END_TIME=$(date +%s.%N)
ELAPSED_MS=$(echo "($END_TIME - $START_TIME) * 1000 / 1" | bc)
echo "Start time: $START_TIME" >> $OUTFILE
echo "End time:   $END_TIME" >> $OUTFILE
echo "Elapsed ms: $ELAPSED_MS" >> $OUTFILE

### Post the results of the command to Hudson.

# We build up our XML payload in a temp file -- this helps avoid 'argument list
# too long' issues.
CURLTEMP=$(mktemp -t hudson_wrapper_curl.XXXXXXXX)
echo "<run><log encoding=\"hexBinary\">$(hexdump -v -e '1/1 "%02x"' $OUTFILE)</log><result>${RESULT}</result><duration>${ELAPSED_MS}</duration></run>" > $CURLTEMP
curl -s -X POST -d @${CURLTEMP} ${HUDSON_URL}/job/${JOB_NAME}/postBuildResult

### Clean up our temp files and we're done.

rm $CURLTEMP
rm $OUTFILE

If you have, for example, a crontab entry that looks like this:

00 02 * * * myscript.sh

you can have it report status to Hudson under a job called "test_job" by changing your crontab to look like this instead:

00 02 * * * hudson_wrapper http://hudson.myco.com test_job myscript.sh

The job "test_job" must be created as an "external job" in Hudson ahead of time for this to work.

One thing of interest here is the "hexBinary" encoding in the XML that is sent to Hudson. There is precious little info out there about "hexBinary", so hopefully I got that part right. From the spec, it seems simple enough, and the script does work for all the inputs I've thrown at it so far. Update: I wrote a more detailed post on hexBinary.

Update 2010-01-29: Added -s to curl to avoid transfer stats showing up on stderr. Also improved the wrapper to be able to handle any size output from the wrapped command. Before you were at the mercy of ARG_MAX, getting Argument list too long errors if your script output too much stuff.

Jungle Disk 3 Linux: Automatically start on reboot

2010-01-16T15:38:00.003-05:00

I've been a long time user of Jungle Disk, and it's historically been pretty good software. Recently, version 3 was released, and among various features and fixes, they also removed the command line version for Linux for "Jungle Disk Desktop", which I use. I wanted to run Jungle Disk on a simple file server that didn't have X, but that isn't possible with version 3. The issue has been raised with Jungle Disk by some other people, and it sounds like there's a chance they will do something about it... maybe/eventually.

In the mean time, I opted to just install a full desktop version of Ubuntu 9.10 on my file server so I could use Jungle Disk 3's GUI. But that brought to light another problem: you can't auto-start Jungle Disk 3 after a reboot because it requires a valid X display... A user must actually log in and start it! Instead of that nonsense, I run vncserver after a reboot using @reboot in a crontab, and get vncserver to run junglediskdesktop via it's xstartup file. A hack, but it seems to work well enough.

Here's a basic overview of what's needed to make this work.

Install vnc4server and set it up for gnome-session, as described nicely here.
Add the following line to the crontab for the user who should run Jungle Disk:
```
@reboot   /usr/bin/vncserver :1
```

Make sure ~/.vnc/xstartup looks something like the following for the user that should run Jungle Disk:

#!/bin/sh

# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
gnome-session &
junglediskdesktop &

To test that this is working, reboot the box and then use a VNC viewer ("Remote Desktop Viewer" in Ubuntu) to connect to the server on port 5901.

Simple Samba/CIFS Configuration

2010-01-15T21:51:00.005-05:00

Any time I have to do something with Samba, I run into stupid configuration and permissions issues. I just set up a dead simple Samba config and am documenting it here for next time. Possibly someone else might get some use out of it too.

Some reading I based this on: creating a public share in Samba, and some Samba on Ubuntu docs.

Overview:

Ubuntu 9.10 Karmic as the smb server
Whatever Samba 3.x it comes with
Windows and Linux client machines
Anyone on 192.168.1.0/24 has access: not secure, but convenient
Machine called myhostname is the file server, at IP 192.168.1.4

Config:

On the file server, myhostname in this example, create a user smbuser to act as the 'guest' in Samba. Client machines that don't authenticate will act as this user:
```
sudo adduser smbuser
```
Then make sure /etc/passwd and /etc/group have lines something like this:
```
# /etc/passwd
smbuser:x:1001:1001:Samba user,,,:/home/smbuser:/usr/sbin/nologin

# /etc/group
smbuser:x:1001:
```

Edit /etc/samba/smb.conf:

[global]
netbios name = myhostname
workgroup = WORKGROUP
server string = File Server
security = user
map to guest = bad user
guest account = smbuser
create mask = 0644
directory mask = 0755
hosts allow = 192.168.1.0/24
hosts deny = 0.0.0.0/0
unix extensions = no  # unless you REALLY need them

# Simple share that anyone can read/write to
[photos]
path = /data/photos
browsable = yes
guest ok = yes
read only = no

Client linux machine's /etc/fstab (Make sure smbfs is installed: sudo apt-get install smbfs):

//192.168.1.4/data /data cifs username=smbuser,password=,uid=bob,gid=bob 0 0
Client Windows machine: just browse to \\192.168.1.4 or \\myhostname

Snapsort: feature #1

2010-01-06T14:32:00.000-05:00

First post for normal humans!

Yesterday we launched the first Snapsort feature: compare any two digital cameras (in a sane way). Here's an example comparison. This is a relatively simple feature, but we think it's interesting. (...And possibly even useful already?) Alex gives a better description here. This is only the beginning, but it's nice to have something "live".

s3cmd 0.9.9 on Ubuntu 9.04/Jaunty

2010-01-04T16:17:00.002-05:00

The version of s3cmd that comes with Ubuntu Jaunty is quite old (0.9.8; from late 2008). It seems that numerous issues have been fixed in 0.9.9, which is part of Karmic. Fortunately, there is a backport to Jaunty. Here's how to install:

Uninstall s3cmd if you've already installed it:
```
sudo apt-get remove s3cmd
```

Add the following lines to /etc/apt/sources.list:

# s3cmd backports
deb http://ppa.launchpad.net/loic-martin3/ppa/ubuntu jaunty main 
deb-src http://ppa.launchpad.net/loic-martin3/ppa/ubuntu jaunty main

Run these commands:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 78822001
sudo apt-get update
sudo apt-get install s3cmd

Check that you've got 0.9.9:
```
$ s3cmd --version
s3cmd version 0.9.9
```

Thanks to Loïc Martin, from whom these packages originate.

Ubuntu 9.10 and GeForce 7800 GT

2009-12-30T08:56:00.004-05:00

Note: This issue seems to affect Ubuntu 9.04 Jaunty, and Ubuntu 9.10 Karmic, in both 32-bit and 64-bit versions. I've seen some anecdotal evidence that it affects earlier releases too.

I recently set out to install Ubuntu 9.10 on my aging old home PC: an Athlon64 3800+, 2 GB RAM, decent (at the time) Asus board. Most importantly, this machine has an NVidia GeForce 7800 GT video card. To my surprise (having installed Ubuntu on numerous other machines of various age), I had a myriad of problems with the install. The symptoms were as follows:

Installer would freeze with corrupt looking graphics if I tried to change certain options
If I took all defaults in the installer, I could get the OS installed, but it would lock up (again with corrupt looking graphics on screen) when selecting my user name in gdm when X started up

Googling around at length revealed others with similar problems, and the common element was the GeForce 7800 GT video card. Related links:

Recent linuxquestions.org thread that I'm on
Three year old thread talking about similar-sounding issue in Edgy
Bug report that finally clued me in on how to "fix" it (post #20; Michael Curran)

I don't really know what the root cause is here, but using the above info, here's a work-around: (This assumes you got the OS installed somehow, and are now at the point where you want to login, but cannot due to X locking up.)

Get to a terminal either by recovery booting to the shell (netboot), or by hitting ctrl-alt-f1 at the graphical login screen.

Add the NVidia ppa to get access to their newest drivers:

sudo add-apt-repository ppa:nvidia-vdpau/ppa
sudo apt-get update

Remove any existing NVidia driver:
```
sudo apt-get purge nvidia-glx-*
```
Install the newest driver:
```
sudo apt-get install nvidia-glx-195
```
Configure the newly installed driver:
```
sudo nvidia-xconfig
```
Reboot and you should be good to go.

EC2, fabric, and "err: stdin: is not a tty"

2009-12-17T12:18:00.007-05:00

Recently, I've been using fabric to script up some deployments to EC2. My EC2 server is running the 32-bit Ubuntu Jaunty 9.04 server image provided by Alestic, but I suspect this issue applies across lots of different OS variants. Fabric seems to work well, but I was getting a nuisance message when running commands on my EC2 instance: err: stdin: is not a tty. You can even see the message in action in the output of this (very useful) intro to fabric that happens to show an EC2 example.

Some searching turned up that this is a result of a command in a .profile or .bashrc on the remove host expecting the shell to be interactive, which it is not with fabric.

In my case, the culprit was /root/.profile, in particular the command mesg n. To fix the message, I wrapped the mesg call in a check for an interactive tty:

if `tty -s`; then
    mesg n
fi

Scala Lift, Jetty 6, static content, and Virtual Directories

2009-12-14T17:33:00.014-05:00

I have a simple web application written using the Lift framework. I am using Maven to build and test with, in the way recommended by various Lift docs (i.e. I use mvn jetty:run to launch the app). I ran into an issue when it came to serving static image content, however.

The app is called webapp
The app deploys to /webapp
Images are on local disk at /hdd/data/images (i.e. not in the .war)
Want to access images from /images (i.e. outside of the context root of the Lift app)
Needs to work when developing and running with Maven's jetty:run

The main issue was simply "how do I setup a virtual directory from http://server/images to /hdd/data/images on disk?" This seems trivial, but I had a really difficult time finding any straightforward docs on how to do it. My complicating requirement was that it had to work when developing with jetty:run. For "real" deployment, the images location will be manually configured, so whether this makes sense in production is not a factor here.

Long story short: you can configure the maven-jetty-plugin to load up additional "context handlers" at startup. Some decent info on this here. A context handler of type WebAppContext will do the job as far as acting like a virtual directory. Here's the config from pom.xml for the project:

<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>maven-jetty-plugin</artifactId>
    <configuration>
        <contextPath>/webapp</contextPath>
        <scanIntervalSeconds>0</scanIntervalSeconds>
        <contextHandlers>
            <!-- setup a simple webapp to serve our images -->
            <contextHandler implementation="org.mortbay.jetty.webapp.WebAppContext">
                <contextPath>/images</contextPath>
                <resourceBase>/hdd/data/images</resourceBase>
            </contextHandler>
        </contextHandlers>
    </configuration>
</plugin>

I make no claims about this being secure or the best way to do things, but after a lot of trial and error it does at least work.

Python: lambda, partial, and scopes

2009-11-04T15:53:00.008-05:00

Python rarely surprises me, which is a good thing. Unfortunately, today I was surprised by how lambda works. It made me worry that I had no idea what I was doing at all. (And I guess I didn't!) A lambda captures a reference to any variables it closes over (vs. captuiring the value of those variables), so in a loop context you can get some bizarre effects.

It turns out this is well documented, but I'm writing this as a note to myself if nothing else.

def adder(x, y):
    return x + y

# make a list of functions with lambda that close over i and call adder()
fns = [lambda x: adder(x, i) for i in range(10)]

# call each of our new functions with a constant value
# expected output: [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]

[f(13) for f in fns]
# output: [22, 22, 22, 22, 22, 22, 22, 22, 22, 22]

# buh!?

The functions that are created in the list fns each capture a reference to the local variable i, but that reference is constantly being rebound to different values as we iterate through range(10). The last value i takes in range(10) is 9, so all of the functions end up being lambda x: adder(x, 9). Below are two "fixes".

Fix 1: capture the value of i as a default argument to the lambda. Hacky, but supposedly idiomatic for this sort of issue.

fns = [lambda x1, x2=i: adder(x1, x2) for i in range(10)]
[f(13) for f in fns]
# output: [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]

Fix 2: use functools.partial. I prefer this.

from functools import partial
fns = [partial(adder, i) for i in range(10)]
[f(13) for f in fns]
# output: [13, 14, 15, 16, 17, 18, 19, 20, 21, 22]

In retrospect, this really shouldn't be surprising. I think the main issue here is that in these examples i is just a plain old integer, which, due to my Java background, I tend to think of as a primitive Java int that can only be passed by value. In Python, everything is just references until interpretation time. e.g.

foo = 42
def test():
    print foo
test() # outputs: 42

foo = 100
test() # outputs: 100 (!)

Installing VirtualBox Guest Additions on Ubuntu 9.04/9.10 Guest, Vista Host

2009-10-19T22:57:00.009-04:00

Here are some notes on how I installed the VirtualBox 3.0.8 Guest Additions on a 32-bit Unbuntu Server (9.04 Jaunty Jackalope) guest running on a 32-bit Windows Vista host.

The Ubuntu server is a very bare-bones server install running only sshd, so as you'll see below, we have to install gcc and friends.

This info is adapted from a post in the VirtualBox forums.

Mount VirtualBoxGuestAdditons.iso via Devices > Mount CD/DVD ROM > CD/DVD ROM Image...
The .iso file should be located on the Windows host here: C:\PROGRA~1\Sun\VIRTUA~1\VBoxGuestAdditions.iso

Install the Guest Additions on the Ubuntu guest (adapted from http://forums.virtualbox.org/viewtopic.php?p=29316):

mount /cdrom; cd /cdrom
sudo apt-get update
sudo apt-get install build-essential linux-headers-`uname -r`
sudo ./VBoxLinuxAdditions-x86.run
cd ~; umount /cdrom

Optional cleanup (I didn't bother since I'm using this box as a development server, and will need gcc et al.)

sudo apt-get remove build-essential linux-headers-`uname -r`
sudo apt-get autoremove
sudo apt-get autoclean

reboot the guest (possibly not necessary, but I did it)

The guest additions are now installed and running. Next step would be to actually do something with them. In my case, I wanted to mount the c:\ drive of the host (Vista) machine.

To share the entire Vista c:\ in the directory /mnt/c on the Ubuntu guest:

On the host window, go to Devices > Shared folders...
Add new shared folder: Folder Path = c:\, Folder Name = c, not Read-only, check Make Permanent

Add the following line to /etc/fstab:

c    /mnt/c    vboxsf    rw,gid=1000,uid=1000,auto    0    0

Note: I've mounted this as my main user. If you've only created one user on your Ubuntu server, this should work. If in doubt, run id as the user you want to mount the Windows share as.
Create the mountpoint: sudo mkdir /mnt/c
Mount the share (this will happen automatically in the future): sudo mount /mnt/c

Update 28-Dec-2009: I just used this same procedure on Karmic 9.10 and it also seems to work fine there.

Installing PostgreSQL 8.4 on Ubuntu Jaunty (9.04)

2009-10-09T09:17:00.013-04:00

Inaugural post!

Ubuntu 9.04 (Jaunty Jackalope) includes PostgreSQL 8.3 via the expected apt-get install postgresql. This is great, but PostgreSQL 8.4.1 has a number of nice features that I want (WITH queries, for example). At first I was just going to wait for Ubuntu 9.10, which is coming very shortly, but I am impatient. It turns out there is a pretty easy "Ubuntu way" to get PostgreSQL 8.4 installed on 9.04 without a lot of headaches (like building from source). The solution is using Ubuntu PPAs.

Add the custom PPA for PostgreSQL to /etc/apt/sources.list:

deb http://ppa.launchpad.net/pitti/postgresql/ubuntu jaunty main
deb-src http://ppa.launchpad.net/pitti/postgresql/ubuntu jaunty main

Get the key for these new sources:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8683D8A2

Update the package list:

sudo apt-get update

Install!

sudo apt-get install postgresql-8.4

PostgreSQL 8.4.1 will now be running, but on port 5433, not the standard 5432. I assume this was done so you can run 8.3 and 8.4 at the same time on Ubuntu 9.04. I only want 8.4.1, so I changed the port and restarted:

sudo sed -i.bak -e 's/port = 5433/port = 5432/' \
    /etc/postgresql/8.4/main/postgresql.conf

sudo /etc/init.d/postgresql-8.4 stop
sudo /etc/init.d/postgresql-8.4 start

These PPAs for PostgreSQL only exist because of Martin Pitt, who appears to be responsible for PostgreSQL on Ubuntu, so many thanks to him!