Troy's Blog

Debugging java.lang.OutOfMemoryError: unable to create new native thread on Cloud Foundry

2014-10-01T13:16:00.000+01:00

Within a multi-threaded Java application running on Cloud Foundry we started seeing the following error during our performance tests:

[App/0]  ERR java.lang.OutOfMemoryError: unable to create new native thread

Doing some basic troubleshooting the testers observed the following:

Increasing the memory allocated to the application did not resolve the error.
Running the same Java application on a Linux VM we did not see the error.

So the question arises what is different between a Cloud Foundry Linux Container and a Linux VM that precipitates these errors?

Research

This article on plumber.eu does a great job of describing the unable to create new native thread error. In short the error occurs when we exhaust the available user processes and then continue trying to create processes from within Java.
The article also describes how to resolve the error on a Linux VM, by increasing the number of user processes using ulimit -u. This is a scale-up solution and Cloud Foundry is a scale-out environment thus the higher user process limits are not made available within containers rendering this solution a non-starter.

Investigation

For us the solution needed to be scale-out run multiple Linux Containers each with fewer threads. To implement this solution effectively we needed to know the following:

How many user processes were available within each Linux Container.
How many of those processes are available for us to use as Java threads.

Additionally we wanted to understand why our testers had not seen the error on the Linux VM.

Determining the number of user processes

On a Linux VM to determine the number of user processes available to a user we simply run ulimit -a and look at the max user processes value. However on Cloud Foundry developers do not have an interactive Shell into the Linux Container.
James Bayer came up with a neat hack to work around this using websocketd so I used his hack to create an interactive shell into the Linux Container.

Using the interactive Shell we can easily observe the max user processes available to us is 512:

$ whoami 
vcap

$ ulimit -a
core file size (blocks, -c) 0 
 data seg size (kbytes, -d) unlimited 
 scheduling priority (-e) 0 
 file size (blocks, -f) unlimited 
 pending signals (-i) 1024 
 max locked memory (kbytes, -l) 64 
 max memory size (kbytes, -m) unlimited 
 open files (-n) 16384 
 pipe size (512 bytes, -p) 8 
 POSIX message queues (bytes, -q) 819200 
 real-time priority (-r) 0 
 stack size (kbytes, -s) 8192 
 cpu time (seconds, -t) unlimited 
 max user processes (-u) 512 
 virtual memory (kbytes, -v) unlimited 
 file locks (-x) unlimited

Determining how many of user processes are available for Java threads

At a guess from the max user processes processes value we should be able to run about 500 threads within each Linux container (this will vary based on your Stack, we are currently using Ubuntu Lucid).
To test this I threw together a basic Java application to exhaust the available threads. Running the application in Cloud Foundry I see the following:

tastle@TASTLE-M-W03P tgobbler (master) $ cf logs tgobbler --recent
Connected, dumping recent logs for app tgobbler in org COL-DEV-FT / space Development as taste...
[DEA]     OUT Starting app instance (index 0) with guid 34c88014-fdf0-4f25-b555-b58291d6f879
[App/0]   OUT Threads started: 1
...
[App/0]   OUT Threads started: 490
[App/0]   ERR Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
...
[DEA]     OUT Starting app instance (index 0) with guid 34c88014-fdf0-4f25-b555-b58291d6f879
[App/0]   OUT Threads started: 1
...
[App/0]   OUT Threads started: 490
[App/0]   ERR Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
[App/0]   ERR  at java.lang.Thread.start0(Native Method)
[App/0]   ERR  at java.lang.Thread.start(Thread.java:714)
[App/0]   ERR  at org.trastle.App.main(App.java:12)

So we can see the following:

The application starts a first time, then consumes 490 threads, then dies.
Cloud Foundry notices, then restarts the failing application.
The application starts a second time, then consumes 490 threads, then dies.

After a few more runs it became clear that 490 threads was a consistent max.

Why do we not see this error on an Ubuntu Linux VM?

The Stack we use in Cloud Foundry is Ubuntu Linux so testing this on a Debian VM we should be able to reproduce the failure. However our testers had already observed this was not the case.
So lets look at the max user processes available on the VM:

tastle@tastle-debian:~$ ulimit -a
 core file size          (blocks, -c) 0
 data seg size           (kbytes, -d) unlimited
 scheduling priority             (-e) 0
 file size               (blocks, -f) unlimited
 pending signals                 (-i) 3824
 max locked memory       (kbytes, -l) 64
 max memory size         (kbytes, -m) unlimited
 open files                      (-n) 1024
 pipe size            (512 bytes, -p) 8
 POSIX message queues     (bytes, -q) 819200
 real-time priority              (-r) 0
 stack size              (kbytes, -s) 8192
 cpu time               (seconds, -t) unlimited
 max user processes              (-u) 3824
 virtual memory          (kbytes, -v) unlimited
 file locks                      (-x) unlimited

There are roughly 7x the user processes available to us on on the VM explaining why we could not see the issue. This is largely because the VM is tuned for a lower level of multi-tenancy the the Cloud Foundry Linux Container. In order to reproduce the error on the VM we can reduce the max user processes

tastle@tastle-debian:~$ ulimit -u 512
tastle@tastle-debian:~$ ulimit -a
 core file size          (blocks, -c) 0
 data seg size           (kbytes, -d) unlimited
 scheduling priority             (-e) 0
 file size               (blocks, -f) unlimited
 pending signals                 (-i) 3824
 max locked memory       (kbytes, -l) 64
 max memory size         (kbytes, -m) unlimited
 open files                      (-n) 1024
 pipe size            (512 bytes, -p) 8
 POSIX message queues     (bytes, -q) 819200
 real-time priority              (-r) 0
 stack size              (kbytes, -s) 8192
 cpu time               (seconds, -t) unlimited
 max user processes              (-u) 512
 virtual memory          (kbytes, -v) unlimited
 file locks                      (-x) unlimited

Running the same test application now we see the following:

tastle@tastle-debian:~$ java -jar tgobbler-1.0.jar
...
Threads started: 500
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread

The result largely similar to what we saw in the Cloud Foundry Linux Container. The difference exists because the user processes available represent everything running as that user not just the Java application.

Resolution

We now know we will always be able to create roughly 490 Java threads within a single Cloud Foundry Linux Container.
It makes sense to leave some head room in each container so we will aim to utilise a maximum of 90% of the available threads. This leaves us with a target maximum of 450 threads per Linux Container. When more threads are required we will scale out to multiple containers.

Update: Changes to the Java Buildpack

After discussing this issue, the Cloud Foundry team have very kindly integrated a call to ulimit -a and df -h into the kill script used in the Java Buildpack. The output of this script will also be surfaced over Loggregator making the information readily available.

Extending Cloud Foundry with Open Source Toys

2014-10-01T09:07:00.002+01:00

A talk that I gave with my colleague Matt Johnson describing how we have extended Cloud Foundry using Open Source tools including: Jenkins, Sensu, CollectD and Graphite.

Presented at the London Cloud Foundry User Group September 2014 meetup.

Extending Cloud Foundry - London CF User Group Sept 2014.pptx from Troy Astle

Taking blobs from BOSH releases

2014-09-25T11:49:00.000+01:00

When developing BOSH releases we quite often take packages from existing BOSH releases to save time. Finding the binary blobs you need for the package (from the .blobs directory) is a pain, the files in this directory are all UUIDs making them difficult to work with.

To resolve this issue I wrote a small function for my .bashrc to copy the blobs to a new directory and reverse the UUID naming process that takes place when you upload the blobs using BOSH.

The usage pattern for the function is as follows:

you@yourpc $ git clone git@github.com:FreightTrain/sensu-client-boshrelease.git
you@yourpc $ cd sensu-client-boshrelease
you@yourpc $ bosh create release
...
you@yourpc $ bosh-blobs-realize
...

I hope someone finds this useful.

Troy

Cloud Foundry Summit 2014 Lighting Talk

2014-06-15T18:18:00.002+01:00

At this year's CF Summit, I was given the opportunity to speak with my colleague Matt Johnson about using Cloud Foundry and BOSH at Cisco. The talk covers some of the pain points we had adopting Cloud Foundry, how things have improved and the open source automation pipeline we built to reliably deploy and test our Cloud Foundry installations.

The slides from the talk are up on SlideShare if you want to take a look.

Cloud Foundry Summit 2014

Setting the Disk Quota for your Cloud Foundry apps

2014-03-04T17:57:00.001+00:00

Setting the disk quota using a manifest

It is possible to set the disk quota for an application using the application's `manifest.yml`. The required property is disk_quota:

---
applications:
- name: testapp-large-disk
  memory: 512M
  instances: 1
  disk_quota: 3072
  path: .

Setting the disk quota manually

There is currently no CLI command to set the disk quota for an app, however it is supported by the Cloud Foundry REST API. This means you can set the disk quota using the `cf curl` command.

Example commands

$ cf curl /v2/apps

# Find the metadata.guid for the app you want to increase the disk quota of

$ cf curl /v2/apps/[APP-GUID-HERE] -X PUT -d '{"disk_quota":2048}'

Note: You will need to set the `disk_quota` to the number of megabytes of disk quota you require for your application. This must be less than the maximum allowed disk quota or the command will fail.

Move the window controls to the left in Chrome on Ubuntu

2013-05-23T14:41:00.003+01:00

I regularly switch between using Mac's and Linux machines at home and at work and as such I like to keep my window controls consistent across machines.

My home machine is a Mac running OSX 10.08 and my work Laptop runs Ubuntu Linux 13.04 with Gnome Shell. Both of these desktops have their window controls on the left in the same order:

OSX 10.8

Ubuntu 13.04 + Gnome Shell

This is all nice and harmonious until I install my preferred browser, Google Chrome, on Ubuntu. Chrome places its window controls on the right as per the MS Windows layout.

Google Chrome + Ubuntu 13.04 + Gnome Shell (Default)

To resolve this you can change the default configuration exposed by Chrome by entering the following command at the terminal:

$ gconftool-2 --set /apps/metacity/general/button_layout --type string "close,minimize,maximize:"

To re-load the configuration you'll need to start a new desktop session (log out or restart). Once this is done your chrome window controls will fit in with the rest of your system.

Google Chrome + Ubuntu 13.04 + Gnome Shell (Modified)

I know I'll want to do this again some time and am writing this here for my own reference, hopefully someone else finds it useful also.

Discover how much memory is installed in OSX from the terminal

2013-03-01T05:30:00.000+00:00

Today I was sitting SSHed into an OSX box and needed to know how much RAM was physically installed in the machine. Top of course told me how much RAM was in use however finding the installed RAM took me a few minutes longer than I would have liked.

So this is a note to myself for next time, use the following:

trastle$ system_profiler |  grep -A 9 "BANK [0-9]/DIMM[0-9]"
        BANK 0/DIMM0:

          Size: 2 GB
          Type: DDR3
          Speed: 1333 MHz
          Status: OK
          Manufacturer: 0x80AD
          Part Number: 0x484D54333235533642465238432D48392020
          Serial Number: 0xDEADBEEF

        BANK 1/DIMM0:

          Size: 2 GB
          Type: DDR3
          Speed: 1333 MHz
          Status: OK
          Manufacturer: 0x80AD
          Part Number: 0x484D54333235533642465238432D48392020
          Serial Number: 0xDEADBEEF

Have fun!

Raspberry Pi + Weather Station = A bit of fun

2013-02-27T00:44:00.000+00:00

Over the past six months or so I have been working on a new release of the UK Met Office's WOW service. It's been a fun project so when my Dad's birthday rolled around earlier in the year I decided the best way to show him what I was working on was to give him a weather station and hook it up to WOW.

In the end I bought him a USB weather station and a Raspberry Pi and hooked the two together to upload data to the web. I have finally gotten around to writing up some instructions for the project and have posted them on github along with a couple of bash and udev files to make the process easier.

For anyone who is interested in using a Raspberry Pi to upload data from their USB weather station the instructions are here: http://trastle.github.com/rpi-weather/

Enjoy.

Hiding elements from Blogger's Dynamic View Template

2012-03-13T10:09:00.000+00:00

I've been playing a little with Blogger's Dynamic Views template in the past week, so far it seems pretty neat for photo based content.

One of the quirks of the new Dynamic Views is that, unlike standard Blogger templates, users cannot edit the HTML of the page. This is a pretty huge drawback but it is tempered by the fact that you can still modify the CSS using the template designer.

In order to make the blog I was working on a little less cluttered I wanted to hide a number of elements on the page. This may be useful to others so I have itemised my changes below.

Hide the search box (from the top right):

/* Hide search */
.header-bar #search{ display: none !important; }

Hide the Google Chrome feedback link (from the bottom right):

/* Hide the google chrome feedback button */
a.feedback  { display: none !important; }

Hide the "Home" button (from the pages list):
Note: This assumes the home button is the first page in your list of pages.
```
/* Hide the home button */
#pages ul li:first-child  { display: none !important; }
```

Hide the "Sharing Controls" (Facebook, Twitter buttons etc):

/* Hide the sharing controls */ 
.share-controls { display: none !important; }

Hide the gadget dock (from the right hand side of the page):

/* Hide the gadget Dock */
#gadget-dock { display: none !important; }

Hide the views on the main page:

/* Hide the views on the main page */
#views  { display: none !important; }

Hide the dynamic view switcher:

/* Hide the views on the main page */
#views { display: none !important; }
/*Hide the vertical bar from before the first page link */
#pages::before { border-left-width: 0 !important;}

It is worth noting that these changes do not remove elements from the structure of the page (anyone reading the source of the page can still see the elements) but it does make the overall page a little cleaner.

You can see the results of my changes here and here.

Bash function to show the linked locations of a binary

2012-02-10T04:24:00.000+00:00

The which command on Linux will show you the location, from your PATH, that a binary will be executed from.

trastle@w500:~$ which javac
/usr/bin/javac

This is very useful, however often you will find that the location in your PATH is actually a symbolic link to another location.

trastle@w500:~$ which java | xargs readlink $1
/etc/alternatives/java

This in turn can be another symbolic link:

trastle@w500:~$ which java | xargs readlink $1 | xargs readlink $1
/opt/ibm/java/ibm-java2-i386-60/bin/java

This means it can take a few commands to view all the links in play when a command is called. Oddly I find myself doing this quite often so I throw a small function in my .bashrc to help out:

function whichlink
{
  local loc=`/usr/bin/which $1`;
  echo "$loc"
  
  while [ -h "$loc" ]
  do
    loc=`/bin/readlink $loc`
    echo "--> $loc"
  done
}

The output is as follows:

trastle@w500:~$ whichlink javac
/usr/bin/javac
--> /etc/alternatives/javac
--> /opt/ibm/java/ibm-java2-i386-60/bin/java

Alternately if you don't want to print all of the links along the way and just want to know the final destination the following is useful:

trastle@w500:~$ readlink -f `which javac`
/opt/ibm/java/ibm-java2-i386-60/bin/javac

I am certain at some point I'll want to do this on another machine and text is more reliable than my long term memory. Enjoy.

Using --exclude-from in rsync

2011-11-29T14:33:00.000+00:00

I've been toying with some rsync scripts tonight and after reading the rsync man page I still needed a little trial and error to determine the exact behaviour of the --exclude-from flag.

The --exclude-from flag specifies a file that contains exclude patterns (one per line). The rsync man page is VERY detailed on the subject of filters and how they can be used.

Looking at a simplified version of my scenario, consider the following directory structure:

   Source
   |---.DS_Store
   |---Alpha
   |  |---.DS_Store
   |  |---Document 1.txt
   |  |---Document 2.txt
   |  `---Temporary Items
   |     |---.DS_Store
   |     |---TI1.tmp
   |     `---TI2.tmp
   |---Beta
   |  |---.DS_Store
   |  |---Document 3.txt
   |  |---Document 4.txt
   |  `---Temporary Items
   |     |---.DS_Store
   |     |---TI2.tmp
   |     `---TI4.tmp
   `---Temporary Items
      |---.DS_Store
      `---TI5.tmp

I want backup the Source directory shown above and exclude the following from my backup:

All of the ".DS_Store" files.
All of the "Temporary Items" directories.
All files contained in the "Temporary Items" directories.

My final backup will look like this:

   Destination
   |---Alpha
   |  |---Document 1.txt
   |  `---Document 2.txt
   `---Beta
      |---Document 3.txt
      `---Document 4.txt

Making this happen is very simple. The exclusions file (skip.txt) only needs to contain the following two lines:

Temporary Items
.DS_Store

The rsync command to perform the backup is as follows:

rsync -az  --exclude-from=./skip.txt ./Source/ ./Destination/

The output of this command is the "Destination" directory shown above.

There were two things that caused me greif getting my script to work:

To exclude a file (or directory) with spaces in the name, the names DO NOT need to be quoted or escaped.
To exclude all files underneath a directory you need only exclude the directory itself.

Now that this is committed to my long term memory I can move on.

Fixing unmappable character during Java compilation on Ubuntu

2011-11-21T05:23:00.001+00:00

In the lab we have a standard header at the top of our source files, a copyright statement and licence. This header contains an ISO 8859-1 encoded copyright character (©).

Normally the header blurs into the background, the exception to this rule is the first time a do a build on a new Ubuntu system and I see the following error:

[javac] /src/com/yourcorp/HelloWorld.java:8: unmappable character for encoding ASCII
[jacac] /* ?? Copyright Yourcorp 2011

The Java compiler is expecting ASCII characters rather than ISO 8859-1. To resolve this error I do the following:

Edit /var/lib/locales/supported.d/local and add:
```
en_AU ISO-8859-1
en_US ISO-8859-1
```
Open a terminal and run:
```
$ sudo dpkg-reconfigure locales
```
Edit /etc/environment and add:
```
LANG="EN_US"
```
Reboot.

With Ubuntu being so stable these days it's a long time between re-installs so I have posted this here for next time when I forget.

Run scripts when gnome-screensaver starts or stops in Ubuntu

2011-06-26T08:54:00.074+01:00

For a project I have been playing with I need to be able to run a script when gnome-screensaver starts or stops on Ubuntu (10.04).

The gnome-screensaver FAQ provides detail on the dbus signal fired when gnome-screensaver activates and deactivates. The FAQ even has sample perl code to respond to the signal. Unfortunately the sample code polls dbus-monitor which is too inefficient for script I plan to run in the background on my laptop.

Python offers a great dbus library which allows signal receivers to be registered with dbus resulting in an efficient event listener.

On to the script.

Step 1: Setup the required files and directories

Make a directory for your work:
```
sudo mkdir -p /opt/ssTrigger
```
Change yourself the owner of the new directory:
```
sudo chown `whoami`:`whoami` /opt/ssTrigger
```

Create stubs for the scripts to be used

touch /opt/ssTrigger/ssTrigger /opt/ssTrigger/ssStart /opt/ssTrigger/ssStop

Make the stub scripts executable:
```
chmod 755 /opt/ssTrigger/*
```

Step 2: Edit the ssTrigger script

Open the ssTriger file created earlier:
```
gedit /opt/ssTrigger/ssTrigger
```

Add the following to ssTrigger:

#!/usr/bin/env python
from gobject import MainLoop
from dbus import SessionBus
from dbus.mainloop.glib import DBusGMainLoop
from subprocess import Popen

class SSTrigger:
    def __init__(self):
        DBusGMainLoop(set_as_default=True)
        self.mem='ActiveChanged'
        self.dest='org.gnome.ScreenSaver'
        self.bus=SessionBus()
        self.loop=MainLoop()
        self.bus.add_signal_receiver(self.catch,self.mem,self.dest)
    def catch(self,ssOn):
        if ssOn == 1: #Screensaver turned on
            Popen(["/opt/ssTrigger/ssStart"])
        else: #Screensaver turned off
            Popen(["/opt/ssTrigger/ssStop"])

SSTrigger().loop.run()

Save and close the file.

Step 3: Set ssTrigger to start when you login:

Open the Startup Applications menu:
System -> Preferences -> Startup Applications
Click "Add" to create a new startup application.
Enter the following details:
Name: Screensaver Trigger Script
Command: /opt/ssTrigger/ssTrigger
Click "Add" to save your new application.
Restart your computer.

Finishing up:
Once you have restarted you can run:

ps aux | grep ssTrigger

Check the output to make sure the ssTrigger script has started at login.

Edit the ssStart and ssStop scripts to run commands as gnome-screensaver starts and stops respectively.

Downgrade to Firefox 3.6 on Ubuntu 11.04 Natty

2011-06-24T14:39:00.011+01:00

We have a requirement in the office to downgrade to Firefox 3.6 on some test machines running Ubuntu Natty (11.04). To do this you can pin the firefox and firefox-branding packages to older packages from the Ubuntu Maverick (10.10) release.

Step 1: Edit your software sources to add Maverick

Open your sources.list file:
```
sudo gedit /etc/apt/sources.list
```

Add the folowing three lines:

# Maverick, used for firefox and firefox-branding 
deb http://ftp.iinet.net.au/pub/ubuntu/ maverick main 
deb http://ftp.iinet.net.au/pub/ubuntu/ maverick-updates main

Save and close the file

Step 2: Pin the packages firefox and firefox-branding

Create the pin file for firefox:

sudo gedit /etc/apt/preferences.d/firefox

Add the following:

Package: firefox
Pin: release n=natty
Pin-Priority: -10

Package: firefox
Pin: release n=maverick
Pin-Priority: 900

Save and close the file.

Create the pin file for firefox-branding:

sudo gedit /etc/apt/preferences.d/firefox-branding

Add the following:

Package: firefox-branding
Pin: release n=natty
Pin-Priority: -10

Package: firefox-branding
Pin: release n=maverick
Pin-Priority: 900

Save and close the file.

Step 3: Check your apt policy to make sure it worked

Update your apt cache:
```
sudo apt-get update
```

Show the apt policy for firefox:

sudo apt-cache policy firefox

Sample output:

sudo apt-cache policy firefox
firefox:
  Installed: 5.0+build1+nobinonly-0ubuntu0.11.04.2
  Candidate: 3.6.18+build2+nobinonly-0ubuntu0.10.10.1
  Package pin: 3.6.18+build2+nobinonly-0ubuntu0.10.10.1
  Version table:
 *** 5.0+build1+nobinonly-0ubuntu0.11.04.2 900
        500 http://ftp.iinet.net.au/pub/ubuntu/ natty-updates/main i386 Packages
        500 http://ftp.iinet.net.au/pub/ubuntu/ natty-security/main i386 Packages
        100 /var/lib/dpkg/status
     4.0+nobinonly-0ubuntu3 900
        500 http://ftp.iinet.net.au/pub/ubuntu/ natty/main i386 Packages
     3.6.18+build2+nobinonly-0ubuntu0.10.10.1 900
        500 http://ftp.iinet.net.au/pub/ubuntu/ maverick-updates/main i386 Packages
     3.6.10+build1+nobinonly-0ubuntu3 900
        500 http://ftp.iinet.net.au/pub/ubuntu/ maverick/main i386 Packages

The line you are most interested in is "Candidate".
This indicates the package that apt ranks as the installation candidate.

Repeat this check for firefox-branding
```
sudo apt-cache policy firefox-branding
```

Step 4: Install firefox 3.6

Remove firefox 5.x:
```
sudo apt-get remove firefox
```
Install firefox 3.6:
```
sudo apt-get install firefox
```

And you're done. Enjoy your outdated version of Firefox!

Update - July 24th 2011:

How to reverse the process:

Remove these three (3) lines you added to /etc/apt/sources.list

# Maverick, used for firefox and firefox-branding 
deb http://ftp.iinet.net.au/pub/ubuntu/ maverick main 
deb http://ftp.iinet.net.au/pub/ubuntu/ maverick-updates main

Unpin the firefox and firefox-branding packages:

sudo rm /etc/apt/preferences.d/firefox
sudo rm /etc/apt/preferences.d/firefox-branding

Update your apt cache
```
sudo apt-get update
```
Install the latest Firefox:
```
sudo apt-get upgrade
```
Close ALL your Firefox windows and re-open Firefox.

Update - December 6th 2011:

From the comments people have posted below I can see that this process works to downgrade Firefox on Ubuntu 11.10 as well as Ubuntu 10.04.

Downgrading packages to the Ubuntu 10.10 release will only work on Ubuntu 11.04 or better. Do not use this procedure on Ubuntu releases older than 11.04. If you try to do this you will be upgrading packages not downgrading.

Lock input without a screen saver in Linux

2011-04-29T08:50:00.008+01:00

We came across this problem in the office today. How do we lock the input devices on a Linux machine (we were using Ubuntu 11.04) and keep a the current applications showing and updating? The solution we ended up with was to use xlock.

First you'll need to install xlock:

sudo apt-get install xlockmore

Then run xlock from a terminal with the following options:

xlock -mode blank -geometry 0x0 -timeout 2

The xlock command locks keyboard and mouse input, the flags do the following:
-mode blank displays a blank box in the top left corner of your monitor.
-geometry 0x0 makes the size of the blank box 0x0 pixels.
-timeout 2 sets the timeout (in seconds) for the password prompt if anyone hits a key.

Update 2011-05-02: A second (better) solution

First you'll need to install xtrlock:

sudo apt-get install xtrlock

Then run xtrlock from a terminal:

xtrlock

This will start xtrlock and lock the screen. To unlock the screen type your password and hit enter.

Upgrading from Lucid to Maverick using a local mirror

2010-10-15T01:56:00.000+01:00

I maintain a local Ubuntu mirror in our Lab at work. This morning the first user tried to update from Lucid to Maverick. There was a couple of teething issues but with a couple of changes it went off without a hitch.

Step 1: Correctly specify your mirror to the Software Sources dialog:

Open a terminal

(Applications -> Accessories -> Terminal)

Open your list of mirrors:

trastle$ sudo gedit /usr/share/python-apt/templates/Ubuntu.mirrors

Add a location for your local mirror. Mine looks like this:
```
...
#LOC:ADL
http://adl-mirror/ubuntu-repo/ubuntu/
...
```
Save and close the file.
Close the terminal.

Step 2: Select your mirror in the software sources dialog

Open the software sources dialog
(System -> Administration -> Software Sources)
Open the "Ubuntu Software" tab
Click the "Download From:" combo box.
Select "Other..."
Select the mirror you just added from the list
Click "Choose Server" at the bottom right.
Click "Close" at the bottom right.
Click "Close" in the information out of date dialog that pops up.

Step 3: Disable all third party sources

Open the software sources dialog
(System -> Administration -> Software Sources)
Open the "Other Software" tab
Untick every source in the list.
Click "Close" at the bottom right.
Click "Close" in the information out of date dialog that pops up.

Step 4: Do the Upgrade

Ensure your on a network with Internet access as well as access to your mirror.
Open a terminal
(Applications -> Accessories -> Terminal)
Change to the root user:
```
trastle$ sudo -i
```
Update your available software:
```
root# apt-get update 
```
Do the upgrade
```
root# do-release-upgrade 
```

The following message will appear shortly after starting the upgrade:

Updating repository information
WARNING: Failed to read mirror file

No valid mirror found 

While scanning your repository information no mirror entry for the 
upgrade was found. This can happen if you run a internal mirror or if 
the mirror information is out of date. 

Do you want to rewrite your 'sources.list' file anyway? If you choose 
'Yes' here it will update all 'lucid' to 'maverick' entries. 
If you select 'No' the upgrade will cancel.

Your using a local mirror so choose Yes

The release upgrade will take some time to complete.

Step 5: Re-enable third party sources

Open the software sources dialog
(System -> Administration -> Software Sources)
Open the "Other Software" tab
Tick the sources you want to use.
Click "Close" at the bottom right.
Click "Reload" in the information out of date dialog that pops up.

Enjoy your upgraded system.
I hope someone finds this helpful.

Building a ARM powered Debian VM with QEMU on Ubuntu Lucid

2010-07-22T13:51:00.002+01:00

I have recently spent some time trying to get an emulated ARM machine up and running on an x86 Ubuntu Lucid host. Initially I wanted Ubuntu for the client OS but I found that the only ARM installers available were for the Ubuntu Netbook Edition, which does not suit my needs at this point. So I went with Debian Lenny (the current Debian stable release) which has a myriad of ARM installers available.

When tracking down info about the Debian ARM port you'll quickly discover that there are two ports available ARM and ARMEL. ARM is the original port (now deprecated) and ARMEL is name of the newer code stream which supports ARM EABI.

The emulation software I am using is QEMU. An open-source project started by Fabrice Bellard (who also founded the ffmpeg project!). QEMU is fairly versatile and will emulate a myriad of CPU architectures including ARM, Sparc, PPC and even s390 (under KVM).

The Ubuntu Lucid repositories do contain packages for QEMU. However after installing via apt-get I found the binary failed out with a segmentation fault almost immediately. Not to worry the QEMU build system is easy to master.

Part 1: Building and installing QEMU:

Download the QEMU source (0.12.4, the current release, is available here):

trastle$ wget http://download.savannah.gnu.org/releases/qemu/qemu-0.12.4.tar.gz

Unpack the tar ball:
```
trastle$ tar -xvvf qemu-0.12.4.tar.gz
```
Install the required packages to build QEMU:
```
trastle$ sudo apt-get build-dep qemu
```

Build and install QEMU:

trastle$ cd qemu-0.12.4
trastle$ ./configure
trastle$ make
trastle$ sudo make install

Part 2: Install Debian Lenny in QEMU:

Change to the directory you want to build your VM in:
```
trastle$ mkdir ~/arm
trastle$ cd ~/arm
```

Download the current build of the Debian Lenny ARMEL kernel (vmlinuz) and installer (initrd) images from a local Debian mirror:

trastle$ wget ftp://ftp.au.debian.org/debian/dists/lenny/main/installer-armel
         /current/images/versatile/netboot/vmlinuz-2.6.26-2-versatile
trastle$ wget ftp://ftp.au.debian.org/debian/dists/lenny/main/installer-armel
         /current/images/versatile/netboot/initrd.gz

Create a disk image for QEMU (more details on qemu-img).
Importantly, the raw format will allow you to mount the image from Ubuntu once its populated:
```
trastle$ qemu-img create -f raw armdisk.img 8G
```
Start the Debian install with QEMU:
```
trastle$ qemu-system-arm -m 256 -M versatilepb \
         -kernel ~/arm/vmlinuz-2.6.26-2-versatile \
         -initrd ~/arm/initrd.gz \
         -hda ~/arm/armdisk.img -append "root=/dev/ram"
```
QEMU will open a terminal window and within that window the Lenny installer will kick into action. Follow the installer's directions and allow the install to begin. Installation will be slower than normal. Allow 4 or 5 hours for it to complete.
As the install completes you'll be informed no boot loader is present, don't worry QEMU takes the place of the boot loader. Once the install completes the VM will reboot kicking off the installer again, don't proceed, just kill QEMU.

Part 3: Running your ARM Lenny install in QEMU:

Once the installation is complete you will need to copy the initrd from the installed system. To do this you must mount the QEMU disk image.
```
trastle$ mkdir ~/arm/mount/
trastle$ sudo mount -o loop,offset=32256 ~/arm/armdisk.img ~/arm/mount
trastle$ cp ~/arm/mount/boot/initrd.img-2.6.26-2-versatile ~/arm/.
trastle$ sudo umount ~/arm/mount
```
32256 is not just a random number, it's the sector where the first disk partition begins.

Boot your Debian Lenny VM:

trastle$ qemu-system-arm -m 256 -M versatilepb \
         -kernel ~/arm/vmlinuz-2.6.26-2-versatile \
         -initrd ~/arm/initrd.img-2.6.26-2-versatile \
         -hda ~/arm/armdisk.img -append "root=/dev/sda1"

Now you have a running ARM VM to do with as you please.

Boot to single user mode in Ubuntu 10.04 Lucid Lynx

2010-06-21T11:43:00.004+01:00

This is symptomatic of how often I break X.org but I often need to boot my Ubuntu 10.04 machine into single user mode and fix some configuration file before rebooting into the GUI. I forget how to do it every time so here is how:

Hold the Shift key at boot to display the Grub boot menu.
Select the top Grub entry, it will be similar to:
```
Ubuntu, with Linux 2.6.32-22-generic-pae
```
Hit e to edit the the Grub entry.

Find the line that looks like this:

linux /boot/vmlinuz-2.6.32-22-generic-pae
root=UUID=xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx
ro vga=794 quiet splash

Change the line to:

linux /boot/vmlinuz-2.6.32-22-generic-pae
root=UUID=xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx
init=/bin/bash rw

Hit crtl+x to boot

Your machine will now do a one time boot into single user mode. Once you reboot the changes you just made to Grub will be reverted.

** /bin/sh corrected to /bin/bash thanks Hola2040.

Activate the cron daemon on a DroboFS

2010-06-20T07:34:00.003+01:00

I recently purchased a DroboFS NAS from DataRobotics. The device runs a minimal Marvell Linux OS on top of a low power ARM CPU (ARM926EJ-S).

The DroboFS supports third party development through the DroboApps platform. This allows end users to compile ARM applications using the GNU tool chain and run them on their DroboFS. On top of this these applications can be run as services using a small service API provided by DroboApps.

A number of DroboApps are available to enable dormant services on the DroboFS including the SSH, HTTP and FTP servers.

The DroboFS ships with a cron daemon tucked away at /usr/sbin/crond however there is no DroboApp available to enable the daemon. To rectify this I have written a small DroboApp script to activate the cron daemon at boot.

To get cron running as a service you'll need to do the following:

Enable DroboApps on your DroboFS.
Install the Dropbear SSH DroboApp.
SSH into your DroboFS

Create a directory for the cron DroboApp:

mkdir /mnt/DroboFS/Shares/DroboApps/crond

Save the following service.sh into your cron DroboApp directory

#!/bin/sh
# ------------------------------------------------------------
# service.sh for cron DroboApp
#
# Exposes the crond binary existing on the DroboFS as a 
# DroboApps service.
# ------------------------------------------------------------

# Binaries used
AWK="/usr/bin/awk"
GREP="/bin/grep"
CROND="/usr/sbin/crond"
DATE="/bin/date +%Y:%m:%d-%H:%M:%S" # Nicely formatted date
ECHO="/bin/echo"
PS="/bin/ps"

# Load the DroboApps service functions
. /etc/service.subr

# Required DroboApps variables
prog_dir=`dirname \`realpath $0\``
name="crond"                    # service name
version="1.14.2"                # program version
pidfile=${prog_dir}/crond.pid # location of pid file
logfile=${prog_dir}/crond.log # location of log file

# Start crond
start()
{
  # Start the service
  $CROND

  # Create the pidfile
  pid=`$PS | $GREP $CROND | $GREP -v grep | $AWK '{print $1}'`
  $ECHO $pid > $pidfile
}

case "$1" in
  start)
    start_service
    $ECHO "`$DATE` Started cron service" >> $logfile    
    ;;
  stop)
    stop_service
    $ECHO "`$DATE` Stopped cron service" >> $logfile    
    ;;
  restart)
    stop_service
    sleep 3
    start_service
    $ECHO "`$DATE` Restarted cron service" >> $logfile    
    ;;
  status)
    status
    ;;
  *)
    $ECHO "Usage: $0 [start|stop|restart|status]"
    exit 1
    ;;
esac

Restart your DroboFS.

Now the cron daemon is running from boot on your DroboFS. To add tasks for the cron daemon to run you will need to SSH into your DroboFS and run the following:

mkdir -p /var/spool/cron/crontabs
crontab -e

I hope someone finds this helpful.

2010-06-24: Edit added "mkdir -p /var/spool/cron/crontabs" Thanks pimvanderzwet

Atheros AR5212 in Ubuntu 10.04 Lucid Lynx

2010-05-09T05:07:00.006+01:00

I've just finished a fresh install of Ubuntu Lucid Lynx 10.04 on my crusty old Thinkpad T60. Everything is working great with the exception of my Atheros AR5212 wireless card.

Clicking the network manager applet in the panel I see the wireless listed as "Device not ready". This is tedious.

The solution to this issue is to remove the mainline ATK5K wireless driver from your Kernel and replace it with non-mainline MadWifi driver (source). MadWifi has superior support for the AR5212 card. This shouldn't take you more than about 20 minutes to sort out.

Open a terminal: Aplications -> Accessories -> Terminal

To check on your wireless card and make sure you have the AR5212:

trastle$ sudo lshw -c network

The result will look similar to this:

*-network
    description: Wireless interface
    product: AR5212 802.11abg NIC
    vendor: Atheros Communications Inc.
    physical id: 0
    bus info: pci@0000:03:00.0
    logical name: wifi0

Now you know your sporting an Atheros AR5212.

Install some software you'll need to build MadWifi:

trastle$ sudo apt-get install subversion linux-kernel-headers build-essential \
libssl-dev

Now go to your desktop
```
trastle$ cd ~/Desktop 
```
Make a directory to put the MadWifi source in:
```
trastle$ mkdir madwifi-src
```

Get the latest MadWifi source:

trastle$ svn checkout http://madwifi-project.org/svn/madwifi/trunk madwifi-src

Change into your new source directory:
```
trastle$ cd madwifi-src
```
Change to the root user:
```
trastle$ sudo -i
```

Blacklist the non working ATH5K drivers:

root$ echo "# Block ATH5K" >> /etc/modprobe.d/blacklist
root$ echo "blacklist ath9k" >> /etc/modprobe.d/blacklist
root$ echo "blacklist ath5k" >> /etc/modprobe.d/blacklist

Change to the MadWifi source directory

root$ cd /home/[your user]/Desktop/madwifi-src

Build and install the MadWifi driver:
```
root$ make && make install -d
```
Add the MadWifi driver to your Kernel
```
root$ echo ath_pci >> /etc/modules
```

Now save anything else you have been working on and reboot your laptop. After the reboot your wireless will be working nicely.

2010-06-20: Edit added step 10. Thanks Xi.

Building XMLStarlet in Snow Leopard (OSX 10.6)

2010-03-20T06:27:00.006+00:00

XMLStarlet is a command line XML toolkit. Very useful if you want to parse some data from a webpage, RSS feed of similar. XMLStarlet does not provide a binary version for Snow Leopard. They do however provide their source under the MIT licence. So your free to download, build and modify.

To get XMLStarlet up and running on Snow Leopard is nice and easy.

Install Xcode with the Unix the Unix Dev Support Option (see my past post about building wget for more details on this).
Go download yourself a copy of the XMLStarlet source (I am using v 1.0.1).
XMLStarlet requires libxml2 and libxslt. Snow Leopard ships with both these libraries installed, however they are not in the location expected by the XMLStarlet build script so some symlinks are required. Enter the following in a terminal:
```
sudo ln -s /usr/lib/libxslt.1.dylib /usr/lib/libxslt.a
sudo ln -s /usr/lib/libexslt.dylib /usr/lib/libexslt.a
sudo ln -s /usr/lib/libxml2.dylib /usr/lib/libxml2.a
```
Unpack the XMLStarlet tarball by double clicking it in Finder.
Take note of the name of the newly unpacked directory (something like xmlstarlet-1.0.1).
Open a terminal and cd to where you just unpacked (by default this will be the directory below).
```
cd ~/Downloads/xmlstarlet-1*
```
Run the configure script:
```
./configure
```
Run the builds
```
make
```
Run the installer:
```
sudo make install
```

Finally, to test that the install worked correctly enter the following in a terminal:

TA:~ troy$ xml 
XMLStarlet Toolkit: Command line utilities for XML
... 
XMLStarlet is a command line toolkit to query/edit/check/transform XML documents 
(for more information see http://xmlstar.sourceforge.net/)

Enjoy.

Proxy environment variables and sudo in Ubuntu

2009-11-21T13:44:00.001+00:00

At work we use a proxy server to access the internet from our test network. We also have an internal Ubuntu mirror on the test network to install updates and packages, this server must be accessed by the test machines directly without the proxy.

Setting up the proxy and the exception for the mirror can be done in a terminal using the environment variables http_proxy, https_proxy, ftp_proxy, and no_proxy.

In Ubuntu these variables are all set in the /etc/environment file. On my test system this file contains the following:

trastle@trastle-test:~$ cat /etc/environment
...
http_proxy="http://webproxy:3128/"
ftp_proxy="ftp://webproxy:3128/"
https_proxy="https://webproxy:3128/"
no_proxy="office-mirror"

Testing that these settings have been applied to the current terminal can be done by calling printenv:

trastle@trastle-test:~$ printenv | grep proxy
http_proxy=http://webproxy:3128/
ftp_proxy=ftp://webproxy:3128/
https_proxy=https://webproxy:3128/
no_proxy=office-mirror

This output shows that all of the required environment variables are set. However when I try to perform an update from the office mirror it fails! The http_proxy is being used to access hosts clearly specified in the no_proxy variable.

trastle@trastle-test:~$ sudo apt-get update
Ign http://office-mirror jaunty Release.gpg
...
Err http://office-mirror jaunty/main Packages
503 Service Unavailable
...
W: Failed to fetch http://office-mirror/ubuntu/dists/jaunty/main/binary-i386/Packages
503 Service Unavailable
...
E: Some index files failed to download, they have been ignored, or old ones used instead.

This all makes more sense when you check the environment variables that sudo is running with:

trastle@trastle-test:~$ sudo printenv | grep proxy
http_proxy=http://webproxy:3128/

For safety sudo runs with an minimal environment. This default list is hard coded into sudo (initial_keepenv_table in env.c if you want to go read the source). Ubuntu patches this list to include http_proxy but not no_proxy. This breaks the ability to get updates from the local mirror. Sudo can be queried to determine which environment variables it preserves by default:

trastle@trastle-test:~$ sudo sudo -V
Sudo version 1.6.9p17
...
Sudoers path: /etc/sudoers
...
Environment variables to preserve:
http_proxy
XAUTHORIZATION
XAUTHORITY
TZ
PS2
PS1
PATH
MAIL
LS_COLORS
KRB5CCNAME
HOSTNAME
HOME
DISPLAY
COLORS
...

The solution to this problem is to modify the list of environment variables that sudo will preserve by editing the /etc/sudoers file and providing an explicit list of which environment variables to be preserved, overwriting the default list. The configuration that must be added to /etc/sudoers is as follows:

Defaults env_keep="no_proxy http_proxy https_proxy ftp_proxy XAUTHORIZATION \
XAUTHORITY TZ PS2 PS1 PATH MAIL LS_COLORS KRB5CCNAME HOSTNAME HOME DISPLAY COLORS"

This will preserve all of the default environment variables as well as no_proxy, https_proxy and ftp_proxy. You can test this setting to prove that sudo is now preserving these extra environment variables by running:

trastle@trastle-test:~$
sudo printenv | grep proxy
http_proxy=http://webproxy:3128/
ftp_proxy=ftp://webproxy:3128/
https_proxy=https://webproxy:3128/
no_proxy=office-mirror

With these additional environment variables preserved updates and installs from the command line using your local mirror and ignoring your proxy server will work like a charm.

Apache Directory Studio Fails to start on Ubuntu 9.04

2009-10-29T10:32:00.002+00:00

I needed to install Apache Directory Studio this morning and after downloading unpacking version 1.4.0 I could not get it to run. Starting ADS at the command line resulted in the following output, the splash screen and a small sad blank window.

$ ./ApacheDirectoryStudio
0    [main] INFO  org.apache.directory.studio.Application  - Entering Apache Directory Studio.
4003 [main] INFO  org.apache.directory.studio.Application  - Exiting Apache Directory Studio.

Reading the logs in ~/.ApacheDirectoryStudio/.metadata/.log revealed an exception:

!ENTRY org.eclipse.ui.workbench 4 0 2009-10-29 09:29:30.476
!MESSAGE Widget disposed too early!
!STACK 0
java.lang.RuntimeException: Widget disposed too early!
at org.eclipse.ui.internal.WorkbenchPartReference$1.widgetDisposed(WorkbenchPartReference.java:171)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:117)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1158)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1182)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1163)
...

After a bit of a hunt through the ADS mailing lists I learned that this error is due to a problem with the Mozilla XULRunner version on my system.

In order to get ADS to run correctly I had to specify the version XULRunner it should run with manually:

./ApacheDirectoryStudio -vmargs -Dorg.eclipse.swt.browser.XULRunnerPath=/usr/lib/xulrunner/xulrunner-1.9.1

The original Mozilla bug is detailed here, and this solution for ADS found here.

Building wget in Snow Leopard

2009-10-28T23:24:00.005+00:00

I like Snow Leopard, it's a nifty OS, but occasionally something will baffle me. This evening it was the exclusion of wget from the command line tools.

Clearly this sucks and living without wget is not an option. Thankfully wget is free software so it's easy to grab the source and build it yourself.

Firstly you'll need to have Xcode installed with "Unix Dev Support". The Unix Dev Support option gives you gcc, make and other essential goodies. Xcode is on the Mac OS 10.6 DVD in the optional installs section (you can also download Xcode from Apple).

Next you'll need to grab the latest wget source from: ftp://ftp.gnu.org/pub/gnu/wget/ or click here.

To compile and install wget complete the following:

Unpack the wget tarball by double clicking it in Finder.
Take note of the name of the newly unpacked directory (something like wget-1.12).
Open a terminal and cd to where you just unpacked (by default this will be the directory below).
```
cd ~/Downloads/wget-1.*
```
Run the configure script:
```
./configure
```
Run the build and installer:
```
sudo make install
```
All done. Now you can delete the wget folder and tarball in your ~/Downloads directory.

Once your done you'll be able to run wget in your terminal.

Much better.

When you forget to start screen...

2009-10-22T02:40:00.001+01:00

Screen and nohup are great tools which can be used, by someone with forethought, to keep their processes running after they disconnect from a server.

The problem with both nohup and screen is that you need to use them before starting a process. If you forgot or didn't realize the process was going to take ages then your all out of luck.

Recently on a man page hunt I discovered a workaround for this situation using disown. At the terminal with the long running process enter the following:

ctrl + z  (pause the current process and background it)
bg        (resume the last backgrounded process in the background)
disown    (disassociate the last process from the current terminal) 
exit      (to close the current terminal)

Now your disconnected, your process is still running and you can go home. If you don't want to go home and instead want to check that your process is still running, ssh back into your server and run ps aux and hunt it down in the process list.