From the Canyon Edge - :-Dustin Kirkland

How My Work Benefits Free Software

2010-08-30T16:24:00.001-07:00

I'm personally offended when Canonical and Ubuntu are dogged about a perceived lack of contribution to the Free Software ecosystem.

So I'd like to talk about the work I have been doing at Canonical on Ubuntu, and how I believe it benefits Free Software in 7 important ways:

Packaging
Documentation
Communication
Conferences
Upstream Contributions
Being an Upstream
Bringing Free Software to the Masses

1. Packaging

Perhaps the highest volume, least glorious work I do is around a process called "packaging". It's not very sexy, but it is tremendously important. Packaging is what allows you to go to the Ubuntu Software Center, choose a program, click 'install' and then run that program. Packaging also allows command line (server) users to simply type "sudo apt-get install " to install a program. Millions of iPhone and Android users are familiar with this concept, with portals like the App Store and the Android Market. Packaging makes Linux accessible to millions of people who are not willing or able to run "./configure && make && make install", or chase down a complex, cascading set of library dependencies. We, the packagers, do that hard work, and ensure that program installation (and uninstallation!) is an simple, painless experience.

I've packaged many dozens of applications for Ubuntu (and some of those have been uploaded to Debian as well). In many cases, this is work that the developers of this software is unwilling or unable to do.

In practice...

About a year ago, I had solar panels installed on my roof. The power inverter attached to those panels actually has a USB input and some Windows-only software that can read statistics about my photo-voltaic output. I first checked the Ubuntu archive for a package that could communicate with my inverter, to no avail. Searching the web, I found a guy named Curt Blank who wrote just such a GPL Free Software program called 'Aurora'. After a few dozen emails back and forth, and sending patches his way, we finally got it working like a charm against my hardware. I wanted to make sure the next Ubuntu user to buy an Aurora inverter could simply "sudo apt-get install aurora", so I packaged Curt's project and uploaded it to Ubuntu. It's available now, as of 10.04 (Lucid). Curt was pleasantly surprised and appreciative of having his software included in a distribution, and available to all Ubuntu users!

2. Documentation

No one should *ever* play down the importance of documentation! Shrink-wrap software always comes with a manual. Many Windows/Mac users buying shrink wrap software consult that manual when something goes awry. Free Software is delivered almost exclusively electronically -- but this is no excuse for omitting a manual! Documentation for Free Software comes in many forms. Web delivered content, such as Wiki's are quite popular. The http://help.ubuntu.com/community documentation wiki is *outstanding*!

As an Ubuntu Community Member, I am a frequent contributor to that documentation, particularly in the UEC and Virtualization sections. This information helps many first time Free Software users clear their first hurdle, and gives them confidence that there is a tremendous community of fellow users who have solve many of the problems they encounter. Ubuntu Community documentation is some of the best around. Google for almost any given Linux help topic (printing, dvd, kvm, etc), and the word "Ubuntu", and you are sure to find some pertinent, accurate, informative documentation.
In practice...

When I first joined the Ubuntu Server Team in 2008, I found myself needing to read hundreds of manpages (text manuals for the command line). In some cases, I did not have the needed manpages on my local system. In other cases, I needed a generic way of search for multiple terms in all manpages. And in general, I prefer reading documentation in a web browser rather than a terminal. Thus, I created http://manpages.ubuntu.com/, which is a web application updated nightly, generating HTML versions of all manpages in the Ubuntu distribution. This site contains hundreds of thousands of pages, all indexed by Google (and thus searchable), marked up using HTML for readability, cross-linked for easy traversing through manpage-to-manpage references, and able to render a printer-friendly PDF of any given page. I continue to maintain the source code that generates manpages.ubuntu.com (which is also Free Software), and the hosting of the content is courtesy of Canonical.

3. Communication

On a daily basis, there are several hundred of Ubuntu developers and community members distributed across hundreds of IRC channels, mailing lists, and web forums. These are open communication channels whereby we develop Free Software and support its millions of users. We respond to messages in real-time, helping individual users with their issues. All of these media are archived and published, and the Internet has a very long memory. Quite often the most important service performed is not the assistance of the single user asking the question. Rather, it's the scores of others who search the web for that same question weeks, months, or years later, find the archive, and solve their own problem without needing to register for or learn the nuances of IRC/lists/forums.

Blogs and Launchpad bug reports can serve these same purposes. Any time a person contacts me privately to discuss a bug or a problem, I always kindly request that we move the conversation to a public forum (usually a launchpad.net bug report) so that others might learn from our discussion. For frequently-asked-questions, I tend to use my blog, so that I can update the response at a later date (should things change), and always refer future questions to the same URL.

In practice...

I developed and maintain a feature in eCryptfs which allows users to encrypt their home directories. Recovery of such encrypted data can be slightly complicated, if you know nothing about data encryption. Many new users have struggled with this, and many have been saved by: http://blog.dustinkirkland.com/2009/03/mounting-your-encrypted-home-from.html. The latest comment, AUGUST 23, 2010 8:46 PM: "scratchr said...THANK YOU THIS SAVES MY BUTT!!!!!!". This is but one of a couple hundred posts that I have syndicated to planet.ubuntu.com in about 2 years.

4. Conferences

Twice-a-year we hold an open developer summit, where we brainstorm our next release. These sessions are broadcast over the Internet, and anyone in the world is welcome to attend, and have their opinions heard. The open communications and friendly nature of our developer summit encourages collaboration and participation across many facets of the Free Software sphere.

External to Ubuntu, we also travel to countless tradeshows, summits, festivals, conferences, and gatherings of software technical and business people. Quite often, we are introducing people to the concept and practice of Free Software. Other times, we are collaborating with other Free Software developers, helping further the over all cause.

In practice...

Over the past 3 years, Canonical has sponsored (all or in part) my attendance to, and I have taken time out of my personal and work schedule to actively participate in multiple Ubuntu Developer summits, LinuxCons, Linux Plumbers, Linux Conf AU, CloudCamps, Texas Linux Fest, et al. Many of these were sponsored by Canonical, helping the conference pay for itself.

5. Upstream Contributions

Ubuntu is a "distribution" -- a collection of the best of open source software. You may think of Ubuntu as a grocery store. We seek out the best fruits and vegetables, and the best canned, bottled, and boxed products we can find, and make them available to you. In some cases, we do the canning/bottling/boxing (aka, packaging) for you. In other cases, we make available the products of our distributors (Debian, typically). You, the shopper at the Ubuntu grocery mart, have hopefully come to trust the quality of goods we provide, and trust your family and friends to use our packages too.

Sometimes the items we receive from our upstream have flaws that we can detect and fix. It is quite easy for us to fix those issues, and make the improved packages available to our users. It takes a bit more effort to send those fixes upstream, to the people we received the subject code from. And although it does take effort (tremendous effort, in some cases), it's a very important part of being a good citizen in our community. Communicating with upstream is a critically important part of my job, and one that I take very seriously, and in which I take much pride.

In practice...

Update-motd is a package that I initially created to allow for Ubuntu Server administrators to dynamically create the Message-of-the-Day, /etc/motd, by placing scripts in a specific directory, /etc/update-motd.d. These scripts would run at login (or at a specified interval), and command-line users would see the latest and greatest information at each login. This program stood alone for a while during its experimental phases, and I eventually ported it to C, generating a patch against PAM itself. Now that the functionality has stabilized, it is included in Ubuntu's upstream, Debian's, PAM package.

6. Being an Upstream

Returning to that grocery store analogy, sometimes the Ubuntu Supermarket lacks a particular "product" that either our customers ask for, or that we think our customers might want. In such cases, we may "become" an upstream and write some new software to serve our purposes and provide new features to Ubuntu users. Some of these projects are essential to improving the Free Software end user experience. A few that come to mind immediately: command-not-found, apport, and upstart.

Ubuntu as a whole, though, is an Upstream to various derivative distributions, such as Kubuntu, Xubuntu, Mint, and TurnKey Linux cloud appliances. We are furthering Free Software by producing a rock solid base that other people can use to build innovative derivatives that serve unique purposes and reach all sorts of interesting people.

"Being an upstream" maintainer is perhaps my favorite part of my job at Canonical, working on Ubuntu. We have an amazing suite of utilities (Launchpad, Bazaar, PPAs, and Ubuntu itself) for making the upstream experience simply outstanding. We have both a corporate and a community culture that strongly encourages the development of GPLv3 (and AGPL) Free Software. The software we develop and maintain as upstream is tightly integrated into Ubuntu, but it is freely available for inclusion in any other distribution as well.

In practice...

I have initiated several upstream projects, to fill gaps that we perceived in our Ubuntu Server distribution. These are all 100% Free Software, developed and maintained in Launchpad/Bazaar, included in Ubuntu, but sometimes included in other distributions as well. Byobu is an elegant configuration and usability layer on GNU Screen, which makes it quite functional as a text-based window manager. TestDrive is a handy utility that synchronizes daily ISOs (from Ubuntu or other distributions) and launches them in a virtual machine (KVM, VirtualBox, or Parallels) for your perusal -- without having to know anything about virtualization. PowerNap operates much like a desktop screen saver for command-line-only servers, bringing systems down to a lower power state when underutilized. eCryptfs is a cryptographic filesystem in the Linux kernel; I maintain the user space tools, which have been enhanced to support per-user encrypted home directories. This is a feature that was pioneered and funded by Canonical, and contributed directly to the upstream project. For fun, and in my spare time, I spend many hours working as an upstream developer of Musica, Pictor, Screenbin, BogoSec, Lynx-web-app, as well as Byobu, PowerNap, and TestDrive.

7. Bringing Free Software to the Masses

By far, the most important contribution I make to the Free Software world is doing my little part within Canonical and Ubuntu that makes Free Software available to the entire world. From Canonical's partnerships with OEMs such as Dell for desktops/laptops/netbooks, to our Cloud server images available in Amazon's EC2, we are making Free Software available to people who have never used Free Software before. Call us a "match maker" or "middle man" if you will, but we are undoubtedly helping connect the dots, and plug many new people into Free Software.

In practice...

Believe me, I had tried for nearly 10 years (1998-2007) to convert my wife, mom, dad, sisters, friends, and extended family to join me in replacing Microsoft Windows with Linux and Free Software. In 2005, I managed to convert my lovely, non-technical wife, Kim, to Fedora -- but I failed on every other account. I'm ever so proud to say that all of the above (wife, mom, dad, siblings, inlaws, and friends) are proud users of Linux and Free Software today, August 24, 2010. And Ubuntu is the reason why. Ubuntu is Linux for Human Beings. Their wireless just works. Their printing just works. They can play music and watch videos and browse the web. They can work on documents, spreadsheets, and presentations. When they need a new program, they can find it and install it. They can upgrade across releases all by themselves. They can plug in peripherals and video conference with their kids and grandkids. All of this was possible with Linux and Free Software in 2004, for an subject matter expert like me (or perhaps you). None of this was possible for my non-technical relatives before Ubuntu. THIS is what we do, and THIS is why Ubuntu is important to the Free Software ecosystem. We are mainstreaming Linux and Free Software. This is a dream thousands of us have had for 20+ years, and it is something that we have been promising (promised?) for that same length of time. It has been co-joined with "Open Source", it has experienced some individual successes on Servers, and with particular projects (Firefox, OpenOffice, Apache), but never before has Free Software opened Desktops by the millions -- yes MILLIONS -- as it has with Ubuntu as a distribution.

I am helping that happen in very tiny ways, but I'm very, very, VERY proud of it.

:-Dustin

And Now, For Something Completely Different...

2010-08-26T21:26:00.000-07:00

My buddy Nate showed me this website today, XtraNormal.com. You can register for an account and direct your own animated movies. Write the script, cast the actors, choose the set, select the score, direct the camera angles, expressions, animations. It worked well enough for me in Chromium in Maverick.

Check out my first video below ... :-)

:-Dustin

Buy 1, get 8 Free! Or, a Useful Awk Hack...

2010-08-26T14:37:00.000-07:00

But first, a couple of updates...

If you appreciated my errno post, you might be happy to know that /usr/bin/errno is now provided by the ubuntu-dev-tools package in Maverick. This probably won't be the permanent home for the utility (currently in discussions with Joey Hess about putting it in moreutils, and the kernel team about putting it in linux-tools). But while we debate among ourselves about the permanent location of the bike shed, you -- our Ubuntu users -- are welcome to go about using the tool.
And if you liked my bash alert post, you might also be happy to know that the alert alias is also in Maverick's skeleton .bashrc. This only affects new Maverick installs, and you'll have to install the libnotify-bin package, but still, it's a useful tool, and a good start. Several people have suggested UDS-Natty sessions on how to make the tool more universally useful, and it would be cool to see those move forward in the next cycle.

Okay, now for another fun set of tools!

All too often, I found myself typing something like this:

... | awk -F":" '{print $3}'

That's a lot of wasted characters, and it's easy to flub them up.

And so I generalized it, with this 2-line shell script in /home/kirkland/bin/1:

#!/bin/sh

[ -n "$1" ] && ifs="-F\"$1"\" || ifs=

eval awk $ifs "'{print \$$(basename $0)}'" /dev/stdin

And then I made symlinks to this one script, for numbers 2-9:

cd $HOME/bin

for i in $(seq 2 9); do

ln -s 1 $i

done

And now I can:

errno "" | 1

euca-describe-instances | 2

cat /etc/passwd | 7 :

I started out with just simple aliases for 1..9, but I found myself often needing to change the input field separator (IFS). In case you'd prefer the simple aliases, you can use these:

for i in $(seq 1 9); do

alias $i="awk '{print \$$i}'"

done

What do you think?

Would you like to see these as /usr/bin/1 ... /usr/bin/9 in Ubuntu Natty?
Do you have a more efficient implementation?

:-Dustin

p.s. My wife, Kim, took both of these pictures recently, on the side of the road in Michigan and New York, respectively. Pretty birds, hard to photograph! :-)

A Glimpse of Ubuntu Desktops in the Financial World

2010-08-21T08:59:00.000-07:00

I worked from Chicago today, meeting up with my buddy Kees and doing my daily hacking from a Caribou Coffee shop with free WiFi on the river and at the foot of the Willis Tower (previously known as the Sears Tower).

Earlier this year, I had the great privilege to visit the Weta Digital Studios in Wellington, New Zealand, where their Ubuntu-based cluster of 35,000 nodes renders blockbuster movies, including Avatar and King Kong.

And today I visited another place, as absolutely amazing as Weta Digital, in terms of what they're doing with Ubuntu... After our work was done, Kees took me to meet one of his friends, a programmer and sys-admin at a financial firm near the Chicago Board of Trade (now merged with the Chicago Mercantile Exchange).

These guys easily have 35,000 square inches of LCD monitors running Ubuntu desktops, displaying in real time thousands of graphs, metrics, monitors, and statuses. Hundreds of multi-head desktops running 8.04 to 10.04, attached to 17" to 42" Samsung LCDs, Ubuntu logos everywhere I turned!

There is no doubt that across both Server and Desktop, Ubuntu is proving itself in enterprise environments. Linux is here, there, everywhere, and Ubuntu is a very important player, helping make that happen. I take great pride in what we're achieving together!

:-Dustin

p.s. Some readers have followed my hiking and travel adventures to Scotland (both times), New Zealand, and others. Many of these adventures have little (if anything) to do with Linux or Ubuntu, so I've moved all of my travel to a different blog, theKirklands.net, which is not syndicated on planet.ubuntu.com. You are welcome to follow my travel stories there. If you do, you have a lot of catching-up to do! ;-)

Count Me In!

2010-08-13T13:49:00.001-07:00

There's a bit of buzz right now, about the canonical-census package -- a package that only exists in the Canonical Partner archive. Please read Rick Spencer's post for clarification on what the package is and what the package is not (I don't want to further explain it here).

Some Ubuntu users/members/developers have voiced opinions against this idea.

I am all for being counted as an Ubuntu user! I am a proud user of free and open source software, and I want everyone to know it. I'm happy for each of my dozens of Ubuntu desktops, notebooks, servers, and virtual machines to be anonymously counted!

I particularly like that this package is called a "census".

It makes me think of people around the world who are proud to be counted. Do you remember the "purple fingers" of so many thousands of Iraqi people after the first free election in years? I am as proud of my purple desktop as those people were of their purple fingers. Count me in ;-)

:-Dustin

Ubuntu Server Team Meeting Minutes from 2010-07-27

2010-08-02T21:09:00.000-07:00

See https://wiki.ubuntu.com/MeetingLogs/Server/20100727 for full log and programmatic interpretation of the moin markup. Cross-posted here because I'm required to do so. But seriously, give your eyes a break and go read them from the wiki instead :-)

Minutes

Action Points Review with kirkland

hggdh to discuss the outcome of server-maverick-qa-workflow in ServerMeeting during the beta cycle
sommer/smoser in sync on cloud-init documentation, action DONE
[ACTION] SpamapS to work with mathiaz on a proposal, and send proposal to -devel on ruby gems in ubuntu

Maverick Developments with jiboumans

we are a bit behind on Alpha3; somewhat expected; only low priority specs affected so far

QA Q&A with hggdh

hggdh has filed a few (4) bugs against eucalyptus-2.0

Kernel Chit Chat with jjohansen

jjohansen has a pv-ops kernel in testing; apparmor fixes queued and should be uploaded; lucid high load average bug may not be an illusion

Doc Talk with sommer

uec section updated from the wiki docs in http://doc.ubuntu.com/ubuntu/serverguide/C/uec.html
[ACTION] Daviey & kirkland to review http://doc.ubuntu.com/ubuntu/serverguide/C/uec.html

SRU Fun with zul

10.04.1 is coming up; need to know about SRU bugs ASAP

Bloody Papercuts with ttx

not in good shape; papercuts need to be fixed by end of the week to make Alpha3
[ACTION] entire team to fix your papercuts by end-of-the-week

Triage Backlog with ttx

regular triage work slipped last week; likely culprits: pilsner and absynthe; everyone to help out clearing the queue

Open Discussion

kim0 to deploy a web app that will let people show where in the world they are running an Ubuntu server
roaksoax says that we should have an HA cluster in Main for Maverick
[ACTION] SpamapS and ScottK to reivew Kolab php5 patches

Next meeting will be on Tuesday, April 14th at 15:00 UTC in #ubuntu-meeting.

:-Dustin

Celebrating the Big 3-0

2010-08-02T19:33:00.000-07:00

Unfortunately, this post isn't about my 30th birthday. Nope, I turn 31 later this month :-(

But it is about a significant milestone: the Byobu 3.0 release!

For over a year now, a growing number of people have been asking for a Byobu tarball that is installable on all sorts of non-Ubuntu/non-Debian/non-RedHat UNIX and Linux systems. Some others just want the ability to install Byobu locally on perhaps a Debian/Ubuntu system where they don't have root access but want Byobu goodness.

For 9+ months, I've been avoiding autohell. I wrote byobu-export to create a tarball that could be exported and uncompressed directly in your $HOME directory anywhere that you can run screen. But some people want to relocate those binaries. And the number of requests for a "./configure && make && make install" style build system seem have spiked recently (most notably from OS X users).

Finally, I caved. I took a holiday from real work last Thursday and spent the day hacking together an autoconf/automake build system into Byobu. I used this grumpy-though-informative tutorial. It got me most of the way there. I had to change a lot of Byobu's code, to make it actually relocatable via the ./configure --prefix option.

The result? You should now be able to:

Download the latest tarball from the Launchpad release page
Uncompress and cd
Run ./configure --prefix=$HOME/somewhere && make && make install
Update your PATH with export PATH=$PATH:$HOME/somewhere/bin
And then just run byobu

And you should be able to do this almost anywhere that has screen installed! To use the graphical F9 menu, you'll also need python-newt, but without python-newt, you can just edit the text files in ~/.byobu by hand.

This was no small amount of work, and I'm quite proud of it :-) Proud enough to bump the major version, from 2.x to 3.x! I've subsequently uploaded it to the backports PPA and pushed to Maverick. Please file any issues or regressions here.

I'm sure I've just opened the floodgates for all sorts of wacky bugs about status notifications that do not work in obscure UNIX flavors... As they say here in Montreal, "C'est la vie!"

Cheers,

:-Dustin

TestDrive, then and now

2010-07-29T06:09:00.000-07:00

Less than a year ago, I introduced TestDrive, a convenient way of incrementally syncing ISOs from ubuntu.com, and running them in KVM, VirtualBox, or Parallels. It's a great way to do your ISO-testing, or just keep up with the Ubuntu release under development. I created the project for a couple of reasons:

To make it even easier to run Ubuntu in a virtual machine
To show off KVM (without the overhead of libvirt or virt-manager)
To make the daily development releases of Ubuntu safely usable by non-technical Ubuntu enthusiasts (who inevitably show up the week before release saying that they just tried the Beta or RC for the first time, and "by golly -- stop the release -- it's not ready!!!"

Well, TestDrive has come a long way -- mostly thanks to Andres Rodriguez, my Google Summer of Code student who has put a nifty GTK front end on TestDrive!

If you haven't tried the TestDrive GUI yet, please check it out. You should be able to install it directly from Maverick, or using the backported packages for Lucid from the PPA. With one command (or, now, one graphical button click), you can have any Ubuntu ISO up and running in a virtual machine!

Please let us know what you think! You can leave opinions here in the blog comments, but please file bugs against the project in Launchpad.

Also, I'd like to add a "file handler" in Gnome, such that you can right-click on an ISO, and select "Open with TestDrive". Can anyone tell me how to do this?

Cheers!

:-Dustin

Does the gild go on first, with the gouache applied afterward?

2010-07-28T04:04:00.000-07:00

I get questions like this from time to time, so I figured I'd respond here...

Someone wrote:

> Hey Dustin,

> I'm a student trying to do some experiments with Byobu Japanese Screens. It's almost

> impossible to find anything on the web that tells me what the atavistic process is. Eg, does

> the Gild go on first, is the gouache applied after , if so what kind? etc.

> I don't know if I've completely mis-understood the essence of what I've read on the pages

> below, but does this link take me to a program that helps design BYOBU, or is it just the

> name you are using?

> Totally confused!...Look forward to hearing from you.

Howdy!

Thanks for the note. I am afraid that the Byobu program does not help design the beautiful Japanese folding screens, or anything like that. I chose the name because:

the old name was boring
there were not many search hits for the word "byobu" (as you found out!)
it rhymes with GNU, and Ubuntu
it is a thought-provoking analogy for what the program is

To understand that analogy, you need a bit of computer history.

GNU Screen is a revered UNIX/Linux program that's been around for ~25 years. It actually predates Microsoft Windows as many of us first knew it. GNU Screen is a text-based program that enables you to run multiple computer programs at the same time.

Today, with modern operating systems, such as Windows, Ubuntu, or Mac OS, you just open up each program in a new "window" (which is another analogy!). But before computers were graphical, GNU Screen provided an innovative way of running multiple programs at the same time.

Many UNIX/Linux users have continued to use the GNU Screen program over the years, but the very plain interface to it, and configuration of it hasn't changed much in that time, and the complexity blocks some new users from utilizing it.

In December of 2008, I wrote a new interface that uses the old GNU Screen program under the covers, but has a more useful interface and easier configuration, making it accessible to more users. The first name of the project was pretty boring -- Screen-profiles. But in May of 2009, it got its current name, Byobu.

And so the analogy...

In the same way that Japanese byobu are detailed, elegant, beautiful enhancements of ordinary folding screens, the Byobu program is a detailed refinement of the plain old GNU Screen program!

Screen	Byobu

:-Dustin

Cogito Errno Sum

2010-07-26T19:19:00.000-07:00

Ah, POSIX error codes. When systems programming, do you ever find yourself with an unfamiliar error code, and then you're off Googling what the heck it actually means? Where's the command line utility to just do this for you, right where you're at, in a terminal? It's actually pretty easy, with a glorified grep...

Try this script, which I store in ~/bin/errno:

#!/bin/sh -e
headers="/usr/include/asm-generic/errno*.h"
code="$1"
if echo "$code" | grep -qs "[0-9]"; then
grep -hw "\W$code\W" $headers | sed 's/^#define\s*//'
else
grep -hi "$code" $headers | sed 's/^#define\s*//'
fi

Now, you can run:

errno 36

ENAMETOOLONG 36 /* File name too long */

Or:

errno EEXIST

EEXIST 17 /* File exists */

Is this useful to you? It has been really useful to me any time I'm doing system level programming or debugging.

I have long wanted to drop this little script in /usr/bin/errno, but I haven't found the right package to own something like this. Maybe a kernel package, since it uses the kernel headers? Opinions? Let me hear them!

:-Dustin

I think, an error number, I am :-)

Dear Bash, please ping me when you're done running $FOO

2010-07-26T04:51:00.000-07:00

UPDATE: The "alert" alias has landed in Maverick's /etc/skel/.bashrc ... thanks for your support!!!

How often do you run a command line tool on your desktop that takes a really long time? Maybe something like make, debuild, rsync, or wget?

You probably kick off the long running job, and then alt-tab over to something more captivating than watching gcc fill your scroll back buffer -- maybe your web browser or news reader.

You occasionally pop back over to your shell to check on your job. Maybe it's still running. But maybe it finished a while ago. Dang.

No need to beat yourself up over wasted cycles. You can tell your shell to ping you when it's done. Just add this alias to your ~/.bashrc.



alias alert='notify-send --urgency=low -i "$([ $? = 0 ] \
  && echo terminal || echo error)" "$(history|tail -n1| \
  sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

Install the notify-send utility, and source your new ~/.bashrc:


sudo apt-get install libnotify-bin
. ~/.bashrc

Now, run a long running job, and append "; alert" to the end of the command, like this:


sleep 20; alert

After running the target command, the alert alias will render a notify-osd pop-up on your desktop, telling you the command you just ran, and its exit code.

Nifty, huh?

:-Dustin

Man your Browser!

2010-07-13T13:39:00.000-07:00

I've been spending more and more time on smaller laptops and netbooks. I recently gave the Chromium web browser a try, and I must say that as of Ubuntu 10.04 LTS, Chromium has totally won my heart! It's quite efficient, snappy, and user-friendly.

It took me a little while to learn some of the ins and outs, but I think I've replaced all of the functionality I needed from Firefox.

The most useful feature I've found are Chromium's custom "search engines". Here, I'll demonstrate how to add a custom manpages.ubuntu.com to Chromium's search engines, such that you can type something like "man kvm" in your browser's URL bar, and go straight to the HTML rendering of the kvm manpage!

Basically:

Click on the wrench in the upper right corner
Select Options
Click Manage near the Default Search option
Click Add
Name: Manpages
Keyword: man
URL: http://manpages.ubuntu.com/%s
Click Add
In the browser, ctrl-L to goto the URL bar
Enter man kvm

And you win!

I actually use several of these every single day...

man
```
http://manpages.ubuntu.com/%s
```

http://linuxsearch.org/?&sa=Search&cof=FORID:9&cx=003883529982892832976:t4dsysmshjs&q=%s

bug
```
https://launchpad.net/bugs/%s
```
pkg
```
http://launchpad.net/ubuntu/+source/%s
```

:-Dustin

LinuxJournal 2010 Readers' Choice Survey

2010-07-07T14:25:00.000-07:00

LinuxJournal has opened their 2010 Readers' Choice survey!

Ubuntu, and Ubuntu-based projects are prominently featured in quite a number of survey questions. Whether you like what we're doing or not, LJ's survey is an excellent year-over-year monitor of what's hot, and what's not in the Linux world.

There is a free-response question toward the very end of the survey asking for your favorite new open source project from 2009 or 2010. The Ubuntu community has generated a vast number of excellent new projects in 2009 and 2010. I certainly encourage you to help support one of your favorite projects in that question.

My favorite new project from 2009-2010? Byobu :-) Toss a vote our way, if you think we're doing some cool stuff, and you'd like to support our project!

:-Dustin

Keeping Pictures from Multiple Cameras in Temporal Order

2010-07-02T20:46:00.001-07:00

I use as many as 4 different cameras, each serving a slightly different purpose.

In order decreasing order of portability, and increasing order of photo quality:

Pocket -- a low-quality, but always present camera phone (either my Palm Pre or my G1)
Small -- Canon Powershot SD1000
Medium -- Kodak z812 IS
Large -- Canon EOS Digital Rebel XTi

When I'm traveling, I might take 2 or more of these cameras, and shoot pictures all day long on different cameras. Sometimes, I'm shooting with one camera, and Kim is shooting with the other. At the end of the day, I want to merge all of these pictures, and look at them in order.

To keep this straight, I take great care to ensure that the embedded clocks in all of these cameras are perfectly in sync.

And when I'm processing my pictures, I often use gthumb to rename my pictures, embedding the timestamp in the name.

Other times, I'll write a small script to do this for me.

Earlier today, as I was stitching my pictures together from 3 different cameras, I realized that one of the clocks was off by about a day. I used this little script to fix the timestamps:


#!/usr/bin/php
<?php
$dh = opendir(".");
while (($file = readdir($dh)) !== false) {
   if (preg_match("/^IMG_/", $file)) {
           $s = stat($file);
           touch($file, $s[9] - 86400);
   }
}
?>

After I did that, I also needed to update the EXIF data embedded in the JPG too. To do that, I used jhead.


jhead -dsft IMG_*

Of course, running jhead on these files modified them yet again, so the file's timestamps are screwed up again. Let's cover our tracks.


#!/usr/bin/php
<?php
$dh = opendir(".");
while (($file = readdir($dh)) !== false) {
        if (preg_match("/^IMG_/", $file)) {
                $exif_data = exif_read_data($file);
                touch($file, strtotime($exif_data['DateTime']));
        }
}
?>

Enjoy!

:-Dustin

Creating Ubuntu Server Disk Images using live-helper

2010-07-02T16:31:00.000-07:00

I recently showed you how to create Ubuntu Server images using vmbuilder, but promised to also show how to use live-helper. Without further ado...

Start by installing live-helper.

sudo apt-get install live-helper

If you want a splash image for the boot loader, grab one. Here's an example

wget https://launchpadlibrarian.net/39290722/byobu_192.png

Next, we use lh_config with a lengthy set of options, to build:

disable apt recommends (keep the image small)
build 64 bit (for large memory instances)
enable main and universe
build a usb disk style image
append a few kernel options (like "toram" and "noprompt")
use the syslinux bootloader
build from ubuntu lucid
use the linux-image-server kernel
add a few packages (like "byobu" and "vim")
add your boot splash image
set the boot timeout to 2 seconds
use aufs
set the root disk size to 2GB

/usr/share/live-helper/helpers/lh_config \
--apt-recommends false \
--architecture amd64 \
--archive-areas "main universe" \
--binary-image usb-hdd \
--bootappend-live "toram persistent noprompt" \
--bootloader syslinux \
--distribution lucid \
--exposed-root true \
--initramfs casper \
--linux-flavours server \
--linux-packages linux-image \
--mode ubuntu \
--packages "byobu vim" \
--syslinux-timeout 2 \
--syslinux-splash ./byobu_192.png \
--union-filesystem aufs \
--virtual-root-size 2000

Configure any "local hooks". These are scripts of your design that can modify your image inside of the chroot.


cat >config/chroot_local-hooks/01-byobu.sh <<EOF
#!/bin/sh
ln -sf /usr/bin/byobu-launch /etc/profile.d/Z98-byobu.sh
EOF
chmod +x config/chroot_local-hooks/01-byobu.sh

Now, you can perform the build. This takes about 8 minutes for me on my laptop, with an SSD and local mirror.


sudo /usr/share/live-helper/helpers/lh_build

Finally, test your image. You can run it in KVM.


kvm -m 512 -hda binary.img

Alternatively, you could write this directly to a hard drive, as an install. Note that this will overwrite ALL data on that drive. Use with great care!!!


dd if=binary.img of=/dev/sda

Enjoy!
:-Dustin

An eCryptfs Backup Strategy

2010-06-28T15:05:00.000-07:00

Disclaimer: I am often asked about best practices regarding eCryptfs backups. I am not necessarily advocating this as the best approach; rather it this is simply my approach. Do with it what you will ;-)

I generally perform two types of backups...

Backups to Trusted, typically Local Storage (~hourly)
Backups to Untrusted, typically Remote Storage (~daily)

For me, trusted local storage generally means hardware that I own the physical control of, and that I am the only person with immediate root access. This might be a system in my home or office, or even static media locked in a safe deposit box at the bank -- understanding of course that I must trust the physical controls in place. If I don't trust the physical controls, then it's not trusted local storage. My laptop, since I often travel with it, is not trusted local storage, since there's a fair possibility that it might be stolen.

And for me, untrusted remote storage generally means a reasonably secure system, but one that I do not have physical control over and on which I may not be the (only) root user. This includes co-lo's and various forms of web and cloud storage (such as Amazon S3).

I will keep backup copies of my cleartext data on trusted local storage. For me, this means an hourly cronjob that does something like this on the LAN:

rsync -aP /home/$USER/ \

trusted.local.storage:/var/backups/home/$USER/

For untrusted remote storage, I never send my cleartext data, but rather my encrypted private data for backup. And since it's usually over a WAN, I use a daily cronjob that does something like:

rsync -azP $HOME/.Private/ \

untrusted.remote.storage:/var/backups/home/$USER/.Private/

And in both cases, I will periodically (once a month?) run rsync with --delete and --dry-run by hand, check the diff, and then re-run with --delete if I'm satisfied with the results. Do this with care ;-)

This may or may not be ideal for you, and some of you probably have even better ideas! Please feel free to leave a comment if you'd like to share your best practices for backing up your eCryptfs data.

:-Dustin

photo © MIROSLAV VAJDIÄ from openphoto.net CC:Attribution-ShareAlike

TurnKey Linux Beta Launches Byobu by Default at Login

2010-06-27T10:15:00.000-07:00

TurnKey Linux is a Canonical Cloud Partner that provides Ubuntu-based Virtual Machine appliances. Alon Swartz of TurnKey recently announced the release of TurnKey Core Beta, re-based on Ubuntu 10.04 LTS, and should release an official version shortly after Ubuntu 10.04.1 is generally available.

TurnKey has a real focus on user-friendliness, excellent defaults, and the user's first-run experience. Thus, I was excited to see this bullet in his TurnKey Core Lucid Beta announcement:

User friendly screen wrapper (byobu) launched by default on login.

I think this is a great idea, as I have proposed as much at the last couple of Ubuntu Developer Summits. Ciemon Dunville has filed a bug, suggesting that the Maverick Ubuntu Server install should also default to launching Byobu. The output of the latest UDS session was that "we might consider doing this, if enough Ubuntu server users are asking us for it."

If you think this is a good idea, please mark that bug as "affecting you too", and leave a comment! And if you think it's not quite ready yet, due to some particular bug or annoyance, please file a new bug -- I follow these very closely in my spare time, as I'm quite passionate about this project, and I work very hard to ensure that Byobu is high quality software, with happy users ;-)

:-Dustin

Creating Ubuntu Server Disk Images using vmbuilder

2010-06-27T07:02:00.000-07:00

From time to time, I need to precisely replicate an Ubuntu Server installation over and over again. If my testing requires turning the installation crank, preseeding can certainly help automate that. But Ubuntu Server installations on ext4 have slowed down dramatically due to an ext4/dpkg/fsync bug.

In any case, there are at least 2 different ways of creating bootable Ubuntu Server disk images, which can be written directly to a hard drive using dd.

vmbuilder
live-helper

This post shows how to do so using vmbuilder. I'll explore live-helper in a subsequent post (as I'm just now learning that utility).

Create a minimal image (without recommended packages), ~20min:

sudo vmbuilder kvm ubuntu --arch 'amd64'  --rootsize '4096' \
--kernel-flavour 'server'  --components 'main,universe' \
--addpkg eucalyptus-nc  --user 'ubuntu'  --pass 'ubuntu' \
-v --debug

N.B. The most important option above, for the sake of this tutorial, is --kernel-flavour 'server'. By default, vmbuilder includes the linux-image-virtual kernel, which lacks some drivers necessary for booting and running from real hardware. vmbuilder supports many options which are not documented in the manpage:
vmbuilder kvm ubuntu --help

Convert the qcow2 to raw, ~4sec:

qemu-img convert -O raw *.qcow2 disk.raw

Optionally, compress the image for transfer over the network, ~90sec:

lzma disk.raw

Finally, decompress it and write it directly to disk. For this step, I often use an Ubuntu Desktop liveCD, and change to the root user.

sudo -s
lzma -dc disk.raw.lzma > /dev/sda

And now, you should be able to boot and run from /dev/sda.

:-Dustin

Ubuntu Server BoF at Velocity

2010-06-17T06:53:00.000-07:00

If you’re going to Velocity 2010 next week, I hope you’ll join some of my team members on the Ubuntu Server team at the Ubuntu Server Users and Developers BoF, Tuesday June 22 @ 8:30pm.

See the full announcement at:

http://ubuntuserver.wordpress.com/2010/06/17/ubuntu-server-bof-velocity-2010/

:-Dustin

Cloud in your Pocket -- UEC LiveISO!

2010-06-14T20:00:00.000-07:00

Background

At UDS in Belgium, I gave a 5-minute Lightning Talk during Friday's plenary. In that talk I gave a demonstration of a UEC LiveISO, based on Ubuntu 10.04 LTS. You can hear the 5 minute talk starting at the 29m30s mark of this audio file.

And you can find the 823MB ISO I used for my demonstration here:

http://people.canonical.com/~kirkland/ubuntu-10.04-ueclive-amd64-custom.iso

I took the 10.04 LTS Desktop, and modified that ISO according to the instructions at:

https://help.ubuntu.com/community/LiveCDCustomization

I removed a ton of unnecessary packages for cloud computing, such as OpenOffice, Translations, and the Games, Graphics, Sound & Video applications. I also added the necessary Eucalyptus applications, and provided some initial configuration. I also pre-loaded a small, ttylinux image in the ISO itself (username=root, password=linux).

Motivation

Why would anyone want to do this? Well, for many of the same reasons Linux Desktops landed on LiveCDs. It's useful for testing, prototyping, and learning about the environment, before deploying to real hardware.

It used to be amazing that you could carry around a complete, bootable desktop operating system in your pocket. Now, you can carry a whole cloud! What's next? :-)

Caveats

So this LiveISO is completely unsupported right now. This was just a weekend hack that I put together because I thought it might be interesting, and because some people said it couldn't be done ;-)

This LiveISO is not meant to replace the Ubuntu 10.04 LTS Server ISO, from which I recommend you install UEC, if that's your goal.

How to Use this LiveISO

Download the ISO.

If you wan to try this out on real hardware, fire up Startup Disk Creator. Insert USB flash disk, at least 2GB in size (the bigger, the better). In Startup Disk Creator, create a very large persistence file with the slider toward the bottom.

You can now boot this on a system that has a CPU with VT extensions (try running kvm-ok to find out if your hardware supports VT), and with at least 4GB of memory. Sorry about the memory requirements -- Java is a memory hog.

Alternatively, you can actually be able to boot this ISO entirely inside of a KVM, if you can give it enough memory.

Basically, I gave this KVM all of my free memory (2300MB), and 2 CPUs, using this command:

kvm -m 2300 -smp 2 -cdrom ubuntu-10.04-ueclive-amd64-custom.iso

A few minutes after booting (~4 minutes), I can see within Byobu that all of the Eucalyptus services are running: CLC,WC,CC,SC,NC.

Once your CLC is running, you can grab your credentials:

sudo euca_conf --get-credentials mycreds.zip
unzip mycreds.zip
ln -s eucarc .eucarc

Now, register the node.

sudo euca_conf --register-nodes 127.0.0.1

And check your cluster's availability.

euca-describe-availability-zones verbose

With a little luck, you should see some free VMs!

Next, register an image. We included a tiny, ttylinux one on the ISO. You can try to run bigger images, but note that it will be rather slow (either you're reading/writing from slow flash media, or you're running in KVM, which means your VM will be running in non-accelerated QEMU).

uec-publish-image -K vmlinuz i386 tty-linux.img foo

And check your image's registration.

euca-describe-images

Now, run your image!

euca-run-instances emi-DEADBEEF --addressing private
watch -n 5 euca-describe-instances

You should that go to "running" eventually. If you're already running inside of KVM, this QEMU emulated virtual machine will be very slow. You should be able to ping it, and you might even eventually be able to ssh to it.

euca-authorize default -P tcp -p 22 -s 0.0.0.0/0
ping 172.19.1.2
ssh root@172.19.1.2

You can also explore the UEC administrative web interface. The LiveISO still has Firefox. You can point it to http://localhost, and login with admin/admin credentials.

So this is all fine and dandy if you want to try out UEC on your own hardware. But that's not very cloud of you... Why don't you just try out UEC in EC2? Do what, huh? Yup, totally doable too. The work described above inspired Scott Moser to publish his own how-to on the matter:

http://ubuntu-smoser.blogspot.com/2010/05/easily-test-or-demo-ubuntu-enterprise.html

So What's Next?

Well, I'm actually trying to improve this a bit during the 10.10 cycle. You can follow our plans here. Basically, I'd like to create a UEC Live ISO seed, and have Ubuntu's cdimage publisher crank one out on a weekly basis. There are a few hiccups around auto-registration (which you might encounter in your testing of the current ISO) -- these should be fixed.

Hopefully you found this article useful, and might use this ISO in your own demonstration, education, and edification around Ubuntu's Enterprise Cloud!

:-Dustin

TestDrive GTK Frontend Underway

2010-06-14T15:35:00.000-07:00

I'm mentoring Andres Rodriguez for his Google Summer of Code project, for which he's putting a GTK graphical frontend on TestDrive.

It's coming along quite well. Check out the screenshots at Andres' blog, and follow its development at:

http://www.roaksoax.com/2010/06/gsoc-update-of-the-week-testdrive-pygtk-front-end-3

:-Dustin

Reading Habits

2010-06-13T19:08:00.000-07:00

Check out Matt Zimmerman's really interesting post on his reading habits. I started responding in his comment's section, and then realized I had written 5 paragraphs. So I popped out of there, and over here in my blog for a response.

I've tried both, and I strongly prefer reading for 4-6 hour blocks at a time, rather than 45 minutes a night. I find my retention and enjoyment is far better once I get into a groove. My startup/shutdown time when reading a few chapters per night seems to put a significant dent in my retention and comprehension. Also, my books don't run out of batteries (ie, I don't use a Kindle etc. yet). And I don't have to shut them down for the first and last 30 minutes of a flight. So most of the book reading I do tends to be in airports and on airplanes. Fortunately, I fly more than the average person. But unfortunately, I don't spend much other time reading these days.

Also, for the past 10 years, I have religiously followed a tradition of always alternating between fiction and non-fiction. Previously, I found myself stuck in a rut, reading a dozen Neal Stephenson, or Steven Levy books in a row. Alternating seems to be healthier for me and rounds out my knowledge a bit better.

As for my daily workflow, I usually start by reading my personal email (Gmail) -- maybe 10 minutes over coffee. I follow that by checking any lingering IRC messages -- 5 minutes or so. And then 15-30 minutes of work email (Canonical/Evolution). And then I tackle a few hundred various filtered emails and mailing lists -- this can take another 10-60+ minutes. I tend to check my newsreader (Liferea) twice a day, once over lunch, and again just before bed -- I can easily spend 60+ minutes reading blogs and newsfeeds. I don't really use Buzz/Twitter/Identica/Facebook, except to spread my blog post (is that wrong?), and I don't really read anyone else's Buzz/Twitter/Identica/Facebook (is that wrong too?).

In any case, I agree with Matt's assessment. Various forms of e-reading has cut into my book-reading time, for better or worse. Fortunately, I have plenty of travel scheduled for the foreseeable future ;-)

:-Dustin

p.s. Anyone remember Reading Rainbow? It's okay to admit you started humming the theme song when you saw that logo :-)

Manual Override of Byobu-auto-launch

2010-06-13T09:36:00.000-07:00

I launch Byobu by default on login on all of my Ubuntu machines. It's just such a rich, useful command line environment, compared to the basic shell.

Occasionally, I need to ssh into a remote machine and not launch Byobu. Sometimes I need to pivot from that machine to another on the same network. Sometimes I need to un-fubar an experimental Byobu configuration when I'm developing. And sometimes I'm ssh'ing from a crappy terminal like on my smart phone.

Here's an easy, one-liner to ssh into a remote host that would ordinarily launch Byobu, but not launch Byobu for this one session:

ssh -t remotehost bash

Note the -t option to SSH, which tells it to allocate a psuedo-tty. And note that you're telling SSH to immediately launch a bash shell.

Enjoy!

:-Dustin

A UEC Beginner's Guide

2010-05-17T12:11:00.000-07:00

Murthyraju Manthena dropped me an email last week, pointing me to some excellent documentation (including charts, diagrams, code snippets, and commands) that he and his company put together based on their experiences with the Ubuntu Enterprise Cloud and Eucalyptus:

http://cssoss.wordpress.com/2010/05/10/eucalyptus-beginner%E2%80%99s-guide-%E2%80%93-uec%C2%A0edition/

You might find some benefit from their documentation, supplementing the wiki docs at:

http://help.ubuntu.com/community/UEC

And the upstream documentation at:

http://eucalyptus.com/resources/overview

Cheers,
:-Dustin

Convert a Running Screen Session to Byobu

2010-05-15T12:12:00.000-07:00

I've been asked this question several times, so I figured I'd create a little demo video...

Is it possible to convert a running GNU Screen session to a Byobu session?

Actually, it's quite easy (if you have Byobu installed already)! Just run:

ctrl-a :source /usr/share/byobu/profiles/byoburc

Enjoy,
:-Dustin