Modern Unix Systems

OpenVPN + OpenLDAP

2009-10-11T10:23:00.002-07:00

Jordan's posted a comprehensive setup guide for OpenVPN with OpenLDAP on his blog. Looks like OpenVPN is a great way to set up a simple client VPN solution, and the cost is certainly a lot lower than solutions from Cisco.

Check it out.

OpenSSH 10th Anniversary

2009-10-01T19:56:00.003-07:00

Today marks the 10th Anniversary of the OpenSSH project, a 100% free implementation of the SSH protocol.

SSH is something we all take for granted now and it's hard to imagine life without it. It certainly forms one of the most important tools in any computer system engineer's toolbox, but we rarely give it much thought.

Kudos to the OpenSSH team for working so hard on this vital piece of infrastructure software.

From the makers of StackOverflow and ServerFault: StackExchange

2009-09-28T18:00:00.001-07:00

As you may or may not know, I'm a big fan of both StackOverflow and ServerFault. I think the community moderated question-answer format is great, and I also envisioned many other niches in which this type of software could be used.

It was probably inevitable, but Fog Creek Software is now launching a SaaS service that lets you create your own similar sites, StackExchange:

Pricing starts at $129/mo for 1m page views. Given that each page is essentially a heavily-keyworded bit of content, it should be quite possible to make this kind of money back using targeted ads.

Bcfg2 1.0 pre3 now available

2009-05-22T16:54:00.003-07:00

Version 1.0pre3 was released today with a rather massive overhaul of the codebase. Most notably, over 10,000 lines of code were removed as the dependency on the tlslite library was dropped in favor of using the native Python SSL bindings.

The complete release announcement from the bcfg2-dev mailing list:

In time for the long weekend, we're pleased to announce the availability
of bcfg2-1.0pre3. This release has a number of large improvements over
the previous prerelease. We have switched to the in-python ssl module,
(which is included with 2.6, and available for 2.3-2.5), threaded the
server, improved many of the client tool drivers, added a server
performance profiling interface, greatly improved the Packages plugin
and many other things. (a high-level detailed summary follows) As
always, there are bugfixes and documentation updates as well.

This release has benefited greatly from a number of users who have done
heroic testing of some of the new features; this release includes a lot
of new code, and a lot less old code. We've been able to remove the
python ssl implementation we've been carrying for quite some time.

As always, this release is the result of the efforts of a number of
folks. Problems can be reported here, in trac or on irc.
-nld

Detailed Changes
* Server Core
** Fix Pkgmgr virtual package target binding (Reported by TimL)
** rework File Monitoring code/adapt to new server infrastructure
** Fix updates for VCS plugins
** New server performance interface
*** Provides scalable aggregate performance data for server operations
** Report deprecated plugins, improve reporting for experimental plugins
** Implement support for .genshi bundles in Bundler
** Packages
*** Yum improvements and bugfixes
*** Support for multi-arch yum sources
*** Implement dependency resolver debugging
*** Improve error handling
** SGenshi: improve error handling
** Schema update from TimL (related to the service schema change)
* SSL
** We now use the ssl module included with python2.6 (this has been backported to 2.3-2.5)
** Certificate-based authentication is supported
** Implementation is backward compatible to 0.9.6 clients
** strict cert auth, cert or password, or bootstrap (password once, then cert only) are supported
** Clients now authenticate servers by commonName (not fingerprint)
** Use of certifications require a CA to be used
** The server is now multithreaded

* Tool driver fixes
** APT.Remove: Split up package names properly
** Chkconfig patch from leggett
** Fix RcUpdate driver regressions
** Initial IPS (Opensolaris) driver
** Fix YUMng -r behavior
** Fix portage driver traceback (Resolves Ticket #649)
** YUMng: Fix for RHEL5 (patch from Tim Lazlo)
** YUMng: Fix version=auto for epoch-sensitive packages
** Update RcUpdate tool driver to catch all services
** Remove deprecated RPM and Yum drivers

* Snapshots
** Add extra/bad entry reporting
** Add revision to bcfg2-admin snapshots reports
** Remove ad-hoc error handling in favor of normal bcfg2-admin mode handling
** fix Statistics data location in importer
** minor cosmetic updates

* Other
** Add bash completion for bcfg2-admin
** Fix daemonize exit status
** Fix builds with the redhat specific rpm packaging
** lots of py 2to3 and pylint updates
** Fix py2.4 portability (try/except/finally is 2.5+) (Reported by Lisa Giacchetti)
** Include ignores for Pkgmgr updates (patch from zultron)
** Update bcfg2 manpage for multiple bundles
** bcfg2 client: remove agent support

* Bugfixes
** Fix fam tracebacks for Ticket #650
** Add support for probed groups in bcfg2-admin query (Resolves Ticket #647)
** Display diff in interactive mode (for Ticket #526)
** Fix fd leak caused by our use of the subprocess API
** Fix reversed options (Reported by Kamil Kisiel)
** Logging: Fix reconnect when using /dev/log
** Handle import errors in the help path (Resolves Ticket #653)
** Modify bcfg2-repo-validate to warn on xml duplicates (for Ticket #643)
** Metadata: fix default group assertion
** Fix exit in bcfg2-info

As for me, I've once again made available RPM packages for most major distributions at my openSUSE build service repository.

As of yet, they're functionally untested so it's possible there may be some dependencies missing. Please try to install them and report any problems.

Offline backups are important

2009-05-15T14:07:00.004-07:00

Just came across this article on the BBC about a popular flight simulation site that was hacked. Apparently their only means of backup was to copy the data between their two servers. Unfortunately for them, the "hackers" got in to both servers and destroyed the data. Approximately 13 years worth of work that now cannot be recovered. While I feel sorry for the owners of the site and hope that they can get much of their information back through Archive.org or Google Cache, the whole thing could probably have been prevented with a small investment in to an offline backup strategy.

Don't buy a Netgear GS105!!!

2009-05-07T19:35:00.002-07:00

I'm talking about the Netgear ProSafe Gigabit Switch. The thing is an utter piece of garbage and has caused me no end of grief.

I bought one with the intent of using it to connect my workstation to my new OpenSolaris file server I've built. It certainly looked like an attractive package. Tiny footprint, low power, just enough ports for my small office, and Gigabit connectivity. Great, right?

That is, until I tried to use the damn thing. I've spent the last week trying numerous network adapters, OS's, drivers, on my desktop. I've tried several different Intel gigabit network adapters, and the onboard Realtek adapter on my desktop. The Realtek connects at 100 mbps, while the Intel adapters can only muster a measly 10 mbps. It's like being back in the early 90's. What the heck? My OpenSolaris machine, which has another Realtek adapter, is able to connect at a full 1000 mbps, but only after negotiating away for a while with the switch. If I connect my machines directly together, they negotiate a 1000 mbps connection in under a second. Connecting them to the switch leads to 30 or 40 seconds of trying to figure some crap out.

And before someone suggests it, yes I've checked the cabling. I've tried something like 5 or 6 different ethernet cables, all CAT 5E, all of them tested with other equipment at the office. No dice.

Apparently I'm not alone.

Funny thing is, I borrowed a Netgear GS608, a similar product but in a shinier looking case, and it had the same fricking problem!

My recommendation is to avoid these products at all costs if you value your sanity.

Bcfg2 RPMs available from openSUSE build service

2009-05-01T11:09:00.004-07:00

I've started building Bcfg2 RPMs using openSUSE's build service. They're available for most popular RPM based distros from http://download.opensuse.org/repositories/home:/kisielk/

Currently I only have builds from the 1.0.0pre2 tarball available but I'm considering also uploading some SVN snapshots in the future.

I'll also eventually be producing builds for deb based systems, once I figure that out.

I'm hoping this will eventually be incorporated "officially" in to the Bcfg2 project

Please note that the packages are currently largely untested in actual use since I don't have virtual machines set up with most of these distros at this point. I'd welcome everyone to test them and report any problems to me either in #bcfg2 on irc.freenode.org or by email to kamil@kamilkisiel.net

enzyme.vim - A Terminal.app friendly color scheme

2008-11-26T17:01:00.002-08:00

I finally got fed up of not being able to read code in Terminal.app when it was being highlighted in Vim. It seems no matter what color scheme I used, there was always some text that I couldn't see properly. This is mostly because Terminal.app renders red and blue too dark to be readable in many conditions if they are set on a black background.

I decided to bite the bullet and write my own Vim theme, which actually wasn't that hard. It's called enzmye and you can get it from vim.org. It currently doesn't do anything fancy, but it's readable under Terminal.app and that's all I care about.

Feedback and improvements are appreciated. Make sure you follow the instructions for setting up your terminal.

Bcfg2 0.9.6 Released!

2008-11-17T10:44:00.001-08:00

Get it while it's hot: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Download

I also have an upcoming series of blog posts about managing systems with Bcfg2 in the works. Looks for it soon.

Bcfg2 0.9.6pre3 released

2008-10-19T10:33:00.003-07:00

The 3rd prerelease of Bcfg2 0.9.6 is now available.

For those not in the know, Bcfg2 is a system that:

helps system administrators produce a consistent, reproducible, and verifiable description of their environment, and offers visualization and reporting tools to aid in day-to-day administrative tasks.

Basically it comes down to managing your system configurations from a central location and then pulling (or optionally, pushing) the configuration data down to each machine. This ensures your machines are in a known state, and eliminates the need to go around to each one and manually verify or copy configuration.

Other tools in this category include Puppet or CFEngine, but IMO Bcfg2 trumps either of those.

Check it out. If you have any questions, feel free to come to #bcfg2 on irc.freenode.net and someone can surely help you out.

Connecting to a Cisco IPSEC VPN from Linux -- without the Cisco client

2008-10-19T01:23:00.007-07:00

So, let's say your workplace uses a Cisco IPSEC VPN solution. Many places do. Let's also say you at home have a Linux machine. Being the good Linux user that you are, you keep your system well patched and run a recent kernel release.

You download the Cisco VPN client -- from your corporate website since, of course. Cisco would never make such a thing publicly downloadable.. who does that anyway?

You extract the tarball, run the vpn_install script as instructed and BAM. The whole thing bombs! Why? Because your system is too cutting edge for the guys at Cisco to keep up (clearly!). So, your possible solutions are:

1. Dig through a bunch of random internet forums, searching for the right combination of patches and command incantations that will make the damn thing work on your particular OS and kernel version.
2. Ditch the piece of junk altogether and install something nicer.

So which should we do? Alright.. let's go with option 1... just kidding, I mean 2.

Enter a wonderful piece of software called vpnc. Now, I'll be the first to admit I don't know much about how this particular piece of software works. And that's the great thing. Getting the VPN connection up and going was just that simple. So here's how:

1. I presume your company uses a PCF file along-side their Cisco VPN client. If not, you have to figure out how to enter the settings yourself. Download this .pcf file and put it somewhere. Say ~/mycompany.pcf
2. Download http://svn.unix-ag.uni-kl.de/vpnc/trunk/pcf2vpnc
3. Install vpnc. If you use Ubuntu, this means aptitude install vpnc. Yes, that is all.
3. Run pcf2vpnc mycompany.pcf mycompany.conf
4. cp mycompany.conf /etc/vpnc/
5. sudo vpnc mycompany
6. There is no step 6!

Oh yeah, at some point you want to disconnect and go do something else other than work. For that use sudo vpnc-disconnect.

I tested this on Hardy Heron, results may vary between distributions.

When running pcf2vpnc you may receive the following message:



Can't exec "cisco-decrypt": No such file or directory at ./pcf2vpnc line 30.

cisco-decrypt not in search path,

adding passwords in obfuscated form

This just means that your vpn configuration will contain your password in obfuscated form instead of plaintext, it does not mean the conversion failed.

Update 2009/02/20:
Someone has posted a howto which can work for OS X as well: http://www.gdanko.net/vpnc.html

Update 2009/06/15:

If you receive an error message such as

vpnc: no response from target

you need to add the line

NAT Traversal Mode cisco-udp

to your mycompany.conf file.

Bcfg2 Now in the Gentoo Tree

2008-07-06T09:44:00.003-07:00

Bcfg2 has finally hit the official Gentoo tree. For those who don't know, it's a configuration management system akin to Puppet or CFEngine. However, I think Bcfg2 has many advantages over these, the biggest of which being an active development and user community and great support on #bcfg2 on Freenode.net. Personally I also find the configuration definitions to be far more understandable that either of the other two programs. It's written in Python, and has a nicely architected plugin infrastructure.

The package management plugin for Gentoo is still a little weak compared to the Yum or Deb plugins, but it works great otherwise. I encourage Gentoo users who manage any number of systems to install it and give it a try. Let's find and squash some bugs and improve this great program. Emerge app-admin/bcfg2 to get started.

If you're on an RPM of Debian-based distro, I also encourage you to give this program a try, it's quite powerful and very fully featured.

Authenticating Windows against Open Directory

2008-06-30T10:40:00.003-07:00

First of all, apologies to everyone for the long time between posts, I've been suffering from a slight shortage of inspiration lately.

However, today I figured out something quite cool. It is possible to authenticate Windows (2000, XP, and possibly Vista) machines against Apple's Open Directory. This is great if you have an Open Directory server as your user account central store.

The software that enables this is called pGina.

To get stared, simply download and install pGina. Then download the additional plugins. The one we're interested in installing is the ldapauth plugin. Install the plugin somewhere in to your pGina installation. eg: c:\pGina\plugins

Now launch the configuration utility for pGina, and in the "Plugin" tab browse to the ldapauth_plus.dll plugin in c:\pGina\plugins\ldapauth\. Click the "Configure" button. Ensure the "LDAP Method" is set to "Search Mode". In the "LDAP Server" field enter the DNS name or IP of your Open Directory server. Leave the port at the default 389. You can leave the rest of the fields blank. Then in contexts add cn=Users,dc=company,dc=com where the last two segments are your base DN. This will depend on your site's configuration. If you're unsure, I recommend using a tool like Apache Directory Studio to examine your LDAP server. Finally you should go to the "Password Configuration" tab and check the "Disable Change Password" box. If a user changes their password only on their LDAP server, it may mess up other things on the system such as their kerberos and keychain passwords.

Unfortunately, I don't think it's possible to use the groups features in the "User Configuration" tab of this plugin as I can't find a way to make it look up group membership in the `cn=Groups` container. Perhaps I'll try hacking this on some day if we ever need to use it here.

Now that you have configured the plugin, you can configure the rest of pGina as you see fit.

The one caveat to using pGina is that if you're using it for the purpose of sharing files out over CIFS/SMB or remote desktop, the users will need to log in to the machine locally first, unless the share is readable by "Everyone". This is because pGina only handles the authentication portion of things at the login window or when connecting remotely. When you are setting permissions on a share, Windows will not be able to look up the users from LDAP, so they will only be available if their account exists on the machine from a previous login.

I hope this has been of use. If you have any further tips then please share them in the comments section.

An exchange analogy

2008-04-16T09:56:00.002-07:00

A funny analogy posted on slashdot.

If the same method that exchange/outlook uses to store email were used in the real world as a paper filing system: Every document is translated into Greek, and the original is burned. Then they are all glued together into one solid block and stuffed into a magic box with a tiny slot, through which you can talk to a little gnome who somehow gets each message for you as needed. Sometimes the gnome gets confused and it takes hours (sometimes days) for him to sort things out; meanwhile he can't find your documents until he is totally finished becoming unconfused again. As an added bonus the gnome costs several thousand dollars and when he dies every few years you need to buy a new gnome. Oh and if the first box gets (arbitrarily) full you have to buy another special gnomebox, which of course costs $$$

Disclaimer: I don't think exchange is all bad, but I think it could be much better and certainly more open.

HOWTO: Make a Nagios Dashboard widget in 50 seconds.

2008-04-07T22:41:00.005-07:00

Do you use Nagios to monitor your network? Do you use a mac? Do you think it would be handy to have a Dashboard widget showing your Nagios overview panel? Well, it's quite easy to do. Check out this screencast where I make a Nagios Dashboard widget using Safari's new web clips feature.

I apologize for the lack of audio, my mini at home doesn't have a microphone. However, I think the images show you all you need to know.

You can also view the full size video (approximately 8 MB)

Why Gentoo?

2008-04-06T21:33:00.002-07:00

Prompted by the editor of DistroWatch.com Gentoo developer Ben de Groot recently appealed to the Gentoo community to post their reasons for using Gentoo. As a big Gentoo fan and long time I user, I feel inclined to respond.

I am currently responsible for maintaining an HPC cluster of 76 nodes, at roughly 360 cores. The nodes all boot off of an NFS-mounted Gentoo image that we have heavily customized. In addition, we have roughly 10 developer machines which use a different Gentoo image. To support this infrastructure we also have a number of servers, which perform all sorts of tasks: Serving files via NFS, PBS scheduler, netboot server, Xen virtual machine host, etc. All of these are also Gentoo. In total we have over 100 machines running Gentoo.

The whole setup is surprisingly easy to manage. I have set up a local portage mirror, and a local overlay for our own packages. Classes of machines use the same image, so bringing up a new developer box is just an rsync away. A new cluster node can be added just by plugging a macine in to the network and netbooting. Xen domU's have a template from which they are cloned. For servers, there's a bootstrap script which builds the machine up with the minimum amount of input at the start of the process. We don't yet have a binary package host, but it's something I'm looking at adding.

Some reasons why I love using Gentoo:

New versions of packages are quickly available. Some as quickly as the day of the release! For example, the recent OpenSSH 5.0 already has ebuilds.
System configuration is not hidden or managed by a restrictive GUI. I'm a configuration file kind of guy, and the way system configuration is handled in Gentoo is very nice for that.
You install only exactly what you need. There isn't really a "default" set of packages, other than the minimum to get the system up and running. This is great for things like our cluster node image.
It's easy to create your own ebuilds. An ebuild for simpler things can take as little as 5 minutes to write, and lets you manage deployment across the network. We also maintain a number of software packages in our SVN repository. With the Subversion eclass, updating an ebuild to pull from a newer release in our repository can be as easy as renaming the file. This is nice.

That's all for now. If you have any questions about our setup, please feel free to contact me and I'll be glad to help you out.

Leopard AFP: Not production ready

2008-04-03T20:12:00.002-07:00

The other day we downgraded our Leopard file server back to Tiger. Luckily we had a spare XServe available for this purpose. For a detailed description of the problem, see my previous post. I can offer this additional information:

The problem is triggered by attempted authentications. We set up a script to monitor the AFP server every 20 minutes to check if it was down. Often times we could see the DirectoryService crash log timestamp corresponding nearly exactly with the time that our script attempted its test.
Sending a HUP signal, or toggling an AFP option like EnableGuestAccess (which does the same thing, I think), allows people to authenticate AFP connections again. At least until the next time it crashes.
Eventually the server comes down hard. We managed to keep it up with our monitoring tool and the HUP periodically, but at some point it seems to give up and die completely. This requires a complete restart of AFP, and a loss of all client connections. It really sucks for home directories, and can corrupt files as well (myself and several other users seem to have lost some preference files which were in use at the time).

Since we're doing testing on a new laptop image, and causing lots of AFP connections, we were getting more than 1 AFP/DirectoryService crash per hour. Unacceptable. There is definitely something wrong with the link between the two.

In the end, we were forced to call a network downtime for the end of the day and rebuilt our XServe with a Tiger image. Today was our first day running that setup, and it was solid as a rock. No slowness, no crashing.

In short, Leopard Server is not ready, at least not for serving AFP. Keep waiting.

AFP + Directory Services on Leopard = Disaster

2008-03-30T10:40:00.004-07:00

All it takes is a cursory search on some Apple discussion boards or mailing lists for "AFP crash" or "DirectoryService crash" to turn up a load of discussions on the topic.

The summary of the problem is basically this: The DirectoryService process crashes for some reason, then gets restarted by launchd. However, AFP (or more specifically, the AppleFileServer process) appears to not regain its connection to it. This prevents any new AFP connections from being able to authenticate, and existing ones are unable to re-authenticate. Couple this with AFP mounted home directories, and now your users can't log in to their workstations, or their existing session hangs.

In said discussions there are dozens of proposed workarounds. These include: Periodically HUP'ing the AppleFileServer process, setting up some crazy firewall rules, periodically toggling guest access, and numerous other things. I personally have tried many of them and can confidently say that none of them are a good solution. The toggling seems to mitigate the problem to some degree, but eventually things still come down hard.

One fix that appeared promising which we tried recently is not running Open Directory (of the network variety) on the same host as AFP. Fortunately we had a second XServe which was acting as an OD replica and not much else, so we demoted it to a server which is just connected to OD, and moved out AFP home share there. This seemed to work fine for at least a day, but then this weekend the DirectoryService process crashed yet again, causing the same problem as before.e

The thing that really blows my mind about this whole issue is that people have been reporting it since November of last year. That's 5 whole months, and still no sign of a fix from Apple! Say what you will about other companies being slow to respond to problems, I've never seen a major issue like this take so long to be fixed by anyone else.

With OS X 10.5.3 being seeded to developers in the last few days, I hope that Apple finally gets on the ball and fixes this glaring problem! This is definitely one of the most frustrating problems I've encountered during my time in the computing industry..

Mounting an HFS+ volume from Linux

2008-03-07T11:22:00.004-08:00

Yesterday I spent some time figuring out how to mount a snapshot of one of our Mac server's HFS+ volumes from our iSCSI SAN.
There doesn't appear to be any clear instructions for how to use an HFS+ formatted device in Linux, but I finally figured it out.

You need the following kernel options:


CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_MAC_PARTITION=y
CONFIG_EFI_PARTITION=y

(of course you can always put y where I put m, but I prefer to use modules).

Most howto's online omit the CONFIG_EFI_PARTITION option, but it's required for the kernel to recognize partitions on devices using the UUID partition scheme. You can tell if your device is one of these by looking at it in Disk Utility on a mac.

Now when you connect to a block device which contains an HFS+ volume, it will show up with several partitions. You can inspect them if you use parted. Typically the first one will be some kind of special EFI partition, and the second one will be the partition that contains the actual data.

You should not be able to mount it with something like
# mount -t hfsplus /dev/sdc2 /mnt/tmp

However, it appears that if you have a journaled filesystem you can only read and not write to the volume, even if mounted rw. Another thing to keep in mind is that it doesn't appear that ACL's or the resource fork are handled in any way at the moment...

Update:
The partition numbering is not entirely correct. After further experimentation, it appears to vary in different cases. In the case of two or three partitions, it's the second one. In the case that four partitions appear, it's the third.

A new focus

2008-02-15T20:48:00.002-08:00

Since returning from Japan I've been quite busy and not posting many personal posts on this blog. Frankly I don't have the time or inclination, and since I can see most of the people who would care about it on a day to day basis, there isn't really much point.

I figure it's time to take this blog to a different level and change the focus a little. I will no longer be posting random personal things things here but instead focus on writing concise and informative articles. The topics will include most of the things I deal with every day: Modern Unix (Linux and OS X) system administration, virtualization, clustering, and high performance computing.

There are lots of exciting new technologies out there, and I am fortunate enough to have the opportunity to work and experiment with many of them. I hope that I can use this blog as an outlet to share many of them with the internet and computing community.

Forcing a zone retransfer in Bind on OS X Leopard Server

2008-02-14T15:17:00.004-08:00

Recently we started migrating to OS X 10.5. Of course the process is fraught with many challenges, and lots of things are done differently.

For example, now views are used by default in DNS. This makes some things a bit more convoluted, but it may make other things easier in the future.

If you've ever administered a bind install, you may know about the rndc tool. It can perform all sorts of things without having to restart BIND (downtime is bad, mmkay?). However, the syntax is not always clear. For example, when using views, how do you retransfer a zone?

It goes something like this:

~# rndc retransfer zone myzone.mydomain.com IN myView

If you're using Leopard server, you probably have the default view name, so this becomes:

~# rndc retransfer zone myzone.mydomain.com IN com.apple.ServerAdmin.DNS.public

Speaking of Server Admin, it was clearly a very rushed application. The DNS portion is particularly horrific... you can't even enable transfers for a reverse zone!

VMWare Linux Guest Clock Synchronization

2008-01-31T10:40:00.002-08:00

One thing many home users of VMWare might not know or care much about is clock synchronization. However, in a networked environment, clock synchronization is an essential fact of life. In particular, some protocols such as Kerberos pretty much depend on your machines' clocks being in sync. Additionally, if you want to have any kind of logging or network monitoring solution in place, the output of such things can be difficult if not impossible to interpret if your clocks are all over the map.

The typical solution for most standard configurations is to run an NTP daemon on every machine and a few time servers on your network which synchronize with an outside source such as an internet time server or something more sophisticated. I won't go in to the details of how the NTP protocol works, but it's important to keep in mind that it is intended to gradually correct for clock drift and not simply jump your clock to the correct time.

In most UNIX environments you will find two programs which can be used to synchronize your time with an NTP server: ntp and ntpdate. ntp is the one which performs gradual corrections, and can also optionally act as a server process for other hosts. ntpdate will simply reset your time to whatever response it receives from the time server. It's intended for use during the boot process to set the initial time, or in the case that something goes horribly wrong with your clock (such as in a misconfigured VMWare guest...).

So how does this all fit in VMWare, we can just use ntp to keep the guest clocks in sync, right? Well, it's not quite that simple. While I won't go in to all the technical details here, because they're described in this excellent VMWare paper, this method won't work. Because of the way VMWare hands out clock interrupts to virtual machines, NTP can become extremely confused. Enter VMWare Tools.

As you may know, VMWare tools enables additional functionality such as video acceleration and time synchronization. So we can just turn on the time sync, right? Well, it's not quite so easy. One thing many people don't realize about the time sync feature of VMWare tools is that it will only catch up the clock if it's running too slow. If it's running too fast, the time sync feature does exactly nothing! Unfortunately, a fast clock is a more common case than a slow one when it comes to running a Linux virtual machine. I'm not going to go in to detail why, but if you want to know, read the paper above.

So what can you do to make your clock run on time? Well, the solution is actually not too difficult. Many people suggest recompiling the kernel of a virtual machine to run at a lower interrupt rate and so forth. That often works, but may not totally alleviate the problem. There's another way.

If you look again at the above mentioned paper, you'll notice that there are many timer sources available to Linux. The simplest is the PIT, or Programmable Interrupt Timer. There's a bunch of other ones such as APIC and and LAPIC. These don't appear to work reliably under VMWare. The solution is to turn them off and force PIT. This can be accomplished by some simple kernel boot flags. The relevant section from my Debian VM's grub.conf:


title           Debian GNU/Linux, kernel 2.6.22-3-686
root            (hd0,0)
kernel          /vmlinuz-2.6.22-3-686 root=/dev/mapper/debian-root ro clocksource=pit nosmp noapic nolapic
initrd          /initrd.img-2.6.22-3-686

Note the kernel line. You need to append clocksource=pit noapic nolapic. If you're not running an SMP VM, add nosmp as well (this may not be strictly necessary, but I don't think it can hurt). After these changes, your kernel should boot up using the PIT clock source. Verify by taking a look at your dmesg output and grepping for "clock".

Now your virtual machine's clock will lag very slightly instead of running fast. Fortunately VMWare Tools (which you did install, right?) will run approximately every minute and synchronize it back to the right time. Make sure you don't run NTP inside the VM, and make sure you do run it on the host.

EDIT (2008/02/26):
I forgot to mention in the original article that you need to enable time synchronization for your virtual machine. You can do this either through the GUI or by adding the line


tools.syncTime = true

To the .vmx file of your virtual machine.

War on...

2007-12-08T10:00:00.001-08:00

While countries like the US may have a "War on Drugs" and a "War on Terror" only the Japanese could bring you... War on Dangerous Bicyclists

Images loading slowly in Firefox?

2007-11-27T20:47:00.000-08:00

Well, it might just be you've disabled your cache. For the longest time I thought that my internet connection was being slow. Instead, it seems that I hit the "disable cache" option on the Firefox Web Developer Toolbar and then at some later time hid the toolbar. Oops.

Installing VMWare Tools on Kernel 2.6.22

2007-11-19T11:33:00.000-08:00

If you're trying to install VMWare Tools on a Linux machine with a recent kernel (I was using 2.6.22, but this affects a couple revisions below as well), some kernel API changes prevent the tools from installing. You'll end up with an error that looks something like this:


Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmhgfs-only'
make -C /lib/modules/2.6.18-1.2200.fc5/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-1.2200.fc5-i686'
CC [M] /tmp/vmware-config0/vmhgfs-only/cpName.o
CC [M] /tmp/vmware-config0/vmhgfs-only/cpNameLinux.o
CC [M] /tmp/vmware-config0/vmhgfs-only/dev.o
CC [M] /tmp/vmware-config0/vmhgfs-only/driver.o
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsChangeFileAttributes":
/tmp/vmware-config0/vmhgfs-only/driver.c:763: error: "struct inode" has no member named "i_blksize"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsInitializeInode":
/tmp/vmware-config0/vmhgfs-only/driver.c:835: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsIget":
/tmp/vmware-config0/vmhgfs-only/driver.c:884: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsCreate":
/tmp/vmware-config0/vmhgfs-only/driver.c:1536: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsLookup":
/tmp/vmware-config0/vmhgfs-only/driver.c:1636: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsMkdir":
/tmp/vmware-config0/vmhgfs-only/driver.c:1728: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsDelete":
/tmp/vmware-config0/vmhgfs-only/driver.c:1855: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsRename":
/tmp/vmware-config0/vmhgfs-only/driver.c:2048: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c:2050: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsRevalidate":
/tmp/vmware-config0/vmhgfs-only/driver.c:2294: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsSetattr":
/tmp/vmware-config0/vmhgfs-only/driver.c:2431: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsOpen":
/tmp/vmware-config0/vmhgfs-only/driver.c:2808: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsDirOpen":
/tmp/vmware-config0/vmhgfs-only/driver.c:3422: error: "struct inode" has no member named "u"
/tmp/vmware-config0/vmhgfs-only/driver.c: In function "HgfsClearInode":
/tmp/vmware-config0/vmhgfs-only/driver.c:4113: error: "struct inode" has no member named "u"
make[2]: *** [/tmp/vmware-config0/vmhgfs-only/driver.o] Error 1
make[1]: *** [_module_/tmp/vmware-config0/vmhgfs-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-1.2200.fc5-i686'
make: *** [vmhgfs.ko] Error 2
make: Leaving directory `/tmp/vmware-config0/vmhgfs-only'
Unable to build the vmhgfs module.

Fortunately the fix is fairly straightforward. Go to the vmware-tools-distrib/lib/modules/source directory, and untar the driver's tar file: tar xvf vmhgfs.tar

Now edit vmhgfs/driver.c and change lines 44 and 45 from


#define INODE_SET_II_P(inode, info) do { (inode)->u.generic_ip = (info); } while (0)
#define INODE_GET_II_P(inode) ((HgfsInodeInfo *)(inode)->u.generic_ip)


#define INODE_SET_II_P(inode, info) do { (inode)->i_private = (info); } while (0)
#define INODE_GET_II_P(inode) ((HgfsInodeInfo *)(inode)->i_private)

Also, remove line 763: inode->i_blksize = HGFS_BLOCKSIZE;

Now just tar everything back up (tar cvf vmhgfs.tar vmhgfs-only) and rerun vmware-config.pl, or vmware-install.pl if you haven't yet installed the tools.