Rants on technology

Ubuntu xenial64 on Vagrant

2017-03-04T18:25:00.000+11:00

If you are getting errors similar to the ones below:

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

hostname -f

Stdout from the command:

Stderr from the command:

sudo: unable to resolve host ubuntu-xenial
mesg: ttyname failed: Inappropriate ioctl for device
hostname: Name or service not known

Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel module.
Please verify that these guest additions are properly installed in the
guest. This is not a bug in Vagrant and is usually caused by a faulty
Vagrant box. For context, the command attempted was:

mount -t vboxsf -o uid=1000,gid=1000 v-csc-78dab358b /tmp/vagrant-chef/1d140fd50fa8a0774caff0f697e00977/cookbooks

The error output from the command was:

mount: unknown filesystem type 'vboxsf'

And keep finding google results for these threads:

Here's a TLDR of what you probably need to do:

1. Upgrade Vagrant to at least 1.9.2.
2. Install vagrant-vbguest.

vagrant plugin vagrant-vbguest

Can't flush bind cache due to rndc: connect failed

2013-10-17T00:57:00.003+11:00

I couldn't find a solution for this in the googles:

$ sudo rndc flushname somedomain.com
rndc: connect failed: 127.0.0.1#953: connection refused

Another symptom:

$ sudo service named restart
Stopping named: . [ OK ]
mount: block device /etc/rndc.key is write-protected, mounting read-only
mount: cannot mount block device /etc/rndc.key read-only
Starting named: [ OK ]

The setup:

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)
$ rpm -q bind
bind-9.8.2-0.17.rc1.el6_4.6.x86_64

The fix:

# cd /var/named/chroot/
# cp /etc/rndc.key .
cp: overwrite `./rndc.key'? y
# /etc/init.d/named restart
# logout
$ sudo rndc flushname somedomain.com

Linux driver file (PPD) for Fuji Xerox printers

2012-11-23T08:07:00.005+11:00

The cheapskates at FX don't provide PPDs for Linux, even though they are the same as the MacOS files.

Now, opening apple's *.dmg files is surprisingly convoluted. Here's a step-by-step guide to, hopefully, save you some time.

1. Download the MacOS ApeosPort IV drivers from Fuji Xerox's site.
2. Download dmg2img (I downloaded the source code for dmg2img and ran `make' - you guys rock!).
3. Convert the dmg file into an image:

./dmg2img ~/Downloads/fxmacprnps1208am105iml.dmg /tmp/out.img

4. Mount the obtained image:

sudo mount -o loop /tmp/out.img /mnt/iso/

5. Use xar (yum install xar) to extract the pkg file:

cp /mnt/iso/Fuji\ Xerox\ PS\ Plug-in\ Installer.pkg /tmp/

cd /tmp && mkdir fx && cd fx

xar -xf ../Fuji\ Xerox\ PS\ Plug-in\ Installer.pkg

6. Inside the extracted folders, locate and copy the Payload file with the PPDs:

cp ppd.pkg/Payload /tmp/Payload.cpio.gz

7. Gunzip the file and extract the cpio archive:

cd /tmp

gunzip Payload.cpio.gz

mkdir ppd && cd /tmp/ppd

cpio -id < ../Payload.cpio

8. Voilà! your PPD files are in Library/Printers/PPDs/Contents/Resources/:

$ls

Fuji Xerox 4112 PS.gz FX ApeosPort-IV C4475 PS.gz FX DocuCentre-IV 7080 PS.gz

Fuji Xerox 4127 PS.gz FX ApeosPort-IV C5570 PS.gz FX DocuCentre-IV C2260 PS.gz

Fuji Xerox D110 PS.gz FX ApeosPort-IV C5575 PS.gz FX DocuCentre-IV C2263 PS.gz

Fuji Xerox D125 PS.gz FX ApeosPort-IV C5580 PS.gz FX DocuCentre-IV C2265 PS.gz

Fuji Xerox D95 PS.gz FX ApeosPort-IV C6680 PS.gz FX DocuCentre-IV C2270 PS.gz

FX ApeosPort 350 I PS B.gz FX ApeosPort-IV C7780 PS.gz FX DocuCentre-IV C2275 PS.gz

FX ApeosPort 350 I PS.gz FX DocuCentre 450 I PS B.gz FX DocuCentre-IV C3370 PS.gz

(...)

Enjoy.

Caching 301 redirects in Varnish while keeping the protocol

2012-11-08T08:26:00.007+11:00

I have noticed a drop in our Varnish cache hit ratio and, upon investigation, found that the backend was not allowing 301 redirects to be cached:

Client request:

42 RxRequest c GET

42 RxURL c /img_resized/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

42 RxProtocol c HTTP/1.1

Backend request:

21 TxRequest b GET

21 TxURL b /img_resized/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

21 TxProtocol b HTTP/1.1

Backend response:

21 RxHeader b Cache-Control: no-cache

21 RxHeader b Location: http://static.example.com/img/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

21 RxHeader b Status: 301

I logged a bug for the application to be fixed and, in the meantime, added a VCL snippet to override the backend headers:

# WSF-xxxx: App is not allowing redirects to be cached

if ((req.http.Host == "static.example.com") &&

(beresp.status == 301) &&

(beresp.http.Cache-Control ~ "no-cache")) {

set beresp.http.Cache-Control = "public, max-age=604800";

}

There's a catch though. We do SSL offloading for static.example.com in our F5 BigIP LTM load balancer. In other words, the load balancer receives a https request and, in turn, makes a http request to Varnish.

As Varnish doesn't support SSL natively it is unaware of the protocol (http or https) being used; thus, the requests below get cached with the first answer it sees:

HTTP request

curl -I http://static.example.com/img/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

(...)

HTTP/1.1 301 Moved Permanently

Location: http://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTPS request

curl -I https://static.example.com/img/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

(...)

HTTP/1.1 301 Moved Permanently

Location: http://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

Notice that the HTTPS request was redirected to a HTTP URL. This will most certainly cause issues, including mixed-mode SSL alerts in some browsers.

Our backend is a Ruby on Rails application and, to my surprise, it understands the X-Forwarded-Proto HTTP header. This means that the application will use the value of X-Forwarded-Proto to build the URL in the Location header.

I improved the F5 iRule by instructing the load balancer to insert a X-Forwarded-Proto header in HTTPS requests which were being off-loaded to Varnish:

when HTTP_REQUEST {

HTTP::header replace X-Forwarded-Proto "https"

}

In addition to this, I also had to instruct Varnish to use the X-Forwarded-Proto header in the cache hash:

sub vcl_hash {

if (req.http.X-Forwarded-Proto) {

set req.hash += req.http.X-Forwarded-Proto;

}

With these tweaks Varnish is now able to account for SSL offloading and correctly serve the appropriate cached version of a page. An improvement on this configuration would be to only use X-Forwarded-Proto in the hash if the response is a redirect; at the moment I can't be bothered that Varnish is caching objects twice.

HTTP request

$curl -I http://static.example.com/img_resized/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTP/1.1 301 Moved Permanently

Location: http://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTPS request

$curl -I https://static.example.com/img_resized/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTP/1.1 301 Moved Permanently

Location: https://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

Using varnish to increase the cache time of slow pages

2012-03-10T09:26:00.000+11:00

If you are having trouble getting your organisation to accept performance as a feature, one solution is to tie it to another feature.

A common discussion between content owners ("the business") and web operators is how long to cache pages for. The content owners want to see their fresh content in minimal time but web operators want to cache expensive (to generate) content longer.

Usually content owners get their way, as they should, or you wouldn't have a business. So you set the default cache time on all pages to something low, like 10 minutes. If you are not quite ready to take the ESI leap, an attractive compromise is to penalise only expensive pages with a higher expiry time.

If your server backend runs on Ruby-On-Rails, the job is already half-done since requests return a X-Runtime HTTP header indicating, in milliseconds, how long it took RoR to generate the page. This header is either available in most web frameworks or would be quite easy to implement.

We use the VCL code below to override the internal varnish TTL for slow-to-generate pages.

# VCL 2.1.5 to extend TTL on pages which take 1s or longer to generate
if (beresp.http.X-Runtime ~ "[0-9]{4,}") {
if (beresp.ttl < 1d) {
set beresp.ttl = 1d;
set beresp.http.X-Cache-Override = "1d";
}
}

We choose not to modify the original headers, which are sent to the client, because this gives you the opportunity to keep or ban the content from Varnish while letting the browsers come back to ask for the "new" content.

The code also sets a "X-Cache-Override" header so you can tell which pages are taking 1 second or more to generate.

This is great because now, when someone complains that their content isn’t refreshing, we can improve performance to areas of the site that matter to the business instead of the whole “make the site faster” approach.

The results will come quickly. Two days after putting this in production Google has shown that the average page load time in our (very large) site dropped half a second. As you can see in the graph below, the trend is still going down (click on the image).

Site speed improvement as perceived by the Google bot

A further improvement to this would be to write an algorithm which sets the TTL based on the page generation time. For example, add 1 day to the TTL for each second it takes the page to generate so if a page takes 5 seconds to generate, keep it for 5 days.

Varnish and the Linux IO bottleneck

2010-07-27T09:20:00.005+10:00

There are 2 main ways Varnish caches your data:

to memory (with the malloc storage config)
to disk (with the file storage config)

As Kristian explains in a best practices post, you get better performance with #1 if your cache size fits in RAM. With either method the most accessed objects end-up in RAM. While the malloc method puts everything in RAM and the kernel swaps it out to disk, the file storage method puts everything in disk and the kernel caches it in memory.

We have been using the file storage for a little more than 3 months on servers with 16GB of RAM and a file storage of 30GB. Recently, because of an application configuration error, the size of the cache grew from its usual maximum of 6GB to 20GB+.

Since the system only has 16GB, the kernel now needs to choose which pages to keep in RAM. When a dirty mmap'ed page needs to be released the kernel will write the changes to disk before doing so. More than that, the linux kernel will proactively write dirty mmap'ed pages to disk.

On a Varnish server with this setup or any other application that uses mmap() for large files, the situation translates to constant disk activity.

At some point the varnish worker process will be blocked by a kernel IO call. The Varnish manager process, with no way to tell what's happening to the child, believes it has stopped responding and kills it. The log entries below are typical:

00:30:10.395590 varnishd[22919]: Child (22920) not responding to ping, killing it.

00:30:10.395622 varnishd[22919]: Child (22920) not responding to ping, killing it.

00:30:10.417309 varnishd[22919]: Child (22920) died signal=3

At this point you lose your cached data and the system goes back to normal. After some time, the size of the cache will grow to be larger than your server's RAM again, repeating the kill-restart cycle.

You could choose to give the client more time to respond to the manager process' ping requests. This is done by increasing the default value of cli_timeout from 10 seconds (e.g. varnish -p cli_timeout=20) but this is merely masking the issue. The real issue is that the Linux kernel is busy writing dirty pages to disk.

There are parameters you can use to control how much time the kernel spends writing dirty mmap pages to disk. I have spent some time fine-tuning the parameters below with little results:

/proc/sys/vm/dirty_writeback_centisecs

/proc/sys/vm/dirty_ratio

/proc/sys/vm/dirty_background_ratio

In RHEL 5.x kernels you can completely disable committing changes to mmap'ed files:

echo 0 > /proc/sys/vm/flush_mmap_pages

Short of reducing the file storage size to fit in RAM, this is the best solution I found so far. Be aware that using this with the upcoming persistent storage in Varnish is a really bad idea as you risk serving corrupt and/or stale data. This is only acceptable because the mmap'ed data, in this setup, is superfluous: varnish throws away the cache on restart and it can always fetch the object from the backend.

I have asked Red Hat if they plan to keep the flush_mmap_pages setting in RHEL 6 but haven't received a response yet. They did, however, confirm that msync() calls are honoured and dirty pages being evicted from RAM are committed to disk.

CruiseControl, ProxyPass

2010-04-30T11:38:00.004+10:00

Dear blog, today I submitted a patch for CruiseControl. It feels great to give something back (even if it's half-baked).

I couldn't find a way to make CC give the correct URL to users when running in a configuration like below:

ProxyVia On
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/
(...)

CC would insist on building URLs using localhost:8080 even though the user can only access via http://your.host.com/. With the patch CruiseControl will use the X-Forwarded-Host HTTP header to build the correct address.

The fallacy of cheap disk space

2010-01-16T09:52:00.007+11:00

At work my concerns for application disk usage/waste are usually met with a scoff of "Why bother? Disk space is cheap!"

This is a myth I intend to destroy in this post. It is true that disk space gets cheaper all the time but it is not cheap.

I'm not even going to step into the NAS/SAN area - this myth can be debunked looking at local storage alone.

When people say disk space is cheap these days they are probably thinking of their videos, mp3 and pr0n collection at home; enterprise-grade disks are another story. I chose the cheapest drives I could find to illustrate my point:

Desktop: 1.5TB SATA 7.2krpm A$133 A$0.089/GB (WD15EADS)
Enterprise: 300GB SCSI 15krpm A$489 A$1.630/GB (SFU300G10K80P)

The comparison already looks grim for the myth: the cost of a data centre-worthy disk is more than 18 times that of a "disk space is cheap" drive.

In the real world redundancy is required so, for your typical RAID1/10 scenario, the cost is 36 times more or A$3.26/GB. You could get a better cost, A$2.45/GB, with a 3-disk RAID5 volume but the recovery time for today's disks makes the alternative risky.

Then you need to backup your data. Regardless of your backup frequency and retention period you will need to buy more tapes if you have more data.

Assuming that 1 LTO drive is able to backup all your data in the alloted time frame, a minimal retention period and 3x 200GB LTO-2 tape (in use, on-site, off-site), you are looking at an additional A$0.57 per GB backed-up and a total cost of A$3.83/GB - MORE THAN 40 TIMES the cost of cheap disk space.

One could argue that this level of performance AND redundancy is not always needed - and one would be right. For instance, if your application data is cacheable you can split the data in layers of increasing cost/performance with something like Blackblaze's solution at the back.

Once we're looking for cheaper storage the "disk space is cheap" advocates have already lost the argument. In case we're dealing with the stubborn kind let's move from the capital to the operational expense of storage.

The following statement is fairly obvious but probably needs to be reminded when storage cost is dismissed as irrelevant: disk space is an asset that depreciates at the purchase price, not market price. Once you've bought 1TB of storage for $A3.26 per GB you're stuck with the entry until it's out of the books, no matter how much cheaper disks get in the future.

Your backup opex also increases. You area going to rotate and store more tapes. You will need off-line storage and transportation for more tapes. You get the picture.

If your data is mirrored/replicated to other sites your network costs also increase.

More: because every procedure takes more time your labour costs also go up.

And more: enjoy while local storage meets your demand - the picture gets much uglier with external storage.

Sen. Conroy and his sinister Internet filter

2009-07-28T08:18:00.001+10:00

Disk IO spikes caused by mod_proxy.c in Apache 1.3

2009-06-22T18:44:00.001+10:00

We have been using Apache 1.3 with mod_proxy as a reverse proxy for some of our applications. The configuration is pretty simple:


CacheRoot /var/adm/apache/cache
CacheSize 4096000
CacheGcInterval 4
CacheMaxExpire 48
CacheDefaultExpire 2

The above tells Apache to inspect the cache directory every 4 hours, keep up to 4GB worth of data, validate objects every 48 hours (no matter what the origin server set the expiry to) and expire them, by default, in 2 hours.

The typical content of the cache directory for our sites is 4.5GB in about 180,000 directories and 310,000 files.

Shortly after deploying this to production we started seeing the pattern below for the load on all web servers:

Because this was happening in regular intervals it was easy to inspect one of the servers during the load spike. The sar data had already told us the load was caused by disk IO:


$sar -b
(...)
                tps      rtps      wtps   bread/s   bwrtn/s
11:30:01 AM     94.53      1.89     92.63     27.68   1275.66
11:40:01 AM   1289.34     57.63   1231.71    555.35  20566.36
11:50:01 AM     79.16      1.23     77.93     21.88   1084.00
12:00:02 PM     94.21      1.77     92.44     39.52   1274.74
12:10:01 PM     84.18      1.64     82.54     35.54   1116.60
12:20:01 PM    117.43      2.62    114.81     44.68   1550.87
12:30:01 PM    150.65      3.30    147.35     82.21   1986.41
12:40:01 PM   1228.90      1.53   1227.37     33.04  20592.15
12:50:01 PM     70.46      2.65     67.80    100.57    917.25

When the load spiked I checked which processes were high on "wa" (waiting for IO) time and listed their open files with lsof.

Sure enough I could see httpd processes with changing files open in /var/adm/apache/cache. This is expected since the cache garbage collector needs to inspect cached objects and decide what to delete in order to stay under the CacheSize limit.

What surprised me is how amazingly expensive this operation is on ext3 filesystems.

To alleviate the problem I have changed some of the filesystem parameters. In /etc/fstab I told the OS to not update access times on files and directories under /var:

/dev/VolGroup00/LogVol04    /var   ext3    defaults,noatime,nodiratime        1 2

Without noatime and nodiratime every file and directory read meant a disk write to update the access time. Since we don't have a use for access times in /var I did away with them.

With tune2fs I change the data option from the default (ordered) to writeback:

# tune2fs -o journal_data_writeback /dev/mapper/VolGroup00-LogVol04

The writeback mode can cause file corruption on unclean shutdowns but the benefits outweights the risk for us. Our web content in /var is replicated from a master server so we would only be looking into losing non-critical data such as some state files and access log entries.

After these changes the GC is barely noticed. If you pay attention to the graph above you will see that there is no load spike for 13:00 onwards.

Apache 1.3.41 HostnameLookups bug?

2009-05-25T09:03:00.004+10:00

This weekend a scheduled maintenance in the data centre took out the name servers. I wasn't worried because all the names we needed to resolve in order to keep serving pages were in /etc/hosts.

Sure enough, during the maintenance alarms started going off. Web pages, static or dynamic, were taking in excess of 5 seconds to be served. We checked /etc/nsswitch.conf to make sure forward lookups were being resolved by /etc/hosts instead of DNS:

hosts: files dns

We then inspected the access log and we could see that some entries had host names instead of IP addresses. This was odd since reverse DNS lookups were turned off in Apache:

HostnameLookups Off

A colleague suggested it could be our LogFormat configuration:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

The documentation is pretty clear that %h will only do a reverse lookup if HostnameLookups is turned on:

If HostnameLookups is set to On, then the server will try to determine the hostname and log it in place of the IP address.

Nevertheless, we replaced %h with %a. That didn't solve the issue - requests still took little more than 5 seconds to be fulfilled. The millisecond part of the response time looked right to me, so I went on to read the resolv.conf man page and found this bit very interesting:

timeout:n
sets the amount of time the resolver will wait for a response from a remote name server before retrying the query via a different name server. Measured in seconds, the default is RES_TIMEOUT (currently 5, see <resolv.h>).

Now I was pretty much convinced that Apache was playing tricks on us. I double-checked all our Allow statements to make sure we weren't using hostnames, since the manual says it will do a reverse lookup in this case:

This configuration will cause the server to perform a double reverse DNS lookup on the client IP address, regardless of the setting of the HostnameLookups directive.

We only had Allow statements for the usual restricted URLs (server-status etc) and even then, were only using IP addresses.

I decided to disable the lookups in source code, since this was starting to look like a bug. In src/main/http_core.c, line 728, I commented the reverse lookup call and replaced it with a fake response:

// hptr = gethostbyaddr((char *)iaddr, sizeof(struct in_addr), AF_INET);
hptr = NULL;

I did the same on line 825:
// hptr = gethostbyaddr((char *)iaddr, sizeof(struct in_addr), AF_INET);
hptr = NULL;

The web server went back to serving static and dynamic objects in the normal time, tens of milliseconds. I deployed the newly compiled Apache to all servers and went back to my weekend.

This hack obviously breaks all functionality that depends on reverse IP address lookups so, if I have some time, I will analyse this condition and, if it is a bug, submit a patch to the chaps at the Apache Foundation.

QEMU/KVM mouse not working?

2009-04-23T16:05:00.002+10:00

This guy knows the answer:

$ export SDL_VIDEO_X11_DGAMOUSE=0

For me, because I use a physical device for a disk and didn't want to give myself permission on the device:

$ sudo SDL_VIDEO_X11_DGAMOUSE=0 qemu-launcher

My mouse stopped working after I configured Xinerama and display rotation on X (I had alsoi upgraded to Ubuntu 9.04 but I didn't test my VM before fiddling with xorg.conf).

XOrg configuration for Samsung LA32A450C1

2009-04-03T23:11:00.003+11:00

Th is guy n ailed it. I had only just found the manual page for my Samsung 32" LCD TV and was still googling for a tool to find the modeline.

I've only added the option to turn off nvidia's logo to his X configuration. Too good:

Section "Monitor"
Identifier "SamsungTV"
ModelName "Samsung LCD TV"
ModeLine "1360x768@60" 85.8 1360 1440 1552 1792 768 771 777 795 +hsync +vsync
Option "dpms"
EndSection

Section "Device"
Identifier "NVIDIAVideocard0"
Driver "nvidia"
Option "ModeValidation" "DFP-0: NoDFPNativeResolutionCheck"
Option "ExactModeTimingsDVI" "TRUE"
#### 100 DPI
Option "UseEDIDDpi" "FALSE"
Option "DPI" "100 x 100"
Option "NoLogo" "True"
EndSection

Section "Screen"
Identifier "Screen0"
Device "NVIDIAVideocard0"
Monitor "SamsungTV"
DefaultDepth 24
SubSection "Display"
Depth 24
##### Native Res Samsung 1360x768@60
Modes "1360x768@60" "1280x720" "1024x768" "800x600" "640x480"
EndSubSection
EndSection

RHN's yum update fails

2009-01-16T15:56:00.002+11:00

I've noticed that quite often Red Hat's yum will fail for no good reason. For example it will just show an error like:

# yum update
(...)

Error Downloading Packages:
samba-common - 3.0.28-1.el5_2.1.x86_64: failed to retrieve getPackage/samba-common-3.0.28-1.el5_2.1.x86_64.rpm from rhel-x86_64-server-5
error was [Errno 14] HTTP Error 404: Not Found

When that's the case, retrying a few times usually does the trick.

I have actually opened a ticket with RHN's support and they've told me these are "temporary errors". Temporary or not they are pretty annoying so here's what I do.

1. Retry manually until you're happy with the list of packages that will be updated
2. Run yum in a while loop so that it retries until it succeeds:

# while ! (yum update -y) ; do echo TRY AGAIN; done

Be aware that if that too many retries can cause your RHN subscription to be temporarily locked-out. I find the benefit too great to care.

Mytharchive: cannot burn a dvd

2008-12-21T09:54:00.003+11:00

I've been playing with MythTV again. Configuring it still is a major pain in the arse - there seems to be something wrong with the channel scanner that throws you off the track. I think.

Anyway, after recording a few programs I wanted to burn a DVD. My first attempt failed because Mytharchive tries writing to /dev/dvd by default. I solved this problem by linking /dev/dvd to my DVD device.

Tried again and Mytharchive refused to enter the burn DVD menu, showing the error below:

Mytharchive: cannot burn a dvd, the last run failed to create a dvd

WTF? A bit harsh on the error flagging, I'd say. So after searching around I found some people with the same problem but none of the helpful people would tell how to clear the error. Here's what you do.

1. Quit mythfrontend
2. Open a command prompt and:

mysql -u root
use mythconverg;
update settings set data='' where value='MythArchiveLastRunStatus';
update settings set data='' where value='MythArchiveLastRunType';

3. Go back to mytharchive and try again. This time it will let you proceed to the DVD burn menus.

How to avoid overlapping rsync instances

2008-12-11T21:07:00.005+11:00

Running rsync as a cron job is probably as popular as sliced bread. For small intervals, however, you risk having a second instance start before the first one has finished, beginning a downward spiral of bandwidth and processes.

To avoid multiple instances getting in each other's way there's a simple solution using flock. Example without lock (may cause overlapping instances):

*/5 * * * * /usr/bin/rsync --delete -a source_server:/source/path/ /dst/path/

Example using flock:

*/5 * * * * flock -xn /tmp/example.lock -c '/usr/bin/rsync --delete -a source_server:/source/path/ /dst/path/'

In the second example, if a rsync instance runs for more than 5 minutes, flock will fail and thus not execute a second rsync process.

This can be used for pretty much any shell problem where locking helps. Have a read on the flock man page for other examples.

No time? No worries!

2008-06-25T07:09:00.003+10:00

IIS asp.net v2.0 fake 403 and 404 errors

2008-06-05T12:44:00.006+10:00

After installing an ASP.NET v2.0 application I tried to browse to it but IIS wouldn't show the page and spit this 403 error instead:

The website declined to show this webpage

Now, there are a few causes to this problem. The most common is when the user forgets to add an index page to the site or, to add the index page to the list of "Enable default content page" under IIS.

Mr. Ian Tinsley found a more sinister cause that has the following symptom: if you browse to an existing .aspx file, ISS will return a 404 error code, even though the file exists. A good way to check if this is your case is to place a static file (htm or gif etc) in your site and try browsing it; if it shows you the static file then you probably have a fake 403/404 scenario.

To make sure, go to Web Service Extensions under IIS and check if ASP.NET v2.x is listed:

The tricky bit is: even if ASP.NET v2.x is not listed above, the v2.x extension still shows on the application properties. So if you don't have it above, you need to run aspnet_regiis.exe:

Running with -ir will help keep your asp.net v1.1 stuff running and still install v2:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -ir

Sun x4450

2008-05-03T11:56:00.004+10:00

With a range of options for the Intel Tigerton Quad-core Xeon processors, the Sun x4450 fitted the recommendations for our new postgres database servers. Solaris 10 was chosen by default as the operating system - why complicate the setup by mixing vendors, right?

Well, long story made short: I've ditched Solaris 10 x86_64 in favour of Red Hat Enterprise 5.1. Why? For one thing the Solaris 10 installation was frustratingly complicated.

Performing an installation via serial console requires you to redirect output from the BNC to the System in the BIOS settings. Then the manual states you should choose grub menu's option "ttb". Of course the manual meant "ttyb". But it is actually a typo AND an error: you must choose ttya or all you'll see after boot is "Sun Solaris 5.10" and nothing else.

Once the serial console installation is finished and the system reboots you have another obstacle. The boot process stops at this error message:

Bad PBR sig

Reboot and Select proper Boot device or
Insert Boot Media in selected Boot device and press a key

Googling around (Sun's KB was useless) I found someone with the same problem reporting that a GUI installation didn't show that error. Very well, I configured the firewall to allow Sun's network KVM:

8890 TCP Remote Console
9000 TCP Remote Console
9001 TCP Remote Console
9002 TCP Remote Console
9003 TCP Remote Console
69 UDP TFTP (for firmware upgrades)
161 UDP SNMP (for monitoring)

As the anonymous poster reported, this did solve the problem. Now to install postgres for x86_64. Hmm. Doesn't run - complains about missing libraries. Probably just need to run crle to set-up the location of the 64bit libraries. Where are they, where are they... WTF? There are no x86_64 libraries installed in my Solaris 10 for x86_64. I check around with `file' and yeah: all bloody system libraries are 32 bits.

I confess I didn't try too hard after this. A day later and we had postgres running on RHEL 5.1. And yes, with a 64bit binary and libraries.

To Sun's credit, the hardware looks a beauty, even though you have to assemble the whole thing by yourself: memory cards, SAS card, Fibre Channel card, hard disks and whatever extras you bought. What a pain in the arse. Compared to the other Dell servers with pre-installed RHEL 5.1 we recently bought the Sun experience is pathetic: the Dells where up on the same day versus 1 week for Sun.

Nagios checks for LSI RAID with MegaCli

2008-04-18T13:41:00.005+10:00

I am pretty sure there is a SNMP object in Dell's DRAC 5/PERC to inform about the status of your RAID volumes and physical disks. I'm not too fond of using SNMP with Nagios so I wrote a check script that uses the MegaCli linux i386 binary (from the LSI web site) to report the status of the RAID on our Dell/Red Hat Linux servers.

Here's what you need:

perl interpreter in /usr/bin/perl
nagios' utils.pm in /usr/local/nagios/libexec/
perl module Time::HiRes (cpan; install Time::HiRes)
sudo in /usr/bin/
MegaCli in /opt/MegaRAID/MegaCli/MegaCli64

You'll also need the check_dellperc in /usr/local/nagios/libexec. Change the paths as you see fit.


#!/usr/bin/perl -wT
#
# CHECK DELL/MegaRAID DISK ARRAYS ON LINUX
# $Id: check_dellperc 142 2008-03-17 22:25:46Z thiago $
#

BEGIN {
 $ENV{'PATH'} = '/usr/bin';
 $ENV{'ENV'} = '';
 $ENV{'BASH_ENV'} = '';
 $ENV{'IFS'} = ' ' if ( defined($ENV{'IFS'}) ) ;
}

use strict;
use lib "/usr/local/nagios/libexec";
use utils qw($TIMEOUT %ERRORS &print_revision &support &usage);

use Getopt::Long;

use Time::HiRes qw ( tv_interval gettimeofday );

use vars qw($opt_h $help $opt_V $version);
use vars qw($PROGNAME $SUDO $MEGACLI);

$PROGNAME = "check_dellperc";
$SUDO = "/usr/bin/sudo";
$MEGACLI = "/opt/MegaRAID/MegaCli/MegaCli64";

my $t_start = [gettimeofday];

Getopt::Long::Configure('bundling');
GetOptions
 ("V"   => \$opt_V,  "version"    => \$opt_V,
  "h"   => \$opt_h,  "help"       => \$opt_h,
 );


if ( $opt_V ) {    print_revision($PROGNAME, '$Id: check_dellperc 142 2008-03-17 22:25:46Z thiago $');
 exit $ERRORS{'OK'};

} elsif ( $opt_h ) {
 print_help();
 exit $ERRORS{'OK'};
}

my $TIMEOUT = $utils::TIMEOUT;
my $start_time = time();
# TODO: add timeout option#if ( $opt_t && $opt_t =~ /^([0-9]+)$/ ) {
#    $TIMEOUT = $1;
#}


# Check state of Logical Devices
my $status = "PERC OK";
my $perfdata = "";
my $errors = $ERRORS{'OK'};
my $vd = "";
my $vds = "";

open(MROUT, "$SUDO $MEGACLI -LDInfo -Lall -aALL -NoLog|");
if (!<MROUT>) {
 print("Can't run $MEGACLI\n");
 exit $ERRORS{'UNKNOWN'};
}

while (<MROUT>) {
 my $line = $_;
 chomp($line);

 if ($line =~ /^Virtual Disk: (\d+)/) {
     $vd = $1;
     next;
 }

 if ($vd =~ /^[0-9]+$/) {
     if ($line =~ /^State: (\w+)/) {
         $vds = $1;            #TODO: verbose print("State for VD #$vd is $vds\n");
         $perfdata = $perfdata." VD$vd=$vds";
         if ($vds !~ /^Optimal$/) {
             $errors = $ERRORS{'CRITICAL'};
             $status = "RAID ERROR";
             #TODO: verbose print("Error found: $status. Skipping remaining Virtual Drive tests.\n");
             last;
         } else {
             $vd = ""; $vds = "";
         }
     }
 }
}
close(MROUT);

# Check state of Physical Drives
my $count_type;my $pd = "";
my $pds = "";

open(MROUT, "$SUDO $MEGACLI -PDList -aALL -NoLog|");
if (!<MROUT>) {
 print("Can't run $MEGACLI\n");
 exit $ERRORS{'UNKNOWN'};
}

while (<MROUT>) {
 my $line = $_;
 chomp($line);

 if ($line =~ /^Device Id: (\d+)/) {
     $pd = $1;
     next;
 }

 if ($pd =~ /^[0-9]+$/) {
     if ($line =~ /^(Media Error|Other Error|Predictive Failure) Count: (\w+)/) {
         $count_type = $1;
         $pds = $2;            #TODO: verbose print("$count_type count for device id #$pd is $pds\n");
         $perfdata = $perfdata." PD$pd=$count_type;$pds";
         if ($pds != 0) {
             if ($errors == $ERRORS{'OK'}) {
                 $status = "DISK ERROR";
                 $errors = $ERRORS{'WARNING'};
             }
         }
     }
 }
}
close(MROUT);

# Got here OK
#
my $t_end = [gettimeofday];
print "$status| time=" . (tv_interval $t_start, $t_end) . "$perfdata\n";
exit $errors;


sub print_usage
{
 print "Usage: $PROGNAME\n";
}


sub print_help
{
 print_revision($PROGNAME, '$Revision: 142 $ ');
 print "Copyright (C) 2007 Westfield Ltd\n\n";
 print "Check Dell/MegaRaid Disk Array plugin for Nagios\n\n";

 print_usage();
 print <<USAGE
-V, --version
    Print program version information
-h, --help
    This help screen


Example:
    $PROGNAME

USAGE
;

}

After installing the script above and changing the paths to match your system, edit your sudoers file (sudo /usr/sbin/visudo) and comment the following line:

# Defaults requiretty

If you are doing NRPE checks, the line above will prevent the script from running sudo because there is no TTY associated with it. There is probably a way around it that doesn't involve disabling this security feature - if you find out please tell me.

While in the sudoers file, also add the following two lines:

nagios ALL=(ALL) NOPASSWD: /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -NoLog
nagios ALL=(ALL) NOPASSWD: /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL –NoLog

If you run NRPE with a user different than "nagios", change the lines above to match it.

That is it, basically. Before adding it to your NRPE checks, give it a try:

$ sudo -u nagios /usr/local/nagios/libexec/check_dellperc
PERC OK| time=0.189185 VD0=Optimal VD1=Optimal PD0=Media Error;0 PD0=Other Error;0 PD0=Predictive Failure;0 PD1=Media Error;0 PD1=Other Error;0 PD1=Predictive Failure;0 PD2=Media Error;0 PD2=Other Error;0 PD2=Predictive Failure;0 PD3=Media Error;0 PD3=Other Error;0 PD3=Predictive Failure;0 PD4=Media Error;0 PD4=Other Error;0 PD4=Predictive Failure;0 PD5=Media Error;0 PD5=Other Error;0 PD5=Predictive Failure;0

It should return a status of zero (unless, of course, your RAID is b0rken):
$ echo $?
0

It is not a toy

2008-04-14T19:06:00.002+10:00

From the book "Things that work but you wouldn't have bet on before actually trying".

$ mplayer your_movie.iso

Teamsite 6.7.1 SP1 won't start

2008-01-31T10:32:00.000+11:00

After installing Teamsite 6.7.1 SP1 on a Solaris 10 machine I tried starting the service to check out the new version. Sadly, the service wouldn't start, dumping a core and printing this message:

[crit] file vhost.c, line 190, assertion "rv == APR_SUCCESS" failed Abort - core dumped /app/teamsite/iw-home/iw-webd/bin/iw.webd start: iwwebd could not be started

In my case, adding "dns" to the "hosts:" line on /etc/nsswitch.conf solved the problem:

hosts: files dns

A little bee tells me that you can also edit /iw-webd/conf/iwwebd.conf.template and change "_default_" on the VirtualHost entry to "*":

<VirtualHost _default_:__IWWEBD_HTTPS_PORT__>

to

<VirtualHost *:__IWWEBD_HTTPS_PORT__>

I didn't try it but that's also the recommendation from an Interwoven's KB article (support account needed).

TCP wrappers: refused connect from ...

2008-01-31T10:30:00.000+11:00

I've used inetd + tcp wrappers + netcat a number of times for migration of TCP-based services to a new server. It goes something like this:

Get the service running on the new box
Point the DNS entry (or IP address of the server on clients) to the new server
Stop the service on the old box
Enable the redirection using inetd

For number 4 and HTTP redirection, an entry like the one below in your /etc/inetd.conf is usually enough:

http stream tcp nowait nobody /usr/bin/tcpd /usr/bin/netcat new-server 80

You then leave the old server running until no more clients connect to it. I do that by inspecting the syslog entries and looking for the netcat redirections. Last time, however, I was seeing these:

Jan 30 14:20:04 old-box netcat[16769]: [ID 947420 mail.warning] refused connect from 189.201.77.65

And sure enough, I started to get complaints that some clients were no longer able to connect to the service. I had left /etc/hosts.allow empty on purpose since there was no need to restrict the service to specific hosts.

After some digging through the tcp wrappers readme, I suspected that the version of tcpd on this SunOS 5.8 (Solaris 8) had been compiled with -DPARANOID. If defined, PARANOID will cause tcpd to reject hosts whose IP address don't resolve to a name (using reverse DNS).

I downloaded the tcp_wrappers source, recompiled without -DPARANOID and installed the newly compiled binary. The refused connection entries were gone from the log and the clients confirmed they were able to reach the server once again.

MSMQ error:0xc00e0027

2008-01-09T10:58:00.000+11:00

After installing windows 2003 Service Pack 2 on a production box I started getting errors 0xc00e0027 when the application tried to access the queues. A number of related articles gave different solutions but what worked for me was reinstalling Message Queuing from the Control Panel.

I had to re-create my queues too, since they were wiped-out when I removed the component.

VMWare Workstation on Debian AMD64

2008-01-08T20:24:00.000+11:00

I've just upgraded to the x86_64 Debian architecture (AMD64). When installing vmware workstation for x86_64 it complained about some missing libraries but the installation finished nevertheless.

However the application would not open any VMs, spitting this error:

/usr/lib/vmware/bin/vmware-vmx: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

Turns out some of the VMWare binaries need 32bit libraries, even on the 64bit version. This post on vmware's knowledge base gave a solution for Red Hat distros. On Debian the solution is analogous: you just need to install the package ia32-libs.

You will need to re-install VMWare so that it regenerates the vmmon kernel module. The vmware-any-any patch is not needed.