ckdake.com

Debugging a (particular) failing boot service on Linux

ckdake — Fri, 22 Jan 2010 00:51:22 +0000

At work I recently rolled out a newer version of the Dell OpenManage tools which included for the first time a build of Openwsman. We didn't specifically need this functionality, but it's good to stay current with the OpenManage tools. To load in the (unrelated) new kernel on a test machine, I rebooted the machine using Cobbler's power management functionality on our administrative system, but after 5 minutes the machine was still not responding to pings so something was broken. I used remote desktop to hop on our one Windows server in the datacenter which we use to get at the interactive consoles of our servers (Thankfully the new DRAC6 card's have console applets that work on Macs!), and pulled up the console for this machine.

The boot process was hung on "Starting openwsman" and didn't seem to be doing anything. Doh!

I restarted the machine again, and at the grub boot menu added a "S" to the boot string to start up the system in single user mode, and booted things up. "chkconfig openwsman off" to disable the service, and another reboot to get the machine back up and running to let me troubleshoot a little better. I took a look in /etc/init.d/openwsman to see what might be hanging, and nothing immediately looked suspicious. It was a pretty standard init script, with the extra feature of generating OpenSSL certificates if they didn't exist already:

if [ ! -f "/etc/openwsman/serverkey.pem" ]; then
        if [ -f "/etc/ssl/servercerts/servercert.pem" \
                -a -f "/etc/ssl/servercerts/serverkey.pem" ]; then
            echo "Using common server certificate /etc/ssl/servercerts/servercert.pem"
            ln -s /etc/ssl/servercerts/server{cert,key}.pem /etc/openwsman/
        else
            echo "Generating Openwsman server public certificate and private key"
            FQDN=`hostname --fqdn`
            if [ "x${FQDN}" = "x" ]; then
                FQDN=localhost.localdomain
            fi
cat << EOF | sh /etc/openwsman/owsmangencert.sh > /dev/null 2>&1
--
SomeState
SomeCity
SomeOrganization
SomeOrganizationalUnit
${FQDN}
root@${FQDN}
EOF
        fi
    fi

It's a little strange, but not unheard of practice to do this, and shouldn't cause any problems. (Puppet and Func, two other systems tools we use, generate their certs in the application which is a lot more common.)

I extracted the only possible culprit from the owsmangencert.sh script and tried running the openssl command manually:

openssl req -days 365 $@ -config /etc/openwsman/ssleay.cnf \
  -new -x509 -nodes -out cert.out \
  -keyout key.out

and it seemed that this was indeed the problem. It just sat there and didn't complete with the speediness I expect from OpenSSL. Time for strace!

cat << EOF | strace openssl req -days 365 -config ./ssleay.cnf.2    -new -x509 -nodes -out cert.out   -keyout key.out
> --
> SomeState
> SomeCity
> SomeOrganization
> SomeOrganizationalUnit
> test
> root@test
> EOF

This ended up doing a long read with output like:

open("/dev/random", O_RDONLY)           = 3
read(3, "\323K\372u_ya'\27\266\320\25\22\373\240\330~'\224\310\243\356\225\350.\245\362\3058\230Zb"..., 1024) = 128
read(3, "K\7:\273Zdr\274\25\227\263\366\260U\337Owp\6y\2333c\361\322\334\217\370.k\375]"..., 896) = 128
read(3, "dH\375V\327\230Bi\221\342\326\26R\301v^Qv5f\347\303g7\2747\345\360\207A!\227"..., 768) = 128
read(3, "X&\254r\331\353<:\36!\333\340\353", 640) = 13
read(3, "\357F\27\347\372atf", 627)     = 8
read(3, "\231\347\232\362\345\215n\227", 619) = 8
read(3, "\324\304\323\30\325\10G\332", 611) = 8

Looks like /dev/random wasn't returning random data nearly fast enough, which makes a whole lot of sense! /dev/random is "good" random data because it is based on environmental entropy and the entropy data is only used once, but on a modern multi-core systems doing lots of things, there usually isn't much entropy available. That means that while this command would eventually finish, it could take a very long time.

The fix: using /dev/urandom instead. It is "not quite as good" random data because the output may have less entropy than /dev/random, and it uses internal entropy bits multiple times to generate it's output, but it's "good enough" for generating cryptographic keys. And, it is non blocking which means that a caller will never have to wait inane amounts of time for enough "random" data. (See http://en.wikipedia.org/wiki//dev/random for a longer explanation.

I replaced the two occurrences of /dev/random, one in /etc/openwsman/ssleay.cnf and one in /etc/openwsman/owsmangencert.sh, and initial startup of openwsman (including key generation) became pretty instantaneous. "chkconfig --levels 2345 openwsmand on" to turn it back on, and a reboot (after removing the generated keys and certs) to confirm, and the machine booted up as expected. To make this work everywhere, I customized those two config files and added them to our Puppet system so that all Dell servers would get Openwsman set up properly when the update is run globally:

 file {
    "ssleay.cnf":
      path => "/etc/openwsman/ssleay.cnf",
      source => "puppet://$server/dell/ssleay.cnf",
  }
 
  file {
    "owsmangencert.sh":
      path => "/etc/openwsman/owsmangencert.sh",
      source => "puppet://$server/dell/owsmangencert.sh",
  }

Problem solved and all machines will automatically get the correct fix, so the next time a machine won't finish starting up, it will be a new and different problem to debug.

Snake Creek Gap Time Trial 2010 #1

ckdake — Mon, 04 Jan 2010 16:31:19 +0000

Last year, I rode the 17 mile version of the Snake Creek Gap Time Trial (and wrote about it here) on a full suspension bike in January at 50F. This year, I rode the 34 mile version on a hard-tail bike at 25F. I've written about not enjoying mass start, short, XC races, and SCGTT is definitely a different sort of beast. Spaced individual starts so there is no constant battle to stay out in front, and completion time instead of crossing the line first is what counts so passing is generally pretty spread out and done safely at good places. Additionally, the section of the Pinhoti that this race takes place on is extremely brutal with lots of climbing (6000+ feet on the 34 mile) and miles and miles of technical rocky sections that had most people walking. The 34 mile version had the special treat of a knee high stream crossing:

Photo taken one mile into the course (and at 22F) by mtbepic61.

Some people did what I did and had a dry rest of the day. Other people attempted to put bags over their shoes or ride it and many of them were wet and miserable for the rest of the day, experiencing fun things like frozen solid rear derailleurs. The only thing that froze up on me was my camel-bak hose, which was new and surprising, but made a lot of sense, and was fixed by re-routing the hose under my jersey. My ride started at around 22F at around 10am and finished up over five hours later at around 28F and I finished 14th of 21 people in the male 30-and-under 34-mile category, including 3 or 4 people that didn't finish.

As awful as this race sounds, I actually had a lot of fun! I took this event as a "pre-ride" of the 34 mile course because the first half was new to me and I'll have two more chances to push for a better time, so I pretty much took it easy riding the entire time and even stopped to give a guy a tube and let him use my pump to fix a flat. This was the longest time I've been in the saddle on a mountain bike which was fine, and the coldest weather I've ridden in which was great. Smartwool socks, neoprene toe covers and keeping my feet dry kept them nice and warm, Smartwool glove liners under my usual cold-weather bike gloves kept all my fingers happy, and by the end of the race I was down to (and plenty warm in) a longsleeve Smartwool base layer under a short-sleeve wool jersey on the top and insulated tights on the bottom. I've heard stories from other people about feeling horribly cold and getting frostbite on fingers and toes, but I think it mostly comes down to clothing and a tiny bit of luck. The only uncomfortable part of my ride was lower back pain from an extra liter of water or so in my camel-bak that I didn't need and the extra clothing I packed and removed while riding. These things would have saved me from freezing if I had a nasty mechanical and had to walk out, but my back just wasn't use to carrying them.

Unlike last year, I remembered to eat enough and felt pretty good energy-wise until the end, but it does make sense to be kind of worn out after 5 hours of riding. I was pretty tired afterwards and slept like a rock, but got up Sunday morning feeling pretty good and had the energy and the legs for a 5K run.

I'm looking forward to riding this course again in February and March, and I'm trying to figure out what my next mountain biking adventure will be that can outdo the Snake.

2010 Goals

ckdake — Sat, 02 Jan 2010 00:24:29 +0000

2009's goals were all pretty reasonable and it worked out pretty well, so I'll make things a little crazier for 2010 and see what happens!

Personal

Save over 20% of my post-tax income
Double ithought revenue
Pay off a big chunk of my mortgage so I can get rid of PMI and escrow requirements
Take a big vacation somewhere new
Make significant progress on programming my exercise tracking tool, and open it up to a public beta

Photography

No more 365, and no more flickr uploading. Use a "stream" album here for uploading from my phone and one-off photos: ckdake.com/gallery/2010/stream
Submit a photo to a DPChallenge challenge every week. I bought a paid subscription today, so I have a few more challenge choices each week.
Print a few 8x10s or larger and frame them

Biking

Do a few more century rides (hospitality, macc one love, tour de cure)
Mountain bike at 10 new places
Get points in a Pro Series event at the velodrome
Do some team pursuit and madison racing at the velodrome
4day+ trip of mountain biking in another state
Ride 150 miles in one day

Running

What? Running. yeah... 30 min of running in 35F weather beats 2 hours on a bike in 35F and supposedly has similar overall fitness benefits, and should help out biking a little bit
Get a sub-54 min qualifying time in a 10k before Peachtree Road Race registration
Run the Peachtree Road Race
Run the Atlanta Marathon

SugarCRM

Build a new app deployment system for dev/stage/test of our production web properties
Finish moving all system configuration into puppet (major things left to go: MySQL replication, automatic building application server groups using puppet stored configs)
Finish removing all legacy hardware, legacy VLANs, legacy IP space, and legacy routing policies
Implement password synchronization between dataenter LDAP and office AD systems
Build a new application server logging system using Spread
Build a Hadoop cluster and use it for generating real reports
Go to Velocity 2010 and SugarCon 2010

2009 In Retrospect

ckdake — Fri, 01 Jan 2010 20:54:32 +0000

I did a small writeup of 2008 last year, and set a few goals. Looking back, 2009 was a pretty big year and my goals were pretty reasonable! Here are some results:

In General

ithought total revenue didn't double, but revenue from yearly hosting customers doubled which is currently the most important area for growth to me.
ithought got a little more legit by becoming a registered Sole Proprietorship in Atlanta
Saved 22% of my post-tax income
Continued getting rid of things that I don't need by selling, donating, and giving away
Almost finished furnishing my house, all that really remains is a mattress
Switched from Linux+Chander+RAID for keeping things organized to Mac+Drobo+Things
69,527 visits to ckdake.com

Photography

Started shooting in RAW on my DSLR, and started using Adobe Lightroom to process and upload things.
Successfully completed "Project 365" - a photo a day, every day, for a year: ckdake 365
1995 photos uploaded to ckdake.com/gallery

Biking

Got in the habit of eating a lot better which got my cholesterol levels from high to normal
Full season of bike racing: Velodrome, SERC, GSC, and Snake Creek Gap. (Head to my bikes page and click on "Click here for 2009 Race Results" for the full season results.)
4 days of mountain biking in Santa Fe
Lots of mountain biking including 18 new places, some listed here and a writeup of one weekend of riding
Completed Six Gap Century, the "hardest" Century in Georgia, with over 11,000ft of climbing.
Didn't ride more miles than 2008, but I rode almost 5x as much at the Velodrome and almost twice as much in the mountains compared to 2008.
Got a bike fit at 55Nine on my old mountain bike to get measurements for a new mountain bike, and got fit on my track bike to squeeze a tiny bit more performance out of it.
Ran the final (and highly successful) FM24 and afterwards, co-wrote the goodbye letter at 24.fastermustache.org
Finally got a hard tail mountain bike, the likes of which have been on my mind for years: 2009 Independent Fabrication Steel Deluxe and sold my full-suspension bike.
First 200k ride, on the Silver Comet with Jim and Paul

SugarCRM

Second full year down!
Spent a week in California going to SugarCON and Lake Tahoe
Converted systems from VMWare to OpenVZ and built a management system for OpenVZ Containers, saving money in licensing and fitting more services on less hardware. writeup on VLANs in OpenVZ, HOWTO for moving MySQL slaves into OpenVZ containers
Built out a new cluster in London using the provisioning system in California over a site-to-site VPN, and running fault-tolerant replicated LDAP and DNS
Converted loadbalancers from NetScalers to clustered nginx servers using wackamole, saving money in needed upgrades and allowing much greater monitoring insight and tuning capabilities. Monitoring nginx with ZenOSS.
Lead moving all private web allocations from the public internet to internal names and addresses, including lab testing of network device configurations including Hairpinning with a Cisco ASA.
Attended no:sql(east) 2009 and learned some cool stuff that I've been tinkering with for data analysis (using Apache Pig).
Ported my personal phone number to Sugar's account and consolidated from a work Blackberry and a Personal iPhone with 2 separate numbers to one iPhone 3GS with one phone number.

That's it for 2009, and my list of things for 2010 should be up in a post later today!

Tour de Cure 2010

ckdake — Thu, 17 Dec 2009 15:32:55 +0000

To add to my bike riding repertoire for 2010, I'll be riding in the American Diabetes Association Tour de Cure in Atlanta in May to help raise money for Diabetes research. I'll be riding 100 miles with some friends at Lorenc+Yoo Design and we're hoping to raise $1000. Please consider visiting my personal page to make a donation. Every bit helps, and this is a great way to contribute to a good cause. Donations are tax deductible, so getting one in before the end of this calendar year might not be a bad idea either!