sam.github.com

Blog

2012-01-26T00:00:00-08:00

I’ve got some server maintenance at midnight tonight. In the mean-time I’m up here at work twiddling my thumbs with three hours to go. Yeah, there’s work things I could be doing, but I’ve already put in more than a full day and I’ve got a cold. :-p

I’ve looked at HackerNews already.

So I’m going to do something. Write some code.

I like Jekyll. Mostly. It’s just that the whole pull, type-out-an-awkward-filename, write some YAML, write some textile, commit, push dance is enough of a speed bump that I don’t post enough. At least that’s my excuse.

So I’m going to try my hand at whipping up a blog. There’s lots of things I could write it in. Node.js is something I’ve been meaning to give a go. ngx_openresty looks really interesting. But for tonight I’ll see what I can accomplish in three hours and stick with Ruby. I could go with Harbor, but I like Sinatra’s router, and I think this is something I can manage in a single “controller” so I’m going to give that a shot. I could persist to PostgreSQL and Sequel which is always a winning combination. But I think I’ll go with Redis this time around. It’s fast, and ought to suit the blog-post format well. At least initially. Not sure of the best fit, but I’ll try a key for each post, and a List to track/retrieve them.

Follow along here, and with any luck I’ll have something in the next few hours. :-)

I’ve got a disused Linode instance already, so if it works out, I’ll deploy, update DNS and we’ll see how it takes.

My Biggest Shell Project

2012-01-19T00:00:00-08:00

‘Tis Bash really. Because it’s the default, or easy to get on FreeBSD, OpenBSD, OSX, Ubuntu and Solaris.

So how many lines is it?

18,865

Yes. I wrote this post just for that. Vanity. Sue me. ;-)

It’s a big collection of scripts to automate server configuration. It spans the gamut from compiling NGinx, to configuring LACP on FreeBSD, to generating HAProxy entries, to cloning git repositories and mounting NFS shares. Pretty much anything you’d need to web-servers, load-balance them, setup replicated database servers, etc is in there.

Why not Chef or Puppet?

Because they’re complex. :-) Force yourself to learn shell-scripting, and soon enough you can fire off a script like the Github Migration in the previous post in a few minutes, saving days of manual labor when you’re migrating over 100 projects. You might be surprised at just how often you use it. I couldn’t imagine doing half of what I used to do manually or with Ruby now.

I mean, Ruby is great in many many ways. But it’s no substitute for being able to fire off a quick for-loop right where your cursor is now.

So if you decide to tackle one new language this year, and you spend a fair about of time on *nix or *bsd like systems, I heartily recommend Bash.

By the way, I named that system’s administration project after this beer. Gubna. ;-)

Migrating to Github

2012-01-19T00:00:00-08:00

There’s probably thousands of ways to do this out there, but here’s mine. A quick little shell script I wrote up to migrate a repository from a local gem-server up to a Github Organization, including your branches and tags. If you give it your login details it’ll create the remote repository if it doesn’t already exist.

Once you tweak the exported variables at the top, you call it like so: moveit my_repository_name

ls Filters

2012-01-19T00:00:00-08:00

Just a quick note. You might sometimes find yourself wanting to filter a list of files on your system. We’ll use a clone of this blog for an example. You could use grep:

Regexps are obviously very powerful, and I often use something similar in my own scripts.

For simple cases like this however, you might want to get familiar with the built-in Bash wildcards:

Quick and easy. Hope you find it useful.

SmartOS Setup

2012-01-16T00:00:00-08:00

Here we’re going to go over the basics of getting the latest SmartOS release (January 12, 2012 as of this post) running. Here’s a screen-cast for the video inclined, but I’ll also provide all the tweaky bits with some more detail below so you can follow along at home.

You’ll need an Intel CPU with the VT-x extensions assuming you want to use KVM virtualization on SmartOS as it doesn’t support AMD CPUs. You’ll get VT-x support on any 2nd Gen i3 processor, the i5 or i7 processors, or a Nehalem or later Xeon. This should cover most any Intel system you might have at your office as long as it’s not more than a couple years old. You can find out for sure on Intel’s Virtualization Technology List page.

If you’re not already familiar with ZFS, I’d suggest downloading the latest FreeBSD, running it under VirtualBox, and playing around with it there. For 99% of what you’ll be doing, aside from standard device-name conventions for disks, it’ll be identical to what you’ll be doing in SmartOS. You might also want to buff up on the Wikipedia article on ZFS.

For everyone else, on with the show!

Single Disk Configurations

First, an issue with SmartOS configuration common to all installations. No swap is created. If you create a zones/swap device, it’ll be used automatically at boot. The device should be a zvol, the same size as your physical memory. So in my case:

The primarycache setting is your ZFS Adaptive Read Cache (in-memory cache). It obviously doesn’t make sense to cache data stored on a swap device since if you had memory to cache it, you’d be better served using that memory for the data directory instead of pushing it out to disk.

The secondarycache is the ZFS L2ARC, ie: a cache device (SSD) attached to the pool. Again, caching swap doesn’t really make much sense, so make sure that it’s not.

I’ve seen it mentioned on IRC that disabling the caches for swap is automatic, but without confirmation, so no harm in making sure.

Multiple Disk Configurations

Now we move onto multi-disk configurations. If this is you, then hold off on adding the swap above. We’ll get around to it in a bit after a few other things. The SmartOS configuration script has a bug that ignores extra disks you pass it. It’ll only use the first.

On our staging system we have a JBOD with 24 drives on it, and 3 drives in the server itself. We want to use that JBOD for our zones zpool. Because you can’t remove a top-level vdev in ZFS, you’re in a bit of a pickle though. If you use one of those disks right off the bat, you won’t be able to get the end result/configuration you want. Unless the disk you use is ultimately intended to be a spare as you can add/remove those at will.

So if you’ve already configured SmartOS, you want a redundant multi-disk configuration (mirrored, raidz, etc) and you don’t have a spare disk, this is probably where you stop and go dig up some old disks you can temporarily use while we shuffle things around.

If you’ve already configured your install, and you chose one of the disks you ultimately want to use in your pool, then you’ll need to delete it, and reconfigure it to use one of your spare disks instead.

The easiest way to blow the pool away so you can get back to the configuration is to run the format command, and blow away the partition on the disk with fdisk.

Making Things Complicated: As an alternative to the fdisk/reboot, you could swap out the disk by adding your spare as a mirror, then removing the original disk like so:

Once that’s done and your running setup is on a spare disk, we can continue.

First up, you can’t configure a dump device on your pool if:

It’s going to include log or cache devices
If it’s going to span multiple raidz vdevs

So you’re going to want to dedicate a raw physical device. As an aside, because of an issue in a critical service (vmadmd) when you have more than one zpool, you can’t put the dump device on a secondary pool. You’re going to want to make sure your system only has the “zones” pool when all is said and done, so that’s why we use a raw device here.

You can’t use this device for anything now else mind you. So in the multi-disk setup you need:

The initial disk you’ll install to (this could be a device you don’t intend to use, or it could ultimately be intended to be add to the pool as a spare)
The disk for the dump device
The disk(s) for your multi-device “zones” zpool

Now we’ll create the multi-device zpool that we’ll eventually turn into our “zones” zpool:

Note, this is a JBOD in my case. Each of these is a 1TB 7200RPM disk, so this will create about 15TB of usable space after subtracting the spares (3TB) and parity (6TB).

It’s really a bit of a silly setup for a production VM host.

If you need that much storage for guest content, it should probably be on NFS or something so you can eventually have redundant, load-balanced guests. If you needed it for performance, you’d use smaller, higher-RPM disks in conjunction with SSDs.

I’m just using this setup here because this is a host for our non-production staging sites. By comparison the production setup consists of twelve 10,000RPM 300GB disks (two of those reserved for spares, and the other ten split into two raidz sets), two mirrored 50GB SSDs for log devices, and one 100GB SSD as a L2ARC cache device. A lot less storage space, but a lot more performance (and guest content is on NFS anyway, this is just for OS installs, swap, logs, etc). When you’re running a lot of VMs, disk IO performance is going to be pretty high up there when it comes to keeping things snappy.

So now we have our “backup” zpool. Let’s replicate our data to it:

Now that our dump device is off of the “zones” pool, and we’ve backed up our install, we can delete it:

Finally, we’ll export our “backup” pool, and reimport it with the “zones” name:

If the disk for the original configure-time “zones” pool was intended as a spare, and you had left it off earlier in the zpool create backup since it was in-use, now is when you’d add it back:

Now we’ll loop back around to the issue we mentioned at the beginning, and create the swap device that the configuration script missed creating for us:

Finally your zpool is how you wanted it. All disks where they belong, and a dedicated dump device humming along. Go ahead and reboot now to ensure everything’s properly mounted, and afterwards ensure that swap -l lists your swap device and dumpadm displays your dedicated device.

If you run a zfs list you’ll probably notice that the backup retained a zones/dump file-system. That’s no longer used since we moved it to a dedicated device, so let’s do a little house-cleaning and delete it:

Ultimately we’ve ended up with something like this:

Now you’re ready to rock, creating your own base image, and provisioning new guests. You’ll find help on that in the SmartOS wiki article: How to create a Virtual Machine in SmartOS.

SmartOS Enhancements

2012-01-16T00:00:00-08:00

Here’s a list of things, in no particular order, I’d like to see in SmartOS:

FIX: Create zones/swap during configuration
FIX: Create raidz vdevs during configuration (pending pull request: https://github.com/joyent/smartos-overlay/pull/4)
ENHANCEMENT: Provide ability to specify dump device
ENHANCEMENT: Provide ability to define “aggr” (LACP) interface during configuration
FIX (low priority): Patch vmadmd/sysinfo to allow multiple zpools
DOCUMENTATION: /opt/custom examples and use-cases

That’s about it off the top of my head. Most of these things have work-arounds. I wouldn’t consider any of them blocking issues, but they’re the things I most frequently run into. If I find the time I’d like to contribute back. Most of these appear to just require a little scripting knowledge, and maybe a bit of node.js.

GeminaboxPlus

2012-01-13T00:00:00-08:00

I just released a quick project to help you get quickly started running your own local Gem server. It’s faster than going out to Gemcutter all the time when you’re bundling a project and you can keep private gems private. The Gemcutter project itself assumes Amazon S3 integration, so Geminabox is really a better fit for most local installations I think.

GeminaboxPlus is just a default app for Geminabox with a patch to move reindexing offline to a Resque worker. Geminabox is great on it’s own, and easy to setup; it’s only real weakness is when you have a lot of Gems (we’ve got ~2400 on our system at the office) because then reindexing takes quite awhile.

How do you get to 2400 Gems? You push everything you need for your projects to your own Gem server. ;-) That way you don’t have to worry about gems you depend on dissapearing at some point in the future, and you don’t need to source http://gemcutter.org in your Gemfiles so your bundle installs are much quicker.

With this you can have a solid setup going in just a few minutes. If you want to increase your project’s Bundler performance, or could use a private Gem server, there’s really no excuse not to just clone and fire up your own. :-)

The README has the full details, but if you want the 2-minute-and-running version I’ve copy/pasted the meat here:

SmartOS Guest Provisioning

2012-01-12T00:00:00-08:00

Here’s a quick screen-cast demonstrating provisioning in SmartOS. I restarted a dozen times to try to keep from rambling. ;-)

If you’re new to SmartOS you can start here: http://smartos.org.

This demonstration assumes you already have a base image setup that you’re happy with, and you’re ready to go to town cloning up new Guests based on it.

NGinx ModZip

2012-01-12T00:00:00-08:00

I had trouble getting NGinx 1.0.11 building under Ubuntu 11.10 yesterday. We use a module called ModZip and it appears GCC 4.6.1 (and maybe Ubuntu’s default flags for it) has introduced some checks to detect if a variable is defined and set, but then unused. This causes build errors.

So I dove in to fix it. I’m not a C developer, so my first attempt messed it up. I removed an iconv call providing a bit of critical functionality because the result wasn’t used. What I missed was that two pointers were passed within the call. So here’s my second attempt. This one fixes the build, and doesn’t break anything.

With this patch I can build everything on GCC 4.6.1 and keep ModZip around, which is critical to many of our projects. Here’s why:

ModZip operates similarly to X-Send-File. You might have a fixed, or limited pool of application servers. Especially in a single-threaded environment like most Ruby deployments. So if someone requests to download a 1GB file that’s not in your web-root you’d have to tie up an application server for the duration of the download. If the client has a slow connection this could be a good long while. Enough clients request such a download, and your small pool of application servers can quickly be exhausted, effectively causing an unintentional Denial Of Service.

Instead of this, X-Send-File allows you to set a header that tells the upstream web-server (Apache, NGinx, etc) where the file is that you want to serve, and what name to give it at download time (many of our files are represented by UUIDs internally for example, though we want something friendly and descriptive to be used for downloads). That way your application server is just setting a header and quickly returning back to the pool to serve other requests. The upstream web-server then reads and strips the header, and does the actual sending of the file.

ModZip is similar, but for (you guessed it!) zip files. You pass multiple paths/filenames in a special header in your application server, and then it, handles creating a zip file and streaming it to the client. While your application server would first need to zip everything up before sending a single byte to the upstream, ModZip can almost immediately begin streaming the zip. This frees up your application server, and ModZip efficiently tackles the rest for you.

It’s a great tool. I definitely recommend it. With this patch, you should have no trouble getting it going in your own NGinx setup.

Bootstrapping

2012-01-08T13:00:00-08:00

I’m finally getting around to bootstrapping this site. Bootstrap itself seems fairly easy so far. Figuring out the best way to override some of the defaults is going to require an actual bit of reading. It’s all there at the project home-page I think; I just glossed over it the first time through.

Reproducing an existing layout is going to take some practice. Otherwise it seems like a nice framework so far. The basic layout didn’t take any thought about floats, clearing them properly, or any effort at all really. I wonder how well the abstraction holds up with something more complex.

Jekyll and Bootstrap

2011-12-13T23:08:00-08:00

Overall I like Jekyll so far. It’s simple and that counts for a lot. I’m wondering how I’ll implement a live-search for the site, but something for the back-burner.

I really prefer Textile syntax to Markdown, so I’ll be making that switch ASAP. With this post actually. We’ll see how that works out.

Other than that, I want to see about reimplementing this design with Twitter’s Bootstrap framework just to get familiar with it. The markup in this design is actually pretty nice in my opinion, but it’s an excuse to give Bootstrap a spin.

That’s about it for tonight. Let’s push this and see how it looks!

Chocolat to TextMate

2011-12-13T23:00:00-08:00

I’ve been using ChocolatApp for about a month now. It’s pretty good. I’m going back to TextMate (1.x) though.

Likes

Out of the box it’s prettier, with a standard Drawer, more polished Find
The default set of light-on-dark themes feels a little more robust
The “Active” documents area is a bit more functional than tabs when you’re jumping around between a lot of files

Dislikes

TextMate will auto outdent standard closing syntax structures for you, like “end” in Ruby or “fi” in Bash; Chocolat, not so much
There is no “block-comment” functionality for the Shell Script syntax, which I spend a good bit of time in
There is no bundle editor for simple fixes
You can’t tab over to the “Don’t Save” option when closing an unsaved file (I find myself hitting CMD+T a lot inadvertently, which in Chocolat creates a new blank file/tab)
It’s text-selection seems a bit off, doesn’t respect periods in hostnames for example, so a keyboard OPTION+SHIFT+ARROW highlight more often than not ends up highlighting way too much and you have to bang at the arrow keys a lot to compensate

Regarding the bundle editor, I’m no tweaker, but it would be nice to fix some of the more glaring oversights, like the lack of a shortcut for the HTML5 doctype snippet in the HTML bundle.

So for now, until the Chocolat folks sort out some of the issues, I’m heading back to TextMate. Overall I do think Chocolat is a significant improvement over TextMate in most ways, but the ways in which it falls short are really annoying/distracting.

Hello World

2011-12-13T22:25:00-08:00

This is my first post with Jekyll.

I’ve used sirupsen.github.com as a design basis for this blog for now. Let’s see if I can reboot this thing.

My "Fail Rated" Jeep Grand Cherokee

2010-12-29T00:00:00-08:00

Days in the shop/Days owned:

44/75

I drove a 2009 Volvo C70 Convertible. I’ve owned and loved several Volvos. The dealership always treated us right. No complaints. Enter Jack.

This little guy needed something more family friendly. My wife has a Volvo XC90, which is truly a great family hauler, but it’s a lease and we were burning through the miles far too quickly. My next car needed to be a purchase so we didn’t have to worry about lease exits and someday maybe even pay it off. ;-)

This commercial got me into the Jeep dealer. No kidding, it’s the best commercial I’ve ever seen. After the test-drive I was hooked.

October 15th, 2010

A couple weeks later my wife, Jack and I were able to make it down to Dodge City Dodge. I spotted a 2011 Jeep Grand Cherokee Laredo X.

It had a V8. Everything I wanted but a sun-roof really. It was kitted out with all the off-roading features. I’ve never been off-roading before, but the idea of camping with the family, maybe hitting some light trails was exciting. If I was going to give up the Convertible, I was going camping damnit.

We were comparing against a Volvo XC60 we had already worked out the numbers on with the Volvo dealer. After tacking on the extended warranty to get rental coverage (which we’d always got from Volvo for free), the Jeep came in a few (within $10) bucks cheaper. We signed the paperwork that evening and went home with our new Jeep.

So now I was the proud owner of a 2011 Jeep Grand Cherokee Laredo X. It had:

Hemi V8
Leather
Quadra Lift Air-Suspension
Quadra Drive II traction-control system

October 16th, 2010

Camping! The closest OHV (Off-Highway-Vehicle) park I could find was Barnwell Mountain. 2-1/2 hours away. My buddy Scott and I loaded up our sons, and hit the road. We got there just after dark. The next morning we drove around on the access roads a bit, looking for the easiest trail we could find.

At a camp-site just at the mouth of a trail we bumped into a couple friendly guys in an older Land Rover Range Rover, lifted and with some big tires on it. Some of the body panels had seen better days. Clearly the truck had seen some fun. You can just spot it (and one of them standing in front of the Jeep, spotting the trail for us to lead us through) in this picture:

We caught the off-roading bug immediately. Three-wheelin’, crossing creaky old bridges, the dusty red dirt that turned to paint if you ran across a puddle. It was a blast!

Scott traded in his Chevy Silverado and went out and bought a Toyota FJ Cruiser Trail Teams Edition about a month later.

October 23rd, 2010

The very next weekend, right after Jack’s swim-class, my wife, Jack and I went out again. It was awesome. A bit brisk Sunday morning, but a bit of coffee to warm us up did the trick.

We went out on the same trails, took ‘em a bit better this time. My wife decided she didn’t care for the off-roading bit. Three wheels on the ground and leans on rutted trails just weren’t her cup o’ tea. Still, the camping was great.

Sunday evening a thunderstorm rolled in. Things got a bit sloshy getting out, and the under-carriage was liberally painted with the red dust, but we made it out before the worst of it hit. We stopped by Bodacious BBQ on the way out:

View Larger Map

Best Ribs I’ve had. Seriously. If you’re even in East Texas do yourself a favor and stop here!

November 6th, 2010

My wife’s sister joined us and the four of us headed out to Barnwell again for camping. Hit a few more trails with my sister-in-law. Found some more scenic ones this time. Some nice rides. Came up to a rock shelf we couldn’t get past. The Grand Cherokee’s wheel-base was just too long for it to avoid getting high-centered on it. Pretty sure about the only thing that could get past that is a Wrangler. Turning around in the dark is a bit scary. Thankfully there’s spots for just that all over the trails, especially right before challenges like that.

The tent was a bit chilly that evening, but camp-eggs and bacon in the morning made it all right. Jack loves camping since it’s one of the few times we can let him run a bit free in a big area without worrying about cars or him getting into danger.

Sunday my wife decided to give off-roading another shot and we hit the trails.

We drove around on one of the more scenic trails and everyone had a good time. Hit the access roads in the park and decided to strike out in some new territory. Most of the two-diamond trails are pretty easy going for the big Jeep, so when we came across one named “The Angry Jeep”, we turned in, ready to tackle it.

You enter the trail in some little trees, and then just beyond a clearing is a good sized hill. My wife took one look and said “take me back to camp”. Well, the least I was going to do at that point was get a good picture. I put the transmission in Park and took my foot off the brake. The Jeep started rolling back slowly (we were on a bit of an incline at the foot of the hill). Odd, but if I’d ever lived in San Francisco I guess I would’ve known that that’s normal. I put the parking brake on and hopped out to snap off a couple pictures.

Since we had just entered the trail, there was a spot to turnaround just about 20’ back. When I hopped back
in, I put the Jeep in Reverse and started backing up. It was a little sluggish, and needed more gas than
seemed normal, but it was moving so I didn’t think much of it.

Keep in mind that the traction-control knob is set to Rock Crawling, and the transfer-case is in
4-Low. When I say “moving”, I mean just barely. 2MPH maybe. Out on the access roads I didn’t
generally break 5MPH since with all the stones laying about the suspension would hit the bump-stops (it was set to
it’s tallest “Off-road 2” height) pretty easily. No rush though.

A little more gas, we’d probably moved about 10’ so far…

PING!

It sounded like a small rock just ricochetted against a skid-plate or something. I hadn’t heard anything
close out on the trails yet, but I’m new at this so. I hop out and look under the vehicle. About a foot of
clearance with the air-suspension lifting it, so I’m able to get a good look around. No dents. No pings. The
skid plates look fine, no way anything touched. Well, that’s a bit of a relief.

Shortly after we get back to camp we pack up and head on out. When we get back to the drive up to the park,
and hit asphalt again I turn off 4-Low and put the traction-control back on Auto. The drive-train
starts slipping something awful. If I turn the traction-control knob to Snow/Mud it gets moving again.
So issue with the traction-control system I think. I’m convinced something to do with forgetting the parking
brake. Either way, it’s a Sunday, not dealers are open and we’re 100 miles from home. I hit the road and keep
it at 55MPH just in case. Under hard acceleration there’s a knocking sound. Loose brake pads? I don’t know. We
make it home without further issue though.

November 8th, 2010

I take the Jeep into the shop. I explain I forgot about the parking brake when the trouble started so I’m
keeping my fingers crossed that it’s covered. In hind-sight who hasn’t driven a car for a block or two and
forgot about the parking brake? Of course it’s covered.

I also asked them to fit the missing pins for the front air-dam. It’s held on by large thumb screws. The
vehicle was missing a couple from the factory. The problem is they also hold up the felt-like sound
insulation in the wheel wells and since the one on the driver’s side was soaked through and weighed down with
the clay like mud the tire rubbed through it and chewed it up.

I’m out $237.19.

I’m a bit annoyed about that one since it’s not like I lost those pins. They were definitely MIA when I
examined the air-dam during removal. The air-dam is made to be removed for extra approach, so I wanted
to figure out how. It’s not as easy as it sounds since there are some snap-in clips to pop out as well. It’s
definitely not something you’ll be messing with on a routine basis.

Anyways, my service guy Stephen calls me back the next day and says the axle is snapped.

You read that right, that loud ping wasn’t a rock. It was the driver’s side rear half-shaft snapping! :o

The dealer says it clear wasn’t the parking brake and the repairs are covered. The theory of the engineers
they talked to is the wheel got spinning, free-wheelin’, touched down, caught a rock edge or something that
stopped it instantly, and the sudden jarring snapped it.

Take a look at the picture again, I’ll wait. See how all four wheels are on the ground? Remember how I said I
had the vehicle in 4-Low with the most aggressive traction-control setting (rock-crawling) on? Obviously the
engineers aren’t considering all the facts. It shouldn’t even be possible with those settings to reproduce
their theory.

Back to the repairs then. I’m waiting around about a week for the axle to be replaced, and then I get a call.

They’ve removed the axle to replace it and discovered that the air-bag (for the air-suspension) and brakes
also need to be replaced. On top of that the end of the axle in the rear-differential is fused to the
differential housing so the Jeep needs a new differential.

Seriously? You see that there’s a snapped axle and it never occurs to you to pull it out and check to see if
there’s any further damage?!? You’ve got to be kidding me.

Fine. So the part is on order. New vehicle, none have had this issue, the part isn’t in the supply-line yet.
No ETA.

So I’ll keep driving around the rental Chevy Cobalt until I hear back. Next thing I know Enterprise
Rent-a-Car is calling me saying I owe them $200 or something. Remember when I said we bought the extended
warranty for the rental coverage? So I call back the dealer and ask what the deal is. They can’t locate my
warranty. Ok. The vehicle is only a few weeks old by this point. I’ll give them the benefit of a doubt. I tell
him to walk across the shop floor to the sales office, and talk to the finance guy who sold us the coverage.
Stephen says he’ll handle it.

A couple days later Enterprise calls back saying they’re about to bill my card on file.

WTF?!?

So I call Stephen back. He never talked to the finance guy. Fine, I’ll sort it out myself. I call the finance
guy, Julian. He says I’m covered, he’ll talk to Stephen. I call Enterprise up later that day and they say no
one from Dodge City Dodge has called. This is getting old. I talk to Julian again, he follows up with
Enterprise, and it’s settled. After talking to Stephen again I learn that the coverage is only for 5 days in a
rental.

Shame on me for not reading the fine print I guess? Forget that:

I want rentals covered.

Buy this extended warranty and they will be.

We were told the same thing by Mazda a few years ago on our Mazda 6 only to
find out that it required it to be in the shop for diagnosis before we’d get a rental, and that could take
more than a day. I don’t want any gotchas, I just want the loaner.

Don’t worry about it. Full coverage. You bring your Jeep in, you get a
loaner.

5 days of coverage? Unacceptable. Stephen talked to his manager, who talked to Chrysler I understand, and
they got it covered.

Just after Thanksgiving I’m swapped into a Chrysler Town & Country minivan for my rental. Not in time to
keep us from having to put more miles on my wife’s vehicle over the holiday, but still a lot better than a
Cobalt.

December 8th, 2010

The Jeep is ready, I can pick it up. 30 days in the shop. Pretty crazy, but it did sound like a lot of work
and they did eventually get the part in after pulling some strings so I appreciated the effort the dealer went
through to get the thing put back together despite the hiccups with the mechanics. I pay the $237.19 to get
that insulation replaced in the wheel well and the air-dam clips.

I get the Jeep back and I’m in love with it as ever. Still singing it’s praises to anyone who’ll listen.
First thing I do is take it to get it detailed since I had to put it in the shop with the spatter and trail
dust all over it.

December 9th, 2010

It’s been a long week at work, with a couple very late nights. I’m ready for the weekend and some camping. On
the way back to the office after stepping out for dinner and coffee I decide I’d better test the 4WD system
since I didn’t get a chance to the evening I picked it up. In the Starbucks parking lot I put the transmission
into neutral, turn the traction-control knob to rock-crawling, and hit the 4-Low button. Everything engages
smoothly. I feather on the gas and in just a couple feet a red light comes on on the traction-control panel,
my dash is displaying “4WD SRV”, “Service 4WD System” and “Service Air Suspension” lights.

Damnit.

I push the 4-Low button again and turn the traction-control dial back to Auto to disable everything.
No joy. It’s stuck in 4-Low. Ok, now I’m pissed. The vehicle was “fixed” and the mechanic never thought to
confirm it actually worked?

So I work near the “High Five” in Dallas. That’s a 30 mile or so ride home. The manual tells me to keep it
under 25MPH in 4-Low. When I’m done at work for the evening I drive home at 20MPH up Coit Rd with my hazard
lights on the whole way. I get home around 3AM.

December 11th, 2010

Less than 48 hours after picking it up, the Jeep is back in the shop and I’m back in a rental. This time the rental is a Chevy Silverado. So that’s better. Not to knock on the Cobalt, it’s a much nicer little car than I expected, but I’m not paying Jeep Grand Cherokee money to drive around a Cobalt.

Or a Silverado for that matter, but you can’t win ’em all I guess.

A week later or so Enterprise calls me saying they need the Silverado because they’ve sold it. A bit odd, but
OK. I drop it off and exchange it for a Nissan Xterra. The Xterra is a lot more powerful than I expected.
Probably about the same acceleration as the Grand Cherokee I’d guess. It’s towing capacity is much lower but
still, it’s got me questioning if I actually need a vehicle that costs twice as much if I just want a nice
family hauler and camper.

I’m pissed this time. I get my sales-guy involved, as well as the service manager. The Service Manager
(Scott) acts like they did me a favor fixing it the first time since I was clearly “muddin’ it”. Let’s get a
couple things straight:

* I wasn’t “muddin’ it”
* If I had, I would’ve been stuck. The stock “off-road” tires aren’t even AT tires, much less MT tires

I call up Chrysler Customer Care. I emphasize that they need to call my number. The following week they call
my wife’s. She tells them to call me as she’s walking into a meeting, they say OK and we never hear back
again.

I call them again a few days later. I expect to have the same rental coverage issues. They promise a call
from someone in that department before the end of the day, as well as the follow up I never actually got
within 48 hours. I never get the call about the rental but the dealership got it sorted this time anyway.

Friends suggest I open up a Twitter account and post to @Chrysler. So I do. I’m asked if Chrysler Customer Care has contacted me. Nope. Later that day I get a call from CCC. I explain the original issue and the guy tells me it’s abuse since the parking brake was on.

The CCC rep is seriously trying to tell me I shouldn’t be surprised when this Trail Rated Jeep snaps an axle,
because the parking brake was left on for a few feet.

Obviously he’s a jack-ass. He says he’ll talk to the dealer. I never hear back.

A few more days pass, and the Jeep is ready again. I drive by the dealer to pick it up the following day.

December 21st

Picked up the Jeep, and it seems to be working this time. Apparently the clutch on the transfer-case had
burned out.

I talk to Scott, the Service Manager. He admits they checked out the tow-hooks and there’s no evidence the
vehicle was towed out of a mud trap, I obviously wasn’t muddin’. He says even though I kept it under the
recommended speed for 4-Low the drive home from work could’ve contributed to the transfer case failure. Please
call customer assistance and get a tow next time.

Fair enough.

I wonder about Sundays or when it’s 3AM though? I need to ask ’em…

Anyway, the tone of this conversation is different. He brings up the theory on why the axle snapped again,
and I explain to him the traction-control setting and 4-Low. At this point it seems the only plausible
explanation I’ve heard is that the axle was simply weak/defective from the factory.

So I take the Jeep home.

A day or two later I try out the air-suspension on my way out of the office. I hit the UP button a couple
times to raise it to it’s highest setting, Off-road 2. On the way past the Off-road 1 height a
warning pops up on the dash saying the system needs to cool off.

That’s odd. Never seen it before and I hadn’t exercised the system at all. I might understand if it was
really hot outside, or if I’d been using it continuously, but I’d just got in the Jeep and it was in the low
40’s outside. Coldest weather since I’d bought it.

Still, maybe just a fluke. Nothing too alarming so I resolve to take it into the shop after the holiday and
head home.

Over the weekend I try it a few more times. I get the message maybe once more, but mostly it just seems to be degrading overall.

December 27th, 2010

By Monday morning when I take it into the shop it has trouble reaching Normal Ride Height (after sitting overnight at Loading Height) and won’t go into either of the taller off-road settings. The light on the button just blinks continuously, the dash indicator showing the car raising doesn’t come on and there’s no movement.

Also the driver’s seat warmer keeps turning itself off after a few seconds.

At the dealer the mechanic comes out and checks it out with me. He explains that the seat warmer shuts off after awhile automatically to avoid overheating. Seat warmers can get uncomfortably warm if you leave them on. This definitely wasn’t overheating. It was simply faulting.

He told me about the warning I’d mentioned on the air-suspension, and while it did make it to Normal Ride Height while at the dealership, it didn’t make it to either of the off-road heights and even the NRH setting took minutes where normally it takes seconds. Something clearly wasn’t right.

So Enterprise was called to pick me up.

I’m out at Enterprise and the Dodge dealership calls saying send me on back, the car is ready. That’s weird but whatever. OK. So Enterprise drops me back off at the dealership.

The mechanic sits in the Jeep with me while I test it. Off-road 1: Success. Kinda. It takes a minute. Something’s definitely not right. Off-road 2: Nothing. I made sure to film it on my phone.

I’m sorry, I thought you were worried about the Off-road 1 setting.

Seriously? Is this also the guy that didn’t take the broken axle off for a week? I want the damn car to work like new. It’s two months old at this point and has been in the shop for 41 days! Obviously nobody would buy the Quadra Lift system if it was going to be considered normal for it to degrade to the point it took minutes to work and half the heights were unavailable just a couple months off the dealer lot.

December 28th, 2010

So there we are. I just got a call back from Stephen to pick it up. Day 75 of ownership. 44 days in the shop so far.

Epilogue

So what do I want from Chrysler at this point?

Well, after the first 30 days I would’ve hoped they’d perform something like the re-certification that used cars get. After the second time at least. I had to ask the dealership a number of times to check the transmission fluid to make sure it wasn’t toast and the next thing to go (they did, and it looked fine to my untrained eyes, not visibly scorched or anything).

I’ve got a vague explanation about Chrysler’s specs on the air-suspension settings being wrong and working with the service department at Dodge City Dodge to sort it all out. I guess something to do with the “fill” of the system since one of the air-bags was replaced earlier in the whole ordeal. I’d put money down that nobody’s even looked at the seat-warmer.

At this point though, that’s someone else’s problem.

December 29th, 2010

After picking my Jeep up this morning it appears the dealer didn’t identify the problem. So instead of simply admitting that and asking me to keep an eye out for it, they pretended like it didn’t exist. As if they hadn’t seen the issue with the Air Suspension with their own eyes just a couple days before.

The pickup paperwork, which in hindsight I probably shouldn’t have acknowledged with a signature, says that “I was using it too much” and causing it to overheat. Which is complete BS. Turning the car on from a cold start, and trying to get it into Off-road 2 height is not using it too much.

Chrysler really needs to do something about making sure people get calls back and that the dealership isn’t trying to sweep genuine issues under the rug. Especially when the customer has it on video. Maybe I’ll post it to Youtube, but I’m a bit busy right now, and all I really care about is getting some results. I’m not interested in bad-mouthing Chrysler for the sake of it. There’s no such thing as a manufacturer that never makes a mistake when they’re putting millions of cars on the road. I just got unlucky with one vehicle that happened to ship with a bad axle. It had to happen to someone I guess. It’s how you respond to such mistakes that matters though.

This morning the dealer is right though, the system appears to work flawlessy. It responds instantly, and doesn’t overheat even after a couple full up/down cycles. Which is what I expect. Might I get it to overheat trying to push it further? Maybe, I don’t know. I’m not expecting miracles though, getting a couple cycles out of it is what I had before, it’s what I expect it to do, I’m fine with that.

That said, the outside temperature readout on the dash from the previous video says 37°. This morning it was 63°. What do you want to bet that when it dips into the low 30’s again this weekend the issue comes right back? Perhaps when the airbag was replaced they just used the air-compressor on the shop-floor, introducing moisture, causing excess strain to the system at lower temperatures? It’s just a theory, but it seems plausible, and is more than I got from the dealer who decided to simply pretend the issue several people there saw with their own eyes simply didn’t exist a couple days later.