Technology never ceases to amaze me. It’s been a long time since I attempted to stitch images together; last time was probably five years ago and the results were.. mediocre. While I was in the process of writing the post about eliminating the smoke odor in our new 2009 Mercury Sable, I wanted to toss a picture of the car up.. but I was being lazy and just had my 50mm lens mounted on the camera, and didn’t feel like changing it. This resulted in a suboptimal picture (the one at the top of that post), but I also did something that I forgot about until later..

As anyone who has used a 50mm lens on a DSLR with an APS sized sensor knows, that lens isn’t exactly ideal for taking a photo of a car sitting in your driveway from your garage (unless your driveway is, well, long), but I figured why not? I ended up with the picture that you see on that post, but on a whim, I had also taken pictures to try to stitch together and see what it looked like. I completely forgot about those photos until I happened to be browsing through Lightroom a few days ago.. I saw them, and went huh, wonder what it’d take to stitch them. Having no idea how to do this with modern software, I highlighted them, and went to right-click to say edit in Photoshop. Much to my surprise, there is now an option to let Photoshop automatically stitch the images in that context menu.. I thought I might as well give it a shot, and here’s the result (click image for a larger version):

I was quite surprised to see how well Photoshop handles this now, especially with handheld shots that were not rotating around the correct axis, etc. Since I took the pictures and have the originals, I can see where it was stitched, but it still looks pretty darn good to me – what do you think?

I will have to play with stitching more often.. it is certainly much better now than it used to be!

Related posts:

1. Playing around with HDR Photography

[Or "my quest for the ultimate home-brew storage array."] At my day job, we use a variety of storage solutions based on the type of data we’re hosting. Over the last year, we have started to deploy SuperMicro-based hardware with OpenSolaris and ZFS for storage of some classes of data. The systems we have built previously have not had any strict performance requirements, and were built with SuperMicro’s SC846E2 chassis, which supports 24 total SAS/SATA drives, with an integrated port multiplier in the backplane to support multipath to SAS drives. We’re building out a new system that we hope to be able to promote to tier-1 for some “less critical data”, so we wanted better drive density and more performance. We landed on the relatively new SuperMicro SC847 chassis, which supports 36 total 3.5″ drives (24 front and 12 rear) in a 4U enclosure. While researching this product, I didn’t find many reviews and detailed pictures of the chassis, so figured I’d take some pictures while building the system and post them for the benefit of anyone else interested in such a solution.

Updates:
[2010-05-19 Some observations on power consumption appended to the bottom of the post.]
[2010-05-20 Updated notes a bit to clarify that I am not doing multilane or SAS - thanks for reminding me to clarify that Mike.]

In the systems we’ve built so far, we’ve only deployed SATA drives since OpenSolaris can still get us decent performance with SSD for read and write cache. This means that in the 4U cases we’ve used with integrated port multipliers, we have only used one of the two SFF-8087 connectors on the backplane; this works fine, but limits the total throughput of all drives in the system to 4 3gbit/s channels (on this chassis, 6 drives would be on each 3gbit channel.) On our most recent build, we built it with the intention of using it both for “nearline”-class storage, and as a test platform to see if we can get the performance we need to store VM images. As part of this decision, we decided to go with a backplane that supports full throughput to each drive. We also decided to use SATA drives for the storage disks, versus 7200rpm SAS drives (which would support multilane, but with the backplane we’re using it doesn’t matter), or faster SAS disks (as the SSD caches should give us all the speed we need.) For redundancy, our plan is to use replication between appliances versus running multi-head stacked to the same storage shelves; for an example of a multi-head/multi-shelf setup, see this build by the local geek Mike Horwath of ipHouse.

When purchasing a SuperMicro chassis with a SAS backplane, there are a few things you should be aware of..

1. There are different models of the chassis that include different style backplanes:
• ‘A’ style (IE – SC847A) – This chassis includes backplanes that allow direct access to each drive (no port multipliers) via SFF-8087 connectors. In the SC847 case, the front backplane has 6 SFF-8087 connectors, and the rear backplane has 3 SFF-8087 connectors. This allows full bandwidth to every drive, and minimizes the number of cables as much as possible. Downside, of course, is that you need enough controllers to provide 9 SFF-8087 connectors!
• ‘TQ’ style – not available for the SC847 cases, but in the SC846 chassis an example part number would be ‘SC846TQ‘. This backplane provides an individual SATA connector for each drive — in other words, you will need 24 SATA cables, and 24 SATA ports to connect them to. This will be a bit of a mess cable-wise.. with the SFF-8087 option, I don’t know why anyone would still be interested in this – if you have a reason, please comment! This is quite a common option on the 2U chassis – it can actually be difficult to purchase a 2U barebones “SuperServer” that includes SFF-8087 connectors.
• ‘E1′ style (IE – SC847E1) – This chassis includes backplanes with integrated 3gbit/s port multipliers, without multipath support. Each backplane has one SFF-8087 connector, so you only need two SFF-8087 ports in a SC847E1 system. The downside is that you are limited to 3gbit/s per channel – so you’d have a total of 6 drives on each 3gbit/s channel for the front backplane, and 3 drives on each channel for the rear backplane. SuperMicro also has a ‘E16′ option (IE – SC847E16) which is upcoming, and supports SATA3/SAS2, for a total of 6gbit/s per channel.
• ‘E2′ style (IE – SC847E2) – Similar to the SC847E1, this includes a port multiplier on the backplane, but also supports multipath for SAS drives. Each backplane has two SFF-8087 connectors. Same caveats as the E1 apply. They also have a ‘E26′ version coming out soon (IE – SC847E26) which will include SAS2 (6gbit/s) multipliers.

I do wish that SuperMicro would offer a “best of both worlds” option – it would be great to be able to get a high amount of bandwidth to each drive, and also support multipath. Maybe something like a SAS2 backplane which only put two or three drives on each channel instead of six drives? If they did two drives per channel with a port multiplier, and supported multipath, it should be possible to get the same amount of total bandwidth to each drive (assuming active/active multipath), and still keep a reasonable number of total SFF-8087 connectors, plus support multipath with SAS drives, and get the bonus of controller redundancy. If anyone knows of an alternate vendor or of plans at SuperMicro to offer this, by all means, comment!

2. You can also choose the type of expansion slots you would like to support on the motherboard tray; you will need to match the tray to the type of motherboard that you purchase. Note that these are the same options available on their 2U chassis – the concept of the SC847 chassis essentially makes your motherboard choices the same as the 2U systems.
• ‘UB’ option (IE, SC847A-R1400UB) – this option supports SuperMicro’s proprietary UIO expansion cards. It uses a proprietary riser card to mount the cards horizontally, and will support 4 full-height cards and 3 low-profile cards in the SC847. They get the card density by mounting the components for one (or more) UIO cards on the opposite site of the PCB than you usually see – the connector itself is still PCI-E x8, but the bracket and components are all on the opposite side. I have not ordered a chassis that uses UIO recently, so I’m not sure if the sample part number would include riser cards or not. Note that you will need to purchase a SuperMicro board that supports UIO for this chassis.
• ‘LPB’ option (IE, SC847A-R1400LPB) – this option supports 7 low-profile expansion slots. If you do not have any need for full-height cards, this gives you the maximum number of high-speed slots. This is the option you will need to go with if you want to use a motherboard from a vendor other than SuperMicro.

For the system I’m building, we went with the following components:

• SuperMicro SC847A-R1400LPB chassis – 36-bay chassis with backplanes that offer direct access to each drive via SFF-8087 connectors. 7 low-profile expansion slots on the motherboard tray.
• SuperMicro X8DTH-6F motherboard – Intel 5520 chipset; supports Intel’s 5500- and 5600- series Xeon CPUs. Has an integrated LSI 2008 SAS2 controller, which supports 8 channels via two SFF-8087 ports. 7 PCI-E 2.0 x8 slots. 12 total memory slots. IPMI with KVMoIP integrated. Two Gig-E network ports based on Intel’s newest 82576 chipset. This board is great.. but what would make it perfect for me would be a version of the board that had 18 memory slots and 4 integrated Gig-E ports instead of two. Ah well, can’t have it all!
• 2x Intel E5620 Westmere processors
• 24gb DDR3 memory; PC3-10600, registered/ecc.
• 4x LSI 9211-8i PCI-E SAS-2 HBA – 2 SFF-8087 ports on each controller; same chipset (LSI 2008) as the onboard controllers. This gives me a total of 10 SFF-8087 SAS2 ports, which is one more than needed to supports all the drive bays. I should also note that we haven’t had any problems with the LSI2008-based controllers dropping offline with timeouts under OpenSolaris; with our other systems, we started with LSI 3081E-R controllers, and had no end of systems failing due to bug ID 6894775 in OpenSolaris, which as far as I’m able to tell has not yet been resolved. Swapping the controllers out with 9211-8i’s solved all the issues we were having.
• Variety of SuperMicro and 3ware SFF-8087 cables in various lengths to reach the ports on the backplanes from the controller locations.
• 2x Seagate 750gb SATA hard drives for boot disks.
• 18x Hitachi 2TB SATA hard drives for data disks.
• 2x Intel 32gb X25-E SATA-2 SSD’s; used in ZFS for a mirrored Zero Intent Log (ZIL); write cache. (Note: 2.5″ drives; needs a SuperMicro MCP-220-00043-0N adapter to mount in the hot-swap bays.)
• 1x Crucial RealSSD C300 128gb SSD; used in ZFS for a L2ARC read cache. (Also a 2.5″ drive; see note above.)

We purchased the system from CDW, with our own customer-specific pricing. I’m not allowed to share what we paid, but for your reference, I’ve whipped up a shopping cart at Provantage with (essentially) the same components. There is no special pricing here; this is just the pricing that their web site listed as of May 8 2010 at 11:18am central time. Note: I have no affiliation with Provantage. I have ordered from them previously, and enjoyed their service, but cannot guarantee you will have a good experience there. The prices here may or may not be valid if you go to order. You may be able to get better pricing by talking to a customer service rep there. I also had to change a few components for parts that Provantage did not have available – namely some of the various lengths of SFF-8087 cables. I error’d on the side of ‘long’, so it should work, but I haven’t built a system with those exact cables, so can’t guarantee anything.

As you can see, the total price for this system came out at just under $8500, or $8717.14 shipped. Not bad at all for a high-performance storage array with 18 2tb data drives and the ability to add 13 more.

If we do decide that this is the route to go for our VM image storage, the config would be similar to above, with the following changes at minimum:

• More memory (probably 48gb) using 8gb modules to leave room for more expansion without having to replace modules.
• Switch from desktop HDDs to enterprise or nearline HDDs (6gb SAS if they are economical); probably also go with lower capacity drives, as our VMs would not require the same amount of total storage, and NexentaStor is priced by the terabyte of raw storage.
• Add more (either 4x or 6x total, still used in pairs of 2) X25-E’s for ZIL/SLOG, possibly also go with 64gb instead of 32gb. (More total drives should mean more total throughput for synchronous writes. If Seagate Pulsars are available, also consider those.
• Add additional RealSSD C300’s for cache drives; the more the better.
• Add additional network capacity in the form of PCI-E NIC cards – either 2x 4-port Gig-E or 2x 10-GigE. This will allow us to make better use of IPMP and LACP to both distribute our network load among our core switches and use more than 2gbit total bandwidth.

In any case, on to some pictures of the chassis and build.

Front of the chassis – 24 drive bays up front.

Rear of the chassis – 12 drive bays, and a tray for the motherboard above them. Also shows the air shroud to direct airflow over the CPUs; the only part of the chassis that feels cheap at all.. but it serves its purpose just fine.

System with the motherboard tray removed. Note that as far as the mounting is concern the tray is pretty much the same as a standard SuperMicro 2U system. You’ll need to order heatsinks, cards, etc that would work in a 2U.

View of the system from the back with the motherboard and four front fans removed. You can see a bit of the front backplane in the upper right; two of the SFF-8087 connectors are visible. All cable routing goes underneath the fans; there is plenty of room under the motherboard for cable slack. You can also see the connectors that the power supplies slide into on the upper left hand corner, and a pile of extra power cables that are unneeded for my configuration underneath that.

Another shot of the front backplane. You can see the five of the six SFF-8087 connectors (the other is on the right-hand side of the backplane which is not visible.) Also note the fans that I’ve removed to get better access to the backplane.

One of the power connectors that the fans slide into (white four-pin connector near the center of the picture); the SFF-8087 connector that is not visible in the picture above is highlighted in red.

Motherboard tray before installing the motherboard. This tray uses a different style screw system than I’ve seen before; instead of having threaded holes that you screw standoffs into, they have standoffs coming up off the bottom (one highlighted in blue), which you screw an adapter onto (highlighted in red) which the motherboard rests on and is secured to.

A partial view of the rear backplane on the system; also the bundle of extra power cables and the ribbon cable connected to the front panel.

Labels on one of the power supplies. This system includes a pair of ‘PWS-1K41P-1R’ power supplies, which output 1400W at 220V or 1100W at 120V.

Motherboard installed on tray, with the four LSI SAS HBAs in their boxes.

One of the two Intel E5620 ‘Westmere’ Xeon processors set in motherboard but not secured yet.

Both processors and 24gb of memory installed. No heatsinks yet.

Motherboard tray complete and ready to be installed in the system. Heatsinks and LSI controllers have been installed. Note the two SFF-8087 connectors integrated on the motherboard, and eight more on the four controllers.

Prep work on the rear backplane; the chassis shipped with the power cables pre-wired; I connected the SFF-8087 cable.

Motherboard tray installed back in the system; SFF-8087 cables connected to three of the four LSI controllers. I ended up moving one controller over for ease of cabling – notice the gap in the middle of the four controllers.

(Note: The pictures of the finished system below this point were taken on 5/7/2010; thanks to my coworker Colleen for letting me borrow her camera since I #natefail‘d to bring mine!)

The seven cooling fans to keep this system running nice and cool.

HBAs with all cables connected.

Finished system build with the top off. One power supply is slightly pulled out since I only have a single power cable plugged in.. if you have one cable plugged in but both power supplies installed, alas, the alarm buzzer is loud.

Front hard drive lights after system is finished – note that we don’t have every drive bay populated yet.

Rear drive lights while system is running.

The build-out on the system went fine for the most part; the only problem I ran into is that the motherboard did not have a BIOS installed which supported the relatively new Westmere processors. Fortunately I had a Nehalem E5520 I could borrow from another system to get the BIOS upgraded.. I wish the BIOS recovery procedure would work for unsupported processors, but ah well. I was pleased with the way the motherboard tray slides out; it makes it easy to get the cabling tucked underneath and routed so that they will not interfere with airflow. There also seems to be plenty of airflow to keep the 36 drives cooled.

I currently have NexentaStor 3.0 running on the system; we have not yet landed on what operating system we will run on this long-term.. but it will likely either be NexentaCore or NexentaStor. If we deploy this solution for our VM images (with some upgrades as mentioned above), we will almost certainly use NexentaStor and the VMDC plugin, but we’ll cross that bridge if we get there!

Here’s the disk configuration I have running at the moment with NexentaStor:

• ’syspool’: Mirrored ZFS zpool with 2×750gb Seagate drives.
• ‘NateVol1′: ZFS zpool with..
• 2 RaidZ3 arrays with 8 2TB disks each
• 2 2TB disks set as spares
• 2 36gb Intel X25-E SSDs as a mirrored log device
• 1 128gb Crucial RealSSD C300 as a cache device

..and the obligatory screenshot of the data volume config:

This nets 18T usable space, and would allow for a simultaneous failure of any three data disks before there is any risk of data loss. (Each of the sub-arrays in ‘NateVol1′ have 3 parity disks – so I could also lose 3 disks from each of the sub-arrays without any issues.)

Again, this system only has two Gig-E NICs at the moment.. I’ve done I/O tests with NFS across one NIC and iSCSI across the other NIC, and can max out the bandwidth on both cards simultaneously with multiple runs of Bonnie++ 1.96 without the system breaking a sweat. I like! I should also note that this is with both deduplication and compression enabled.

Another note – before putting this into production, I did some simple “amp clamp” power usage tests on the box, with one power supply unplugged. The other power supply was plugged into 120V. While idling, it consumed 3.3A, and while running multiple copies of Bonnie in the ZFS storage pool (with all active disks lighting up nicely), it consumed 4.1A. Not bad at all for the amount of disk in this machine! I’d estimate that if the 13 additional drive bays were occupied with 2TB disks, and all those disks were active, the machine would consume about 5.5A – maybe slightly more. When we racked it up at the data center (in one of our legacy racks that is still 120V), the power usage bumped up by 3.2A combined across the A+B power, which matches nicely with my clamped readings. I’m very impressed – under 500 watts while running full out.. wow.

I will update this post once we decide on a final configuration “for real” and put this into production, but so far I’d highly recommend this configuration! If you’ve used the SC847 chassis, I’d love to hear what you’ve thought. I’d also love to try out the 45-bay storage expansion version of this chassis at some point – talk about some dense storage! :)

Related posts:

1. LSI command-line utility for SAS2 non-RAID controllers
2. Sun adds block-level deduplication to zfs
3. Sun’s Unified Storage 7210 – designed to disappoint?

..but I can dig it. (License plate: ‘A RED CAR’. On a red car. A red Honda Civic to be specific.)

[And yes, my apologies for the poor quality of the photo - it's what you get when taking a photo at 45mph with the crummy camera on an HTC Kaiser. Sorry.]

Related posts:

1. something i’ve learned today..
2. I think I found a fellow geek..
3. Lots of rain in a short period of time != good roads

As I mentioned in my previous post, we purchased a 2009 Mercury Sable that had been smoked in by the previous owner. The smell of smoke wasn’t horrible, but it was enough to give both Tiff and I sore throats while test driving the car (we’re both sensitive to smoke.) If you’re interested in what we did to eliminate the smoke smell, please read more!

The first treatment was performed by the dealership. They use the Auto Vaccine product by Biocide Systems. While I was sitting in the dealership waiting for the salesman to get some paperwork done, I had done some quick research on my phone, and this is the solution that I found that most people recommended – so I was glad to hear that they use it. From the reviews I found, here is the recommended procedure:

1. Get the car in a garage or other area that is not exposed to sunlight (apparently sunlight breaks down the product.)
2. Seal up the car – put all the windows up, make sure the seals on the doors are good, etc.
3. Try to make sure the product can reach all corners of the car – open all compartments, if seats can fold down for trunk access fold them down, fold the visors halfway down — basically, if it can be opened without blocking access to another part of the interior, open it.
4. Start the air conditioner in ‘recirculation’ mode. (Not sure if most people leave the car running for this, or put it on a battery charger, or what.) This will allow the product to circulate through the ventilation system.
5. Place the Auto Vaccine in the car, and follow the directions to start the chemical reaction.
6. Leave the car and make sure it’s sealed tight.
7. Let sit for 24 hours.
8. Open all windows and doors, and let the car air out for at least 3 hours.

When we picked up the car from the dealership after the treatment, there wasn’t a smoke smell in the interior anymore; it just had a slight chemical smell. However, when we fired up the air conditioning, there was a definitive smoke odor again (which faded to near-nil after shutting the AC off) — I have a hunch they didn’t let the product circulate through all the vents. The dealer included a second treatment in their deal, but we don’t have time to let the car sit for 24 hours at the moment, and I decided to try a few treatments myself.

The first one I did myself is a treatment with 1Z einszett Klima-Cleaner. It is basically a can of foam with a tube about two feet long that you feed into your center air conditioning vent and spray the foam until the can is empty (it took ~4 minutes for me – felt like forever!), which will cover the surface of the evaporator core with their foam. Then you let it sit for 30 minutes, and then start the car and run the AC full blast for a few minutes. This worked magic on the smoke smell from the AC – it was horrible before the treatment, and I couldn’t smell it at all in the air coming from the vents after the treatment was finished. I did read a review on the ‘net where the poster mentioned the foam sprayed all over the place; it turned out that he turned the can upside down while applying it, which apparently can cause the foam not to get all the way down to the core. So when you fire up the AC, whoosh, foam! If you try this, be careful and follow the directions on the can. They do warn on the can to follow the vehicle manufacturer’s cleaning instructions or to have a certified technician apply it – I’m imagining this is because of the possible damage that could be done if this product were to be applied incorrectly.

After that treatment, I could smell a slight smoke smell on the interior surfaces again (probably because it circulated out of the AC after the Auto Vaccine treatment.) When I purchased the 1Z cleaner, I also picked up a can of Dakota Odor Bomb – neutral scent. I figured I would give this a shot, as the reviews I had read of it said a few hours was enough time. I did the same steps I had read about for Auto Vaccine (above), except I didn’t run the AC during the process (since I had just treated that.) I went ahead and unleashed the bomb. The next morning, I let the car air out for a bit, and found that there was a nasty film all over the entire interior – blah! I spent the next half hour or so wiping down the interior to get the film off, and then went to work. The smell was really strong for a few days; once it started to fade, it was actually much better – no hints of smoke left, so it seems it did some good. However, if I were to do this again, I would probably not use the Dakota product, but find a time to do a full Auto Vaccine treatment again – it costs a bit more, and takes more time, but we didn’t have any of the nastiness with it!

A picture of the car while being fogged with the Odor Bomb

I also noticed that the leather on the car seemed a bit dry (not sure if it was because of the Auto Vaccine, Odor Bomb, or if it just hadn’t been treated before), so I picked up a container of Lexol 3-in-1 Leather Care Spray. I applied it using a microfiber cloth, rubbed it in with a small brush, and let it sit for about 5 minutes before wiping it off and polishing with another microfiber cloth. I had great results with it; the leather feels much better than it did before – smoother and softer. I will probably reapply this a few times over the next few months to ensure the leather’s in good shape, and then keep it up on a semi-regular basis to keep the leather nice.

The steps above have much improved the smoke scent in my car – as of right now, I can’t even smell that it was ever smoked in. If you try any of these, or have other tips for getting rid of the smoke odor, I’d love to hear about it!

Related posts:

1. Oh Passat, how I will miss thee… (or, “my tale of a new car”)
2. Fun with cameras and Photoshop..

Our beloved 2007 Passat just came up on lease end, which means that it was time to decide if we wanted to keep it, or if it was time to go car shopping.

We loved the car, but our old Passat (which we traded in for this leased ‘07 Passat) ended up costing us a ton of money just for routine maintenance as it got over the 60,000 mile mark. Note that we didn’t have any real problems with it; all the maintenance was either routine (oil/lube/filter, other fluids, belts, etc) or replacement of wear components (brakes, boot on the front suspension, etc.) We hadn’t had to do any of the expensive work on our ‘07 Passat – we basically just had to do 4 oil changes in the time we owned it.

Here’s some of the expensive / annoying repair notes from our previous Passat:

1. All the parts were significantly more expensive than the equivalent components for cars I’d owned in the past (all domestic.) A lot of them also weren’t stocked by the general auto part delivery places (NAPA and similar), so the repair shop would end up having to order the part from a local VW dealer, which cost more and caused delays.
2. The car required ’specialty’ fluids in some cases – for example, the engine coolant was VW-specific. Again, it had to be ordered from a VW dealer.
3. I found that I couldn’t have the maintenance done at any generic repair shop – for example, I had trouble with quick-oil change places (Valvoline specifically) breaking components while doing oil changes as they didn’t have the VW-specific tool to get the filter out, and also had problems when I had a Goodyear Gemini location replace a front suspension component (I believe it was a control arm) due to a leaky boot (apparently there were two slightly different versions of the control arm for the year and model that I had, depending on the engine type, and they ordered the wrong one. When they dropped the car off the lift, the part snapped. Oops!) After this occured, we decided only to have maintenance done at either a VW/Audi dealership or at a specialty import mechanic.. which means more expensive labor on future repairs.
4. VW “highly recommended” replacing the rotors with each brake job.. something about high-performance rotors that will only last the life of one set of brake pads. Of course, both parts were more expensive than the domestic equivalents.
5. VW required synthetic oil for all models. Nothing wrong with this (lets me go longer between oil changes), but still has a bit of sticker shock when changing the oil.

..and the one thing that did apply to both the old Passat and our ‘07 Passat — expensive gas (they require premium – although I have heard that you can do ok with regular gas, didn’t want to risk it.)

However, I really did love our Passat, and we had driven way less than the contracted miles on our lease.. which meant that the residual amount on the lease (the price we’d have to pay to purchase it, plus taxes of course) was slightly under the wholesale value of the vehicle, and therefore a reasonable deal. I tried to get some independent confirmation of what my gut was telling me about maintenance cost, and after some searching, I found Edmund’s True Cost to Own tool. Punching in both the 2007 Passat and a 2007 Impala (domestic car kinda/sorta similar to the Passat?), I found what I was hoping not to find – that the Passat was more expensive to maintain.

Passat TCO:

Impala TCO:

The only columns that I’m really paying attention to are the ‘Maintenance’ and ‘Repairs’ – the rest are kind of out of scope, as we plan on driving the car into the ground.. 5-year maintenance+repairs on the Passat ended up at $7005, and the 5-year maintenance+repairs on the Impala ended up at $4985, for a $2000 difference.. kind of a big jump.

So, we decided to go ahead and find another car.. we came up with a list of requirements:

• Relatively new (2005+) with fairly low miles (under 50,000 or so.)
• Capable of comfortably seating four adults or two adults with two car seats – this means a fairly large car, as I’m a tall guy, and an adult or a rear-facing car seat won’t really fit behind me in many vehicles.
• Decent gas mileage – we have a Trailblazer for our second vehicle (which Tiff currently drives most days, and for rather short distances), and since I have a ~25 mile commute, keeping a car with good gas mileage around is a good idea. The Passat averaged 30mpg on my way to work, so we wanted a car that would get similar mileage.
• Leather seats – child+dogs == mess – it’s amazing how much easier it is to clean up leather than cloth.
• Inexpensive maintenance that any mechanic can deal with – from my experience, this means either a domestic or a very common import.
• As inexpensive as possible (ideally, $10k – $15k) – we were trying to come in with a payment on a 60-month loan that saved us a decent chunk of change compared to the Passat lease.

After many gruelling nights spent on Edmunds, Craigslist and Carsoup, I ran across Alliance Car Sales. Their offerings intrigued me, as it seemed that they sold primarily low-mile late-model vehicles, and for prices that were quite good. I shot them a (very, very, very, very long) email with what I was looking for, and got a reply back from Dave in short order with a few suggestions, and asking when Tiff and I could come out to check out some cars. (They buy cars off auction, then use them as a rental fleet until a customer is interested in buying one of the.) We were able to meet Dave a few days later, and tried out a few cars. When we went in, we were considering the following, in no particular order:

• 07+ Pontiac Grand Prix
• 07+ Chevrolet Impala
• Any year Ford Fusion / Mercury Milan
• 2005-2007 Ford Five Hundred / Mercury Montego
• 2008+ Ford Taurus / Mercury Sable
• Recent model Hyundai Sonata
• Recent model Nissan Altima
• Recent model Honda Accord
• Recent model Toyota Camry

Note – the 2008+ Taurus/Sable are actually the Five Hundred/Montego with an upgraded engine.. very different from the Taurus of old. They are larger, get better mileage, have a much more modern design – in fact, they share their platform with the Volvo S60. I personally wouldn’t really consider a previous-generation Taurus or Sable.. heard way too many horror stories.

We were able to knock off the Sonata, Accord, and Camry just due to size.. the Altima would probably work, but they were quite expensive, so we dropped that too. The Grand Prix and Impala (I guess they are essentially the same car, like a Taurus vs. Sable, just different body and trim) worked, but they were hard to find (and expensive) in leather. All of the Milans and the Taurus that Alliance had in stock were out for rental, so we met on meeting Dave later on in the week when he’d have them back on the lot. (Side note – if you are in the metro area and interested in a newer used car, I highly recommend Alliance – it was really great to work with a person who was only interested in getting us the right car, and didn’t give us any pressure at all. Dave was even willing to offer me advice after I told him we were purchasing a car elsewhere – who else will do that?)

..and then one of my Craigslist RSS feeds turned up something interesting – a loaded ‘09 Sable Premier at a Mercedes dealership for a very reasonable price. The car is newer than we were really looking for, and also more expensive ($16,500.) However, it had everything we wanted (plus more), and would more than meet our goal of lasting at least five years – it’d even have a warranty included for the first couple years. I stopped out at the dealership for a test drive, and only found one issue with the vehicle.. it had been smoked in by the previous owner (who only owned it for a few months.) The smoke wasn’t super-heavy, but both Tiff and I are sensitive to cigarette smoke, so this could be a problem for us. The dealer assured me that they had a solution to fix that, so I went ahead and took the car to my mechanic for a mechanical inspection, which it passed with flying colors. I also got his impressions on how the cars were to work on in general (going back to the Ford Five Hundred / Mercury Montego days, since they are essentially the same car, just with a smaller engine), and he said they were generally quite reliable, and were one of the better cars out there for repair costs, etc. An Edmunds TCO backed this up:

Granted, this is for a new Sable (they don’t have 2009 available for ‘Used TCO’ yet), but the 2008 was there for used, and came out at $4746 for maintenance and repairs – still quite good. Short version – we signed a purchase contract on the vehicle, with a stipulation that we had to be satisfied with the smoke scent after their treatment, and then agreed to purchase it the next day (they used the Auto Vaccine treatment, which worked fairly well – however, it did leave a hint of smoke behind after the scent of Auto Vaccine faded away, especially when the AC was on.. the dealership promised to do the Auto Vaccine again whenever we desire, but I decided to do some cleanup myself in advance of that – will detail what I was able to do in a future post.

We’ve liked the new car quite well so far; it’s big, comfortable, has tons of features, and gets good mileage.. but it’s a sedan, not a sports sedan. Ah, Passat, how I will miss you! Especially the tiptronic transmission..

I hope the next owner of our goold ‘ol Passat will love it as much as we did! :)

Related posts:

1. How I got rid of the smoke smell in my new (to me) car

So I’ve been trying to have some wins in life.. and in the last weeks, I’ve managed to make a few, albeit small ones. I figured I’d post these publicly so all my friends, enemies, and random haters on the ‘net can mock me mercilessly if necessary. ;)

• No drive-thru since 2/18. I apologize to my local McDonalds for putting them out of business.
• No non-frozen/non-homemade pizza since 2/19!
• For the last two weeks, I’ve been successfully getting up by the time I target (~6:30 on work days, and whatever time I need to be up on weekends) a majority of the days.
• For the last two weeks, I’ve been averaging two cans or less of sugary caffeinated goodness, down from ~6-8 cans (I know, I know!) I’ve discovered my addiction is caffeine and not the cola itself – so coffee and the occasional 200mg “evil” booster.
• For the last two to three weeks, I’ve been getting the pups out for a walk whenever I need to (IE, when Tiff, Kai and I are going to be out of the house) – next up, at least once a day, every day

I know, nothing major at all – but hey, small steps, right? :)

No related posts.

At my day job, we purchased 3 Sun (er, Oracle) 7210’s (part of their ‘Unified Storage’ platform; the machines are X4540’s with their specialized Unified Storage OS) on the recommendation of one of our vendors. The models we purchased contained 48 250gb disks and 32gb memory — no SSD’s. Our intent was to use these for our VM image storage (VM infrastructure is Xen, using LVM-over-iSCSI to the 7210’s.) We planned to install two units in production, one in our DR environment, and to use ZFS replication to have VM disaster recovery bliss.

It’s been a disaster.

Update [2010/02/26]: Sun has really stepped up to the plate and gotten the right people on this issue. I’m not sure if it’s what I posted here, the numerous ex-Sun contacts that I had who pinged people they knew, or the president of our VAR hammering at some of his contacts within Sun/Oracle. I will keep updating as we work towards RCA’s and a resolution.

Update [2010/08/27]: Thanks to some hard work from our reseller and people at Oracle, we were able to return these units a few months ago. I wish we had been able to work through the issues with Oracle, but needed to get something that we could trust online ASAP. For the record, I do think that we probably did receive a “batch” of bad units; I still have not heard from anyone else who has had multiple independent units fail simultaneously as we did. I will also keep comments open here, and encourage anyone who has had great deployments to post a followup – I do believe that there are people that have had great success with these units out there, or else they wouldn’t be selling so well! :) And again, thank you to the hard work from our reseller, and for all the people from Oracle (and those who used to work for Sun but did not move to Oracle) who did their best to help us!

Here’s the pre-sales issues we had with the unit..

1. ZFS does not use traditional raid controllers, and uses your system memory as cache. This is great for reads (massive cache!), but for writes, it means that it’s extremely dangerous to turn write caching on. The memory used for write caching (which could be up to 32G in our system, or more in one with more memory) is not battery-backed, so if the system fails in any way (power supply failure, crash, power failure, etc.), any data in that cache is gone. On a “normal” ZFS configuration with SSD’s for the ZIL, there wouldn’t be a need for write caching in memory – data is cached on the write-optimized SSD’s, and if your system crashes, it will still be there. However, the model we ended up with does not contain a SSD, and the Sun Oracle-supported SSD for this unit is around $6000 (18gb ‘Logzilla’.) YIKES.
2. Without SSD or write cache (we need the write cache disabled for the reasons described above), the maximum streaming synchronous write speed I have been able to get over iSCSI is ~35MB/s – that’s with triple-mirrored disks. With this configuration, if I have multiple virtual machines running and start a write-heavy process, the other VMs will slow to a crawl with high I/O wait. Not acceptable in production.
3. For the reasons above, I firmly believe that Sun Oracle should NOT sell any ZFS-based storage appliances without SSDs – at least not without a big warning sticker that says “Make sure you can afford to either lose any data in cache (in some environments this would not be a huge issue), or work with the speeds described above.
4. Sun Oracle no longer offers Try-and-Buy programs. We’ve been told that once you open the unit, it’s yours. Not a strong sign that they are confident in their technology. Oh, and we were told this after we had purchased and installed the units.

Now, for the technical problems we’ve been having..

1. The DR unit has had problems since a few days after we installed it. It’s been throwing fan errors, which Sun Oracle has told us is a ‘firmware bug’ with no fix (even though it only happens on one of three nodes) – the workaround is to restart the service processor. It also has had a bad CPU, and worst of all, it started rebooting every night. Sun Oracle support had us jump through a zillion hoops for over a month to try to debug the nightly reboot issue, replaced various components (including the motherboard), and never came up with a workable solution. Then, a few weeks ago, after 4-5 days of not touching the system (I had been assigned to other tasks and the coworker who was taking it over was on vacation, plus a weekend), the reboots suddenly stopped. We were never able to get the reboots to either alert Sun Oracle via the phone home mechanism, or trigger the alerts we had set up via the web GUI – even though it was rebooting unexpectedly. There was no easy way to figure out that it had rebooted, in fact – I ended up having to go to the shell (naughty!) and run ‘last’ to see the reboot times. If we were relying on the GUI, and didn’t have monitoring set up of the system (which is what alerted us to the reboots), we wouldn’t have known until we put a production load on it — which, since this is a DR unit, would only have been when we failed to our DR site. Sun Oracle support has not been able to offer a RCA, except to say that whenever it rebooted the mpt driver threw errors. Since it ‘fixed itself’, Sun Oracle considered the case closed. The combination of the machine being flaky from the start combined with the fact that Sun Oracle couldn’t debug the issue has made us push for a replacement unit, but it seems nobody in Sun Oracle support has the authority to replace a chunk of hardware that can’t be trusted.
2. A few weeks into debugging the issues with the DR unit, both of the production units (which were taking a test workload) developed problems at the same time. One of them stopped answering on the VLAN that we were running iSCSI on (which was trunked on top of the management VLAN; that VLAN continued to work fine), and the other turned itself off. The best part is this happened within 10 minutes of each other. Again, Sun Oracle support is unable to give us a root cause for either issue. The systems are completely isolated from each other physically (separate circuits, separate racks in different areas of the data center, etc) — the only shared component is the network (both are dual-homed to two switches.) Nothing else in the data center exhibited any issues during this period – the switches didn’t show any oddities, other systems and PDU’s on the same UPS branch did not show power issues, etc. Again, Sun Oracle’s response was “if it happens again let us know.” Even though we would have taken a full production outage of _everything_ because of this. Again, neither of the units triggered the phone-home or self-alert functionality when these issues occurred.
3. When troubleshooting with Sun Oracle, their first step is to reboot the system into a debug mode. These are production storage appliances – having to reboot to get a RCA for a previous issue is *not* acceptable! Again, this makes it feel more like “this a server that should be in a redundant zero-shared config” more than a storage appliance.

Really, I can understand technical issues; they happen with every product. However, it appears that the way the specialized version of Solaris is set up on these, they are incapable of getting useful logs. On top of that, the support has been worse than useless. They have chewed through a ton of our time (we’ve probably spent at least 4 man-weeks working with them on this), and have not come back with anything useful. After escalating the tickets, nobody has the authority to either replace a box or allow us to return it. The escalation process is slow and doesn’t actually really seem to do anything; after requesting escalations, we don’t get higher-level contacts (the only way I’ve managed to do that so far is by blind-calling the service center when our regular tech wasn’t available, and requesting a callback), and the people it’s escalated to still have no power to resolve our issues. Every time we request escalation it also seems to slow the process by at least a few days – it’s crazy. The support people all seem to want to help, but the system is stacked against them.

So, right now I’m stuck with 3 7210’s that have all had issues, that I don’t trust in production, and I can’t return. Thanks a lot Sun Oracle! What happened to the old-fashioned Sun, where the gear was way too expensive (and yes, these 7210’s were expensive), but the support made up for it?

My advice to you if you are looking at a Unified Storage appliance is to run far away – the ones with SSDs included would in all likelyhood perform as expected, but the quality of the support and lack of customer service will scare me away from ever making a Sun Oracle purchase again.

Update [2010/02/24] – After chatting with many contacts that I used to have within Sun, I have changed most of the relevant ‘Sun’ names to ‘Oracle’. It seems that when Oracle purchased Sun, many, many good people were let go. Bummer.

Related posts:

1. Review: SuperMicro’s SC847 (SC847A) 4U chassis with 36 drive bays
2. Are there any open-source worm filesystems?
3. LSI command-line utility for SAS2 non-RAID controllers

Sarah Anthony of the band The Letter Black singing; shot at the Minneapolis Convention Center on November 14, 2009 during Skillet’s ‘Awake and Alive’ concert tour.

More Sarah Anthony of The Letter Black. Again, they rock — buy their album.

Related posts:

1. Sarah Anthony of The Letter Black performing during Skillet’s Awake and Alive tour
2. Jen Ledger of the band Skillet singing and playing the drums
3. Ben Kasica of Skillet during a guitar solo in front of fog

Sarah Anthony of the band The Letter Black; shot at the Minneapolis Convention Center on November 14, 2009 during Skillet’s ‘Awake and Alive’ concert tour.

This is Sarah Anthony of the band The Letter Black. I had never heard of this band before Skillet’s Awake and Alive tour, but they are awesome. Remind me a lot of Evanescence, but different. Highly recommend you pick up their album, Breaking the Silence – it’s not expensive. I’m also looking forward to their next one, and seeing them live in concert again!

I should note that they were significantly easier to photograph than Skillet.. no pyro and no fog == happy photographer. ;)

Related posts:

1. Sarah Anthony of The Letter Black (aka Rock Photography is Fun Part 8)
2. Jen Ledger of the band Skillet singing and playing the drums
3. Ben Kasica of the band Skillet with pyrotechnics in the background

As announced on Florian Haas’s blog, DRBD has been accepted into the mainline Linux kernel, and will be released with 2.6.33. Congratulations to the DRBD team on this major milestone! This will make the lives of all DRBD users much, much easier.

For those of you who are not aware, DRBD is a synchronous block-level replication solution for Linux that allows you to do nifty things like active/active fileservers, active/passive database configurations, and many other nifty solutions. With a commercial add-on called DRBD proxy, it’s possible to do long-distance replication for DR clusters or whatnot. DRBD is also completely filesystem independent.

In any case, if you’ve been hesitating to try DRBD because of the pain of out-of-tree modules, this will no longer be an issue.

Related posts:

1. How to compile Nvidia kernel modules on 2.6.20+ with paravirt_ops enabled
2. Types of VPN available on Linux
3. new favorite music player for linux

Mediacom keeps getting worse and worse. As I’ve mentioned before. Mediacom is my ISP at home. Unfortunately, it’s the only service of reasonable speed at a decent price available for my address.

For the past few months, I’ve noticed that I seemingly randomly get redirected to a “Mediacom Assist” page, located at http://assist.mediacomcable.com. I assumed that they were being evil, and inspecting dns packets for NXDOMAIN (I don’t use their DNS servers, so it couldn’t just be Sitefinder-type junk.) However, I seemed to randomly hit links to working sites (ie, google.com) that also returned this error – just figured it was a fluke in their mechanism, and moved on. I just had a kid, after all, I’m too busy to be digging into what stupid things my ISP is doing! ;)
Tonight, I was reading the fun thread about MacPadd at Tom’s Hardware, and they have a link to the LiveJournal blog that MacPadd was posting at for awhile. The given URL was macpadd.livejournal.com. When I hit this page, Mediacom sends me over to “Assist”. I know that LiveJournal has a wildcard set up, so this should not be possible. For the fun of it, I set up a tcpdump, and checked what happens when I try to hit that page.. here are the results:

Request:

GET / HTTP/1.1
Host: macpadd.livejournal.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5 FirePHP/0.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflatehttp://www.natecarlson.com/wp-admin/plugins.php
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

Reply:

HTTP/1.1 200 OK
<script>window.location='http://assist.mediacomcable.com/mediacomassist_pnf/dnsassist/main/?domain='+escape(window.location);The Search Guide redirection service has been enabled to provide helpful searches from browser queries. You entered a non-existent url and your browser attempted to redirect you with Javascript. To enable this please update your browser preferences. To turn off this feature please click this here

Ahem. The same query from a non-mediacom machine returns the expected 404 page from LiveJournal. So, apparently Mediacom is doing deep packet inspection, and rewriting 404’s under certain circumstances. The page that they redirect you to also has a pile of ‘Sponsored Links’ that are going to make them cash. Chances are this is all about the cash for them. They offer an opt-out, but who knows what else they are hijacking? This is just wrong and EVIL. For all I know, they could intercepting ads for Comcast or satellite TV and replacing them with their own. Or just about anything else, as long as money is involved.

Could we please get a net neutrality bill passed that says that ISP’s are not allowed to muck with our packets in this way? Well, maybe not – any net neutrality bill that would actually get passed would probably contain more harm than good. ;)

After figuring out what they were doing, I found lots of other posts about this.. anyone else find this to be pathetic?

Related posts:

1. why must mediacom be so terrible?
2. firefox finally passes ie for hits on my site!

One of the things that is nice with the release of Xen Cloud Platform (which I mentioned previously) is Project XVP. As the site mentions, it has four major components – xvp (a VNC proxy to allow you to connect to the console of a VM hosted on XenServer or Xen Cloud Platform), xvpviewer (TightVNC’s Java client extended to support XenServer-specific functions), xvpweb (a simple web interface to manage your VM’s), and xvpdiscover (an app to build a config file suitable for xvpweb and xvp.)

The executive summary is that there is now an open-source project available to get console access to XenServer machines – without the need for XenCenter on Windows. I believe this is also the only method to get access to the console on Xen Cloud Platform machines, but would love to be proven wrong. ;)

I followed the directions provided at the xvp page to build the RPM’s.. I was building on a CentOS 5.0 x64 system (I’m usually a Debian user and this was a spare VM I had laying around), and while building libxenserver, ran into the following errors:


/usr/bin/ld: skipping incompatible /usr/lib/libgssapi_krb5.so when searching for -lgssapi_krb5
/usr/bin/ld: skipping incompatible /usr/lib/libgssapi_krb5.a when searching for -lgssapi_krb5


It looks like older versions of RHEL/CentOS5 on 64-bit put the wrong development libraries in /usr/lib, gah! This page has one method of fixing this; the other is to remove the devel packages, and re-install only the 64-bit versions.. RHEL/CentOS bugs me on 64-bit since they install both 32-bit and 64-bit versions of stuff in the same areas, sigh.

In any case, after fixing the dev libraries, the xvp SRPM built as expected. Be sure that you have the CentOS ‘Extras’ repo enabled (needed for php-mcrypt), and install the RPM’s generated.

To generate a config file, you can run the ‘xvpdiscover’ command. It will prompt you for a hostname, username, and password (these are creds for a XenServer host), vnc password (this is a password needed to VNC in), and output file. Save the output to /etc/xvp.conf, and make sure the permissions are set so that your web server can access it. Once this is done, fire up xvp with ‘/etc/init.d/xvp start’.

After doing this, when I tried to hit the web page. I got a blank page back, and the Apache error log shows:


PHP Parse error: syntax error, unexpected T_STRING in /var/www/html/xvpweb/index.php on line 35


Turns out that CentOS leaves short_open_tags turned on by default. Edit /etc/php.ini, and change ’short_open_tag = On’ to ’short_open_tag = Off’, and restart Apache.

After doing that, I hit the web page again, and after a few moments, a list of all the VM’s along with their status popped up. There is a small Console icon by each VM, click it, and you’ve got the console. The console gives you simple operations such as shutting down the VM, rebooting the VM, etc. Nice!

The web interface is actually not necessary at all if you just want to use a raw VNC client and don’t care about those extra functions – you can check /var/log/xvp.log once xvp is started, and get the VNC port number for each VM. Then you can use your favorite VNC client to connect.

Things I’d love to see in the future:

1. Automatic reconfiguration — if you add or remove a VM right now, you will need to regenerate the config file. You can do this via cron, but it’d be nice if it could happen dynamically.
2. Ability to authenticate both the web page and the VNC passwords against the XenServer database. I guess VNC password doesn’t matter if you never manually connect to VNC, but it would be great to not have to configure authentication sources in two different places.
3. Ability to see tags, descriptions, etc in the web interface, and sort in various ways (next item down covers this, though.)
4. The web interface is very, very basic. It’d be nice to see other simple features integrated – IE, reboot, live migrate, etc. Shouldn’t be hard at all to do, just needs someone to do it.
5. It would also be nice to see the web interface extended to fully replace XenCenter, at least for monitoring functions (ie, graphing, etc.). That will take a lot of development. ;) I’d love to be able to delete XenCenter (and my Windows VM).
6. Debian or Ubuntu packages. ;) This would not be difficult at all.

In any case, this means I no longer have to fire up a Windows VM just to access the consoles of my XenServer VM’s, which is very much appreciated!

Related posts:

1. Xen Cloud Platform
2. citrix to open-source xenserver
3. mosso – “the hosting cloud”?