Anywho, on the Windows server I’m using the User Name Mapping service to map *nix user accounts to Windows user accounts. There are other ways to do this, but it seemed the simplest and easiest for my purposes. I just mapped the root user from the ESXi server to my domain account and then use NFS shares from Windows for backup purposes (using ghettoVCB).

Last week I thought it would be prudent to ensure that I could actually do something with these backups I’m stacking up. Since they’re already on an NFS share I figured I’d just connect to that share with my Windows 7 machine running VMware Workstation.

There is a NFS client is built-in to Windows 7 (I’m unsure of which editions though – Enterprise and Ultimate maybe?) and just needs to be activated via the “Turn Windows features on/off” control panel applet. Check the boxes, wait a few moments and you’re good to go.

Oh, and reboot. I didn’t get prompted to reboot but it definitely didn’t start working until I did so!

Once that’s all done just a bit of configuration needs to happen. First, launch the “Services for Network File System” app on the Windows 7 machine. Right click on “Services for NFS” and click on Properties. In the resulting dialog check User Name Mapping and specify the server running the service. That’s really about all you need to do.

Now just type “servername:/sharename” into an Explorer address bar and you’ll see your stuff. You can map from there as well. Or drop to a command prompt and use the mount command.

But it wouldn’t work for me. I’d get connected but clearly had authority issues (ain’t that a lifelong issue..?). Turns out I had one more little step to go – one frequently glossed over.

By default the User Name Mapping service (on the server) is only able to accept requests from the machine on which it is running. So when my Windows 7 machine tried to use it nothing would happen. It was blocked. Fortunately, I found a technet article that explained what was going on and what I needed to change.

On the server I had to edit the %windir$\msnfs\.maphosts file (use the text editor of your choice). Specifically, I had to add one line to the bottom that listed my Win7 client’s name. Restarted the service and everything works great now!

You control which computers can access User Name Mapping by editing a file named .maphosts in the %windir%\msnfs directory of the computer User Name Mapping. By adding entries in this file, you can easily permit or deny access by any other computer to User Name Mapping. If the .maphosts file is present but not empty, then only those computers allowed access by entries in the file can access User Name mapping. If the .maphosts file is present but empty (the default), no computers except the computer running User Name Mapping itself can access User Name Mapping. If the .maphosts file is not present, no computers (including the computer running User Name Mapping) can access User Name Mapping.

Now I realize it is a bit odd to use NFS between Windows computers, but it was a fun challenge getting it all working. At some point I might try to do some benchmarking to see how NFS maps compare to normal SMB shares.

Related posts:

1. Wrestling with ESXi Client Backups
2. Windows 7 Beta
3. Windows Live Mesh or DropBox?

Related posts:

1. Google Toolbar Beta: Make “New Tab” your Home Page
2. Faviconize – a New Favorite Firefox Extension
3. K-Meleon Tab Management

For starters, Soluto is probably the prettiest little startup manager you’ve ever used for Windows. Definitely a non-traditional GUI with lots of whiz-bang effects. Be sure to explore when you first start it up. Notice what happens when you hover over various objects to find all the options, buttons and knobs.

Install it, reboot and watch it analyze and time the boot process. It will categorize all the startup stuff as “No- brainer – remove from boot”, “potentially removable”, “Required” and then finally “removed.” The software collects and aggregates everyone’s decisions and can show you what others have chosen. Hopefully they choose wisely…

The No-brainer items you can opt to delay them to start after the boot or just pause them. If you pause them you have to remember to start them manually when you need them (a lot of the silly little tray apps can quite safely be paused). If you hover over an item you get more information, including stats on what other folks have done with it (aka the “wisdom of the crowds”):

From there you can choose Pause, Delay or click “Advanced for even more info.

Pretty cool, eh? Just work through all the startup items and decided what you can pause and delay. Reboot and see how it works out. There’s a History option at the bottom of the Soluto screen that will show you a historical graph of your boot times.

I’d suggest not getting too aggressive on the first try. I ended up with a dramatically faster boot but some odd side-effects due to my over-aggressive ways. Fortunately, the UI displays what you’ve paused/disabled so you can re-enable them all back to boot and then try again (The FAQ says the next version will have a one-button reset which will be nice!).

Soluto’s goal is to bring an end to the frustrations PC users encounter, with transparency, killer technology, and the wisdom of the crowd. Soluto’s software combines low-level driver technology with collective wisdom, to detect PC users’ frustrations, reveal their cause, learn which actions really eliminate them and improve user experience. All this data is gathered in Soluto’s PC Genome, a knowledgebase of frustrations and solutions built automatically through the usage of Soluto software, for the benefit of all PC users.

Related posts:

1. Windows Startup Tuning
2. How to Slow Windows
3. Updating ProLiants Firmware and Software

I was having issues getting ghettoVCB backups to an NFS share on a Windows 2003 server. The VMware ESXi server would sporadically lose connection to the 2k3 server and then kill the backup. I finally replaced the little DLink SOHO 1GB switch with an HP ProCurve and then replaced all the sketchy old network cables with shiny new CAT 6 cables. The backups became noticeably faster and the intermittent connection losses completely disappeared.

Now I can get good backups for 3 out of the 6 Virtual Machines (VMs) on this server. Using any sort of file copy I can get copies of those same 3.

The other three? I’m starting to lose faith – I think we’re hosed. The copies or backups always end with a series of errors. The log errors point to the datastore where the VMs currently reside, not the copy destination. Read errors. Ugh.

Tip: I can’t seem to get “thin” backups to the Windows (or OpenFiler for that mattter) NFS shares. So, regardless of how much data is actually used in that 150 GB virtual disk, I get a full 150 GB backup file. As a workaround, I turned on NTFS compression for the NFS share at the Windows server. Slows the copy speed down by almost half with barely any extra CPU utilization. Worth it though as it took 280 GB of backups down to 16.5 GB!

I have a VMware forum post out there languishing. It did result in me making sure I had the latest/greatest firmware, ESXi updates and HP tools installed though. It also took me down a few unnecessary paths, but that’s OK as it was educational. I’ll probably close that post soon and try a much shorter and summarized version. I may have to figure out how to contact paid support.

I also tried a ServerFault.com post but I think I tried to cover too much territory in it. Face it, many geeks suffer from tl;dr syndrom. Think I’ll close that topic soon as well.

I am trying to get some help from HP now, but this time they’re not so interested in helping. See, at boot time the P400 array controller gives an error 1716 “unrecoverable media error.” HP says, logically enough, that I need to rebuild the array. OK, I’d like to do that but I want image backups first. They say I should’ve had good backups before I did any drive replacements. Well, that’s a good point. I did… but that was almost two weeks ago! *cough* excuse me… I’d just like fresh backups before I toast the array. These machines are all still in use and the company hasn’t been standing still.

There doesn’t appear to be a chkdsk or fsck for vmfs formatted volumes. Seems like that would be useful.

GhettoVCB errors are:

• Failed to clone disk : Connection timed out (7208969)
• Failed to clone disk : Input/output error (327689)

Sample message log errors:

Jul 15 19:18:50 vmkernel: 0:17:59:18.890 cpu4:16218)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x4100051614c0) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Jul 15 19:18:50 vmkernel: 0:17:59:18.890 cpu4:16218)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "mpx.vmhba1:C0:T1:L0" state in doubt; requested fast path state update...
Jul 15 19:18:50 vmkernel: 0:17:59:18.890 cpu4:16218)ScsiDeviceIO: 770: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Jul 15 19:18:53 vmkernel: 0:17:59:22.500 cpu4:5365)<4>cciss: cmd 0x4100b1402000 has CHECK CONDITION  byte 2 = 0x3

Here’s another set:

Jul 15 19:49:08 vmkernel: 0:18:29:37.553 cpu7:10409)<4>cciss: cmd 0x4100b1402000 has CHECK CONDITION  byte 2 = 0x3
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu7:10409)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x4100050fa000) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu7:10409)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "mpx.vmhba1:C0:T1:L0" state in doubt; requested fast path state update...
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu7:10409)ScsiDeviceIO: 770: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu6:312122)Fil3: 5354:  Sync READ error ('EFops_wsus-flat.vmdk') (ioFlags:  : Timeout

Unless I can figure out a way to get those last 3 VMs images or copied – or an alternative way to fix the read errors – I see a long weekend rebuilding machines in my future. Fortunately I can still get all the data from the VMs. I just can’t copy the VMs directly!

Related posts:

1. Wrestling with ESXi Client Backups
2. Windows NFS: Don’t Forget .maphosts

To recap, I had two new drives and a backplane en route and really wanted to get complete image backups of the VM clients running on this VMware ESXi server before I started swapping drives. I’d tried the vSphere built-in file copy utility, FTP and SCP but all had timeout issues on copying the large disk image (.vmdk) files.

While looking for more backup options I recalled reading about ghettoVCB in the past and this seemed like a great time to check it out. This is a very well written – free! – script that you can use to backup ESX(i) clients’ files to direct attached storage (which I’m avoiding for obvious reasons!) as well as NFS or iSCSI volumes. While I don’t have anything currently offering NFS or iSCSI drives I figured I could whip something together easily enough.

First off, I was looking at one of my Windows 2003 file servers. This machine has tons of unused storage and an unused network card so I figured it would be a good place to start. Unused network card is nice to keep all the network traffic for NFS/iSCSI off of the main network.

Hey, did you know that Windows Services for NFS is built-in to Windows 2003 and 2008 Server versions? I didn’t either until last week. Seemed worth a shot so I checked the box on the Windows Component Installer and gave it a shot.

Tip: When prompted for Windows media pay attention to the differences between x32 and x64 versions. I had no issues installing this component from x32 media – alas, the server is x64 and the service definitely didn’t start! Took an uninstall and a reboot to get back on track…

Once I sorted through some “user issues” I was ready to try connecting the ESXi server to the Windows exported NFS share. Sadly, no matter what I tried I couldn’t seem to hit the right combination of options to get this to work.

A few days later I noticed that checking “Microsoft Services for NFS” doesn’t give you all of the NFS service options… In particular, you don’t get the options around authentication and user mapping. This could very well explain why I couldn’t get it working. I will be revisiting this in the near future. I guess the other way to do this is via Windows Services for Unix, something I have very little experience with but might check into.

Time being of essence I set aside the Windows option and moved on to try with my OpenFiler server (first mentioned back in December of ‘09). Enabling the secondary network card and then configuring NFS on it took about 5 minutes. Connecting the ESXi server to it took about another minute. Easy!

I configured ghettoVCB and gave it a try. The first few machines backed up nicely! Alas, the next one had issues – a few file copy restarts and then it finally gave up after a timeout error.

Side note: My OpenFiler server is running an older pair of Hyper-Threaded Xeon 3.0 Ghz processors. When the backup to NFS is running it is seriously bogged down. That caught me by surprise.

After farting around with backups for (literally) over 12 hours I finally opted to accept the fact that I wasn’t going to get them all backed up and went ahead and did the hardware swaps. I figured perhaps the drives being in a degraded state were causing my issues.

Nope.

All the hardware has been replaced. All drives happy, RAID arrays rebuilt and everything is “green” and good to go on the server. I still can’t get reliable ghettoVCB backups and, perhaps related, I still can’t get full copies of the images using vSphere, FTP or SCP either.

At the moment I’m thinking this may be more of an ESXi issue since I can’t get good copies with or without NFS in the mix. Or maybe I need to try iSCSI from the OpenFiler server? I’ll need to spend more time on this one as I think ghettoVCB backups would be invaluable – once I can get all my stuff working.

Related posts:

1. Windows NFS: Don’t Forget .maphosts
2. Office Backups: The Journey, Part 2
3. Upgraded to VMware ESXi

Finally! I’m not afraid to admit that when I first read that post I actually thought “April Fools” and checked my calendar. I’m so glad that it isn’t April right now…

So not only are we getting rich text in sigs but we’re getting unique signatures for each email address associated with an account.

The next time you log in and visit the Settings page, you’ll see a rich text editor in the signature section. Here, you can customize your signature by adding pretty formatting, links, and images — or decide to leave things nice and simple.

Gmail also now supports a unique signature for each email address associated with your account. So, if you send mail using a custom “From:” address, you can use a different signature for that address. From the Settings page, you can edit the signature for each account by changing the email address that appears in the dropdown menu.

Can you tell I’m a little bit pumped about this? I am!

Related posts:

1. Google Mail Signature Management
2. Gmail Remote Sign Out Coming
3. Managing Multiple Gmail Accounts

Our patient was an HP ProLiant DL380 server that runs VMware ESXi as part of our nascent virtualization initiative. It used to be a database server and I’ve been quite pleased with how it handles running the majority of our development and QA servers. The pair of quad core CPUs rarely break a sweat.

Last week I happened to notice one of the hard drives (4 of which are in a RAID 1+0 array) had a solid amber LED lit. That is indicative of a failure so I called my friends at HP and they over-nighted a drive. I popped into the office Sunday afternoon and hot-swapped the old drive for the new one. Saw it spin up and go green and it seemed things were on track. I checked an hour or so later and it was still green and busy (presumably rebuilding the array) so I headed out of the office for a family function.

I came in Tuesday morning (2 days later) and was surprised to see the new drive — and another one two bays over — blinking amber. The green activity lights on both drives would sporadically flicker with activity but the blinking amber indicated a “Predictive Failure” is nigh.

HP wanted me to run diagnostics and send them reports so I scheduled an outage for Tuesday afternoon/evening. I figured that before I ran (potentially destructive) diagnostics I should maybe try to get some copies of the VM clients. See, I tend to treat each VM client just like any other server when it comes to backups. They all run backup software and enjoy nightly data backups. I’ve just never taken advantage of the fact that the VM client is really just a pile of files. And hey, if I lose the array I’d sure rather recover from an image of the server and then restore backups vs. building an all new server, patching it and then restoring from backups.

Don’t worry: I was shutting each machine down before trying to copy it.

I fired up  the vSphere Client app and ran the built in Datastore Browser. From there it is pretty simple to copy the machine directories. Simple and painfully slow! While looking for other options I stumbled over an article about enabling SSH on ESXi servers.

Tip: While on the console of your ESXi server hit alt + F1 and type “unsupported” and hit enter. You won’t see while you’re typing that but this will get you into the unsupported “Tech Support Mode” console. Very handy.

Tip: If you inadvertantly type “exit” in that console it appears to shut down. Toggling between alt + F1 and (the default) alt + F2 won’t help. Instead, while on the “dead” console, hit a bunch of enters. I don’t know how many, but more than 3… then type “unsupported” and hit enter again. Back in business!

Now that I had SSH working I fired up trusty WinSCP and tried pulling the files with it. Hmm, nope. That’s not any faster.

Tip: If using WinSCP, change the encryption cipher to Blowfish instead of the default AES for a bit of performance boost.

Some more digging turned up the fact that you can run an FTP server on the ESXi box. So, starting to panic about progress I gave that a shot next. Definitely faster but I wasn’t getting complete files. For one machine I might get a 20 GB .vmdk file, but then for the next I’d only get 9 out of 100 GB. It was inconsistent and frustrating since in some cases it would take an hour before crapping out. I tried watching the message log but it was so full of I/O errors that it wasn’t hardly worth using. At times it was scrolling by faster than I could read it!

Tip: Hit alt + F12 on the console to get to the ESXi “live” message log. You can scroll around all you want to examine errors but be aware that will stop the updates. Hit the spacebar when ready for it to resume updating.

I finally had to give up on copying server images. I just couldn’t get good copies and I’m sure it was related to the degraded drives (at least, I hope so!). I realized I was going to have to gut it out with the existing backups.

To run diags I had to boot the server from an HP SmartStart CD. While looking for a current version of that CD I stumbled over an update released last month: ** Critical ** Firmware CD Supplemental Online ROM Flash Component for Linux – Smart Array P400 and P400i. Huh. Critical, eh? I grabbed that update and added it to my bootable USB  key with the Smart Update Firmware DVD image.

Once booted from SmartStart I went into Maintenance mode and collected the array diag reports and then went to the Diagnostics and ran some complete diagnostics. The HP tech wanted me to run the complete 5 times… heck, it took over 2 hours to do it twice. I called that good enough. Saved every report option and then called HP back.

They reviewed the reports and agreed things did indeed look broke. Both drives were logging tons of hard read and write faults and S.M.A.R.T. was having fits (thus the predicted failure…). The also agreed that I should do that critical firmware flash. I did. They deemed that good enough and ordered me a pair of drives and a backplane card just in case it was a hardware fault (hey, I won’t turn down extra hardware!).

Now, I don’t know if was the reboots post hot-swap or that firmware update but when I brought the server back up I sure had a LOT fewer I/O errors in the ESXi log so that was progress. However, I still had two unhappy drives.

While waiting for the parts to show up I’d have a day to continue trying to get better client images.

(to be continued)

Related posts:

1. Upgraded to VMware ESXi
2. Wrestling with ESXi Client Backups
3. VMware ESXi Goes Free — Hmmm…