Well this is interesting. First of all, do not move any vhd or avhd files around, whether your guest VM is running or not.

I came back from a week’s vacation to find that my VMs were pretty much all broken. Awesomesauce. What happened was that the server that I run SCVMM on is also the Backup Exec server, and due to a mistake by some end-user, the size of the weekly backup jumped about 600gb and the backup2disk folder ran out of space and halted all backups. All the Virtual Machines paused themselves too because the host was out of hard drive space.

To alleviate the situation, a co-worker found 100gb or so of files in a “snapshot” folder under the VM’s folder and moved them elsewhere. What he didn’t know or realize was that these VM files have very specific ACLs that are tied to a username called NT Virtual Machine\{SID}.

When you move a file in Windows, if you’re copying on the same volume (say from My Pictures to My Pictures\vacation 2011) it will take it’s permissions with it. When you move a file to a different volume (to a D drive, or a flash drive or a network drive) it will inherit the permissions of it’s new home. Normally that’s a good thing, but for these snapshot files, it’s a bad thing. a very bad thing.

I discovered this when I found & moved the files back to where they were. The VM still would not start up and was giving all kinds of cryptic errors. unable to mount, unable to start virtual controller, things like that. I should have made a note of the exact errors and put them here for people to find, because figuring out what to do was a bit of a pain. Ultimately I found a KB article that described how to re-set the permissions and re-assign full control to the NT Virtual Machine\GUID user to the folder and then each of the avhd files directly using your favorite tool and mine: icacls.exe

This allowed the machine to re-start up and everything seemed to be OK so after 24 hours I thought I’d figure out how to get rid of those snapshot files and free up that space “the right way”. The first problem was that I did not have any snapshots of this VM, so how could I have snapshot files??

I found this article called “Hyper-V: What are these *.avhd files for? Snapshots? But I have no snapshots!” while Googling around and at first was stumped, because what he was displaying I could not see. I followed his directions to shut down the VM and power it off (the guest) and realized that yes it had been paused and rebooted, but it had never been shut down in nearly two years. I powered it off (it’s an MDT and WSUS server, so no “production” data on it) and looked around for the “merging 1%” to show up and it didn’t. I couldn’t figure it out! why couldn’t I see this happening in my SCVMM administrator’s console? On a whim, I decided to try the “local” Hyper-V MMC snap-in, so I fired up the Server Manager and drilled down to it. There it was, on the main screen under “Operations”: Merge in progress: 11%

I watched it for a few minutes and saw that one of the AVHD files disapeared! it was working! Awesome! so now it’s merging “the big file” which is where all the deployment images and WSUS download data was and is taking a while longer. As soon as the first AVHD file disapeared, I looked at the drives and saw that there was now 80GB free and the backup jobs resumed their steady march.

Once this is done, I’m going to have to do the same to the other Guest VM on this machine, which IS a production machine and probably has even more data in it, so that will have to wait for 5pm and run overnight.

Absolutely there is… and stop calling me Shirley.

Importing and exporting mailboxes is a pretty basic function, yeah? Take for example, hosted Exchange. Exchange in the cloud. Makes you feel light, fluffy… makes you think of ice cream, puppy dogs and unicorns… but if you’re not careful it will also make your wallet lighter, too.

This is one of the big things in “cloud” computing these days, is a lack of governance. A few years ago, it was virtualization and avoiding server-sprawl. Then it was SharePoint, governing that from spreading across your network like a cancer. Now it’s the cloud’s turn.

If you’re paying-by-the-cycle for hosted servers in the cloud, or even just for storage in the cloud, it makes sense to store the minimum required and pay the least amount of money, right? Same goes with hosting your mail server in the cloud. In our case, we pay a small fee per-month to cover the server and the account and that sorta sorta, but then we also pay per-mailbox on top of that. What happens when someone leaves the company? in the past, that sort of housekeeping, or mowing the lawn to use another analogy, just kinda got forgotten about. One day you’re running low on space and you start looking around and you find 20-30 old employees mailboxes are still laying around on the Exchange server. You don’t want to outright delete them, and you don’t have a proper archive solution in place, but you want to get them off the server because NOW, you’re paying a few bucks per month just to leave them sitting on the server.

It’s not so bad if there are four or five mailboxes sitting around. if you’re paying $10/month, that’s only $50 per month or $600 per year… pocket change when compared to the rest of the IT budget. A rounding error. What if it’s 50 mailboxes? what if it’s 100? now we’re talking the equivalent of someone’s bonus and that’s the kind of money Accounts is going to notice on Monday morning, Michael Bolton! You and your shitty little decimal error!… but I digress.

Easiest thing to do would be to export the ex-employees mailbox to a PST and store it somewhere on your own file server for now. There are two ways you can do it: you can set up a profile in Outlook, log in as them, sync your mailbox to a local OST file and then export to PST. OR you can go to the server, right-click on their mailbox and “export”… except they took that feature out from Exchange Server 2010 RC to the finished product. Oh well, at least you can do it in the Exchange Management Console, right? sorta.

First of all, the power wielded by import and export is so great that mere Enterprise Administrators cannot hold it. Like the sword of fucking Excalibur, only the anointed, chosen one can pull the Export-Mailbox cmdlet out of the stone.

Obstacle #1: add the “import export mailbox” role to yourself. Some of the blog posts I found actually recommended creating a new administrator account JUST for holding this role. I dunno about you, but I have enough passwords and logins to try and remember.

OK, so now I’m anointed, let me export the mailbox… not so fast.

Obstacle #2: you have to have Outlook installed. Outlook 2100 to be exact. No wait, it HAS to be Outlook 2010 x64. The version that Microsoft themselves tell you not to bother installing unless you have a specific need for Office x64. I suppose this would be one them. Once you have all those ducks in a row, congratulations, now you can do the sorts of tasks that are generally handed down the chain of command to the new guy or the intern.

Except I pretty much fly solo around here, so it’s back to me again. I suppose in the grander scheme of things, that because email is now evidentiary, and because of those ridiculous SOX compliance laws in the US and A, that you have to make sure no one can tamper with the mailbox, but what a friggin hassle. SOX is the main reason I didn’t even bother returning any phone calls for jobs in the US last time I was looking.

To finish off this tragedy, in order to exercise some governance over cloud sprawl, and save a few bucks per month paying for mailboxes that are no longer used, I’m having to spin up a Windows 7 Pro x64 Virtual Machine, patch it, then install Office 2010 x64 (even though I’m not even going to use it) and then install the Exchange Management Tools and configure those and Powershell, JUST to export some frigging mailboxes from the mail server.

Last year I set up a Windows Server 2008 Core server. It was a Hyper-V virtual machine, it was minimum-spec, it didn’t do much other than be a second Domain Controller on the network so I hardly ever had to interact with it. Based on that criteria, and because I wanted to see what it was like, I installed Windows Server 2008 Core.

Windows Server 2008 Core if you’re not familiar is a Windows server with no windows: when you log in, you get a command prompt, and that’s it.

Configuring it after installing was a bit of a bear, because instead of clicking anything, you had to learn, know and type the commands into the terminal, along with all the arguments/switches. I got it set up, configured, joined to the domain and then promoted to be a domain controller and that was pretty much it. I set it up so that I could use Remote Desktop to connect to it, but what I really wanted to do was use the Server Manager on another server to connect to it and manipulate it that way.

I found out the hard way that you can’t really do that. I did find a piece of software written in Visual Basic called CoreConfigurator which created a text-menu-based configuration helper and it was pretty good. They also had a Version 2 which was written in Powershell that had a bit of a GUI to it… but it wasn’t compatible with Windows Server 2008 (the Vista server, if you will) only Windows Server 2008 R2 (the Windows 7 server). I pretty much dropped it after that, since it was running and I didn’t need to do anything to it.

Eventually I upgraded it to Server 2008 R2 when my licensing allowed me to and then I could use CoreConfigurator V2.0. Remote management still wasn’t working, despite the server’s command-line status updates to the contrary. Again, it was working and I had more important things to do.

Today I was trying to track down something (seemingly) entirely unrelated. Some clients could access a DFS share on the domain, and others could not. I followed the trail to the Domain Controller (DC1) and checked DNS services, and they were all fine. I then looked at DC1’s DNS servers and it was pointing at DC2 (the Server Core) so I opened it up and checked it out. I thought to myself “Wouldn’t it be nice if I could control DC2 with the Server Manager on DC1?” so I decided to take another run at it.

On DC2 I entered winrm quickconfig to see what was configured. As expected, it said:
WinRM already is set up to receive requests on this machine.
WinRM already is set up for remote management on this machine.

So I tried “Connect to another computer” in Server Manager and… bonk. “Server Manager cannot connect to server_name. Click retry to try to connect again.” opening the details tab had more detail, but it’s pretty much all gibberish even to me. “Connecting to remote server failed with the following error message: The WS-Management service cannot process the request. The resource URI ...://schemas.microsoft.com/powershell/Microsoft.ServerM... was not found in the WS-Management catalog. The catalog contains the metadata that describes resources, or logical endpoints.” Right.

I started with the error code, and then the hex code and ultimately ended up at a Microsoft KnowledgeBase article that hit the nail right on the head.

Error message in Windows Server 2008 R2 or in Windows 7 when you try to connect to a remote server: "Server Manager cannot connect to <server_name>"

Following this article, I typed sconfig from the command-line on the server core, chose item 4 “Configure Remote Management” and then option 3 “Allow Server Manager Remote Management”. It then re-configured Win-RM (which was already configured correctly) but interestingly added three new rules! It didn’t say what those rules were, but after restarting the server (because I had to enable PowerShell) I was able to connect to the server using Server Manager from any of my other servers or my Windows 7 laptop.

You know what I like about taking Part Time Studies computer classes at BCIT at night? The crowd is a little older, and they’re all already nerds.

I’m taking COMP1451 this semester, which is part two/continuation of COMP1409, “Introduction to Object Oriented Programming” For this class, we’ve been using BlueJ as the… well, I don’t want to call it a development environment, but it kinda is. It’s pretty cool, it’s great for illustrating concepts using Java, but it’s not really a full-on Java development tool.

We’re just over half-way through part 2 of the course and tonight the instructor introduced Eclipse. Eclipse IS a full-blown Java Development Environment, and we spent the evening learning (and re-learning) the differences between what we thought was Java and what really is Java.

One of the exercises we did was to use Eclipse to generate constructors and source code automatically, saving a lot of grunt work typing of really basic things. In BlueJ, one of the things you could do was hit ctrl+m and it would insert a new method at the cursor, complete with javadoc comments. You could fill in what you needed, take out what you didn’t and carry on.

/**
     * An example of a method - replace this comment with your own
     *
     * @param  y   a sample parameter for a method
     * @return     the sum of x and y
     */
    public int sampleMethod(int y)
    {
        // put your code here
        return y;
    }

I used that shortcut a lot, and even developed a little muscle-memory to ctrl-M every time I needed to create a new method. I asked the teacher if there was something similar in Eclipse to save a few keystrokes and she wasn’t sure. We poked around a bit in the Source menu before she said “well, I guess you’ll have to do it the old-fashioned way” and pointed at the keyboard.

I entwined my fingers, turned them inside out as if to crack my knuckles and said “A keyboard. How quaint…” I reached for the mouse, picked it and spoke into it like a microphone “Oh, Computer…” and without even imitating a bad Scottish accent, the instructor, the TA and the two guys on either side of me cracked up.

Someone later pointed out that that movie is 25 years old, but ya know what? Classics never get old.

It took almost five years. Coincidentally that’s the length of the warranty I was trying to get it repaired under, but finally, something was done. There was a management change at Divers Supply Grand Cayman, and the new manager wanted to make things right whereas the old manager told me to go fuck myself.

Shortly before Christmas 2010, I was contacted by the new manager of the store (on Facebook) who reached out to make things right.

We talked back and forth and came to an agreement, and I edited and updated the posts from 2006 calling out his store for treating me like an inmate in a shower. I posted an update at the top of the three posts with the date, and then edited some of the piss & vinegar out of my rants. As I figured, it was the then-manager who was behind the absolutely horrendous “service” and treatment. Now that he’s gone (and probably most of the staff) it’s not fair to paint them all with the same brush as him (although given a chance, I don’t think I’d use a paintbrush on him).

It’s going to take awhile (if ever) for the Google search results for Divers Supply Grand Cayman to drop the old titles of the posts and replace them with the newer, gentler ones, but if anyone clicks on any of the links to my previous posts about Divers Supply Grand Cayman then they’ll see the new titles and the update, in bold, at the top of the page.

My replacement dive computer is up here now (although it’s still out at my friend’s house who carried it up here at Christmas from Cayman) so it was the right time to make a new post and edit the old posts.

I started out the task flying pretty high. I worked on a deployment for some new HP laptops and Windows 7 Pro x64 and things were working out as planned.

Once I got it to where I could PXE boot the laptop, connect to the deployment share and lay the Windows 7 x64 image down on it, I was time to get down to the nitty gritty: Drivers. Applications. Packages. Automation.

Drivers were fairly easy, I’ve been importing them for awhile now, but what I wanted to do was to segregate them into distinct little piles, rather than one motherlovin’ huge pile of inf files and I wanted a computer to only get the drivers it needed for itself, not the whole lot of them.

MDT 2010 provides for this, and there are plenty of goodtutorials out there on the net waiting to be found, so I won’t “waste ink” posting it here again. I highly recommend you use the Readability bookmarklet before going to any of the articles on that site, though. They have ads and crap on all 3 sides and a narrow column in the middle with small text for the actual article.

So we got a bare-bones Windows 7 install at this point, with a bunch of Unknown Devices in the Device Manager. Windows 7 is smart enough that most of them have drivers advertised through Windows Update so right-clicking them and selecting “update driver” will find it… but that’s not why we’re using deployment tools, I want it to come out the other end of my process shiny and clean and ready to be used. Following information in those links above and elsewhere, I was able to have WindowsPE detect the make & model of the laptop, and then look that up in my deployment database and download the drivers I specified. Awesome! All but one… one sticky wicket that wouldn’t work because the manufacturer chose to make the driver file a software installation, instead of just a driver. (hate)

On to the Applications settings in MDT 2010 then! Applications don’t work as well as the drivers do. There’s no Selection Profiles for applications like there are for Drivers. Sure you can set MandatoryInstallation <guid> in the customsettings.ini file for the whole deployment share, but then they get installed on every machine that connects, not just the one laptop model that needs this particular driver, so that’s out, too.

Searching around on this topic led me to the Make & Model settings under Advanced Settings>Database. I created a new entry using the Make and Model of the laptop using the data I got from the BIOS. To find out what yours is, drop to a command prompt and type ‘wmic csproduct get vendor’ or get name. Once you’ve created an entry, you can double-click it to open it’s properties and assign things like Applications, Roles and Administrators. Applications is the one we’re looking for here so I clicked on that tab and then clicked Add. I then selected the Driver software.exe that I had set up (as a silent install… another topic!) and then clicked OK. I updated my deployment share and… it didn’t work.

I tried a few different things, I checked, double-checked, and triple-checked that I got the Vendor and Name correct, I tried moving the application around within the deployment share, but nothing worked. Because I was working with a physical machine, it took about 30 minutes to test out each iteration. While it was doing that, I opened the ZTIGather.log on my virtual machine that I had deployed to yesterday, which is in C:\Windows\Temp\DeploymentLogs and using the Vendor and Name in there, I created another entry in the database and assigned it a very small application (most of the apps I have in the repository are huge… Autocad, Office, etc.) to try that one out. I updated the deployment share and this time, just in case, I also went into Windows Deployment System and replaced the boot image with this newly generated one.

I booted the VM up, let it PXE boot, selected x64 boot image and stepped through the Wizard and when I got to the Applications screen… Holy smokes it was there! pre-checked! I tried un-checking it and then clicked next, but then went back and it was re-checked, so it was treating it as a mandatory application, but only on that make & model of computer! I then rebooted the laptop into the same x64 boot image to see if it was working for my original problem. If it wasn’t, at least I had proved that it wasn’t an error with my database. I flipped through the screens to Applications and the driver was there and pre-checked! Hooray! hurried through the rest of it and let it deploy. Once it got to the Windows 7 desktop and the last stages of the deployment were running, it installed the driver software. I rebooted (windows update kicked in right away) and when it restarted, I checked out the device manager: Nothing was showing as Unknown Device! Hooray! One machine down, 2 more to go, get a few more apps in there and my MDT 2010 deployment share will be ready to kick out the Win7 Pro x64 jams to all comers! (well, within my company and licensing agreement, anyway)

Last night I logged into work from home to initiate a reboot of all the servers. Windows Updates were pending, and had been pending for about a week, but it’s hard to reboot production servers in the middle of the day when people are using it. Throw in some Flex Hours, and they’re in use from 6am to about 8pm.

The Domain Controllers have their own policy for updates, and they’re still required to be initiated manually, and then “restart now” clicked to reboot them.

When new “critical” patches are released and there are known 0-day flaws being exploited, I’ll use the ‘deadline’ feature in Windows Software Update Services (sort of a mini Windows Update server you can run on your own, approve and distribute updates around your own network but only downloading it once from Microsoft) where if a deadline passes and a user has been clicking “restart later” it will disable that button and start a 15 minute countdown before it forcibly reboots.

There was no deadline on this latest batch of updates from the last Patch Tuesday, so the (member) servers were politely asking to be rebooted. I logged into each of them one by one and clicked “restart now” and then waited for them to shut down, restart, and start back up again.

All of them worked and came back up (according to pinging them for responsiveness) except one. It SEEMED to come back up. I could ping it and it responded, so I moved on to the next and the next and the next.

It wasn’t until this morning when I walked in the door and had four people waiting for me saying “the network is down” (which of course was a misnomer, the network wasn’t down, it was just the shares on THE MAIN FILE SERVER that were disconnected) I poked my head into the server room, and the KVM was already set to that server and on the screen (which was blue, but not that Blue) was “Configuring Updates stage 3 of 3 0% Do not turn off your computer” I watched it for a minute to see what happens, as the hard drive LEDs were blinking away, so it WAS doing SOMETHING… then the screen went black.

The cursor was flashing up in the upper-left, so I waited some more… then the BIOS splash screen came up. The server had rebooted itself.

Turns out it had been in this startup, stage 3, fail, reboot loop since 9:00 last night.

Step 1, try a cold-boot. I waited for it to fail again, and then I held down the power button until it powered off. I removed the power cables and let it sit for 30 seconds to make sure everything had powered off, plugged it back in and tried again. Same result.

Step 2, try Safe Mode…. Applying Computer Settings… Configuring Updates stage 3 of 3… reboot. Crap.

Step 3, Last Known Good Configuration. This resets key windows files back to how they were the last time you successfully logged in. You would think that this would break it out of a bad update loop. You would be wrong.

Step 4, booted from the Windows Server 2008 x64 DVD and clicked on Repair. There’s a new “Startup repair” tool that’s incl-wait, it’s not? only in Server 2008 R2 that’s based on Windows 7 and NOT in Server 2008 that’s based on Vista? There are NO repair options for Server 2008 other than re-imaging of the system from the latest full-system-image? You DO have one of those, right?

Step 5, Uncle Google suggested I click through to “Get Vista out of the Infinite Reboot Loop” and the comment there by Tribus was:

I know a different way to resolve this issue without using a restore point.
1. Insert your Vista Media into your dirve and boot from it.
2. Select "Repair your Computer" from the list.
3. Select "Command Prompt" from the recovery choices.
4. At the command prompt change your directory to C:WindowsWinSxS
5. Type: del pending.xml
6. Exit and reboot
This will fix all Windows update reboot loops and does not require you to restore your PC to and earlier state.

Figuring I had nothing else left to lose, I gave this suggestion a shot, even though it was for Vista. If this didn’t work, then I’d be getting on the horn to Microsoft Support for some help. Instead of deleting it, I renamed it pending.xml.old and then exited and rebooted.

“Applying computer settings…” OK so far so good…

“Configuring Updates stage 3 of 3 0%. Do not turn off your computer…” FUCKBURGERS!!!

“Press Ctrl+Alt+Del to Begin” WHAAAAAAAAAAAT? it worked.

Once it was up and running the first thing I did (other than tell the users they could access their files again) was to look in the event log and see what happened. On the first reboot last night at 9pm, there was an event from source Winlogon, Event ID 6004 “The winlogon notification subscriber <TrustedInstaller>failed a critical notification event".”

So the next step is to research that error and see if I can figure out WHICH update caused it… it could be a moot point though because my co-worker turned up some early results that once you do this, you’ve pretty much broken Windows Update on this computer forever. I can live with that for now, because people are working and the data is intact. If I figure out that that is the case, and figure out a workaround, I’ll post a follow-up.