I can hear some of you (both of you still reading, thank you loyal fans) asking what finally brought me back, and I have to say it’s a rant. A rant about autism (and Asperger’s, and the rest of the spectrum), how it is perceived, and how trendy equals insensitive. You have been warned.

Hip To Be Square

After karate class tonight on the drive home, Steph was reading through Facebook (something I do but occasionally these days, having overdosed myself on social media some time ago) and came across the following comment on a mutual friend’s post:

Yes, that really does say that stupid thing

For some reason, this really punched my buttons. I don’t know much about the person who posted it. I don’t know if they’re a fellow spectrum traveller or not. I don’t know how many close friends or family members they have who have autism. To a certain extent, it really doesn’t matter, because this comment is a textbook illustration of a fallacy that I’m seeing more and more:

If geeks are cool, and a lot of geeks are autistic, they must be cool because they are autistic.

This is a fallacy because it is the living embodiment of failure to grasp proper logic and set theory. This growing "Autism Is The New Cool" meme (AITNC for those of us who adore our acronyms), for lack of a better word, is reaching stupid proportions.

Venn We Dance

Now listen up, because if you’d paid attention in Algebra the first time, I wouldn’t have to be telling you this shit now.

What we are talking about here are properties that people have: the property of being cool, the property of being a geek, and the property of being on the autism spectrum. These are not variables that we can just slam together in a transitive[1] orgy of equation signs, as much as someone might like to be able to write on a whiteboard that A=B=C.

You get to stay after class and wipe down the whiteboard

Instead, we need to head over to set theory, which is where we look at groupings (or "sets") of objects, where said sets are organized by a shared trait. Such as being a geek, or being cool, or being on the autism spectrum. We represent these sets by drawing circles. Then we can make useful and interesting (and sometimes even more occasionally related to real life) observations by seeing where these sets overlap and what that tells us. This is a Venn diagram, and it helps us immediately destroy AITNC, because it reminds us that people (the members of the sets) are not single-value variables like A and B and C and the rest of their letter trash, but complex people who are not in any way entirely equal. This is my AITNC mega-buster Venn diagram, whipped up on this evening when I had lots of better stuff to do, just for your edification:

Filling in the missing names is left as an exercise for the reader[2]

Note that there are plenty of places where there is no overlap. Note that there are four separate regions where there are overlap. I can think of people who are examples of each of those areas, but I’m not enough of a dick to tell you who they are.

The Big Boy/Girl Panties Are Right Over There

I have, I shit you not, had parents ask me how to get their kid diagnosed with Asperger’s so they can "give him an extra educational advantage" (or some such nonsense). Yeah, I know. Fucked up, right?

I’m no child psychology professional, but I know spoiled, overly sugared kids when I see them. You want your kid to get an extra educational advantage? Don’t let the little bastards play video games and watch TV when they get home from school. Make them do homework and chores. Stop buying them everything they want and make them earn a meager amount of money and prioritize they things they really want from passing whims. Spend time with them and find out what they’re learning. Teach them about things you’re doing, which means you might want to put down the remote and pick up some more books or spend time outdoors or in your shop. Take the time to buy and prepare healthy food instead of boxed-up pre-digested pap. Teach them how to cook and clean, while you’re at it. Get involved with what they’re doing at school and be ruthlessly nosy about their grades and progress. Limit their after-school activities so they have time to study. Make and enforce a reasonable bedtime. In short, be a fucking parent. Stick with that for a year, and I guarantee your kids will have an educational advantage that you can’t believe.

Unless you want it in kebabs for dinner

Once you’ve done that for a few years and your kids have adjusted to having the meanest parents on the block like mine have, then you can worry about whether your precious little shit belongs on the autism spectrum, or has ADHD, or whatever other crutch diagnosis you think you need to compensate for being a mere gamete donor instead of a real parent.

People Are Strange (When You’re A Stranger)

I’m not going to sing a litany of woes about how tough it is being Asperger’s. I have fought most of my adult life to keep this thing from defining who I am. Devin != autism, not by a long shot. It’s one of a large number of properties about me, and it’s a mere footnote at that. I refuse to self-identify as an "Aspie" because I see that many of them (not all, but a significant fraction of them) use it as a Get Out Of Life Free card. "Oh, boohoo, I can’t make friends. Boohoo, I can’t have a relationship. Boohoo, my boss doesn’t understand me." I’ll grant it makes things difficult at times, but you know what? I look at so-called "neurotypical" people and they seem to have rough patches too. Life isn’t perfect for anyone. I don’t know how much harder my life is because of Asperger’s, and you don’t either. Anyone who claims to know is full of shit. At best, they’re making wild-ass guesses.

I choose not to play "what-if" games, because there is always something you think of after the fact. This wiring malfunction in my brain does not define or control me unless I choose to let it. The only reason its effects dominated my life through my early adulthood is that I didn’t know. Once I knew…well, I went all G. I. Joe[3] on its ass.

You know what really sucks? That my wife and kids have to be hyper-vigilant about what food they eat because their bodies are attacking their own auto-immune systems. I can tell you exactly how much of a crimp that’s put into their enjoyment of life. One thoughtless dweeb in a restaurant kitchen who doesn’t properly wash bread crumbs off a counter, or clean off that dollop of butter on the knife, can make them miserable for a week. That’s a pretty raw deal, friends. Asperger’s has nothing on that. Try traveling or going out to a restaurant with friends. The number of things you can eat with one of the 8 major food allergies quickly limits your options. Enjoy two of them (like my family) and you can start counting your dining options on one hand.

So if you’re one of those assholes who thinks autism is cool or glamorous, get a life. Seriously. Be thankful for what you have. And recognize that people are cool not because of their afflictions but because they are cool people.

[1] You’ll probably have forgotten in five minutes, but transitive means if one thing is equal to a second thing, and a third thing is also equal to the second thing, then the first and third things are equal too. This only usually works in math and quantum mechanics, because how often are two things actually equal in the real world?

[2] Extra credit if you noticed that I really did match the color coding between the two diagrams.Without thinking.

[3] "Knowing is half the battle."

Here…watch this:

Now, go do something about it.

Last week, I was talking with a co-worker about Exchange’s design requirements in a virtual environment. I casually mentioned the “no file-level storage protocols” restriction for the underlying storage and suddenly, the conversation turned a bit more serious. Many people who deploy VMware create large data stores on their SAN and share them to the ESX cluster via the NFS protocol. There are a lot of advantages to doing it this way, and it’s a very flexible and relatively easy way to deploy VMs. However, it’s not supported for Exchange VMs.

The Heck You Say?

“But Devin,” I can hear some of you say, “what do you mean it’s not supported to run Exchange VMs on NFS-mounted data stores? I deploy all of my virtual machines using VMDKs on NFS-mounted data stores. I have my Exchange servers there. It all works.”

It probably does work. Whether or not it works, though, it’s not a supported configuration, and one thing Masters are trained to hate with a passion is letting people deploy Exchange in a way that gives them no safety net. It is an essential tool in your toolkit to have the benefit of Microsoft product support to walk you through the times when you get into a strange or deep problem.

Let’s take a look at Microsoft’s actual support statements. For Exchange 2010, Microsoft has the following to say in http://technet.microsoft.com/en-us/library/aa996719.aspx under virtualization (emphasis added):

The storage used by the Exchange guest machine for storage of Exchange data (for example, mailbox databases or Hub transport queues) can be virtual storage of a fixed size (for example, fixed virtual hard disks (VHDs) in a Hyper-V environment), SCSI pass-through storage, or Internet SCSI (iSCSI) storage. Pass-through storage is storage that’s configured at the host level and dedicated to one guest machine. All storage used by an Exchange guest machine for storage of Exchange data must be block-level storage because Exchange 2010 doesn’t support the use of network attached storage (NAS) volumes. Also, NAS storage that’s presented to the guest as block-level storage via the hypervisor isn’t supported.

Exchange 2007 has pretty much the same restrictions as shown in the http://technet.microsoft.com/en-us/library/bb738146(EXCHG.80).aspx TechNet topic. What about Exchange 2003? Well, that’s trickier; Exchange 2003 was never officially supported under any virtualization environment other than Microsoft Virtual Server 2005 R2.

The gist of the message is this: it is not supported by Microsoft for Exchange virtual machines to use disk volumes that are on file-level storage such as NFS or CIFS/SMB, if those disk volumes hold Exchange data. I realize this is a huge statement, so let me unpack this a bit. I’m going to assume a VMware environment here, but these statements are equally true for Hyper-V or any other hypervisor supported under the Microsoft SVVP.

While the rest of the discussion will focus on VMware and NFS, all of the points made are equally valid for SMB/CIFS and other virtualization system. (From a performance standpoint, I would not personally want to use SMB for backing virtual data stores; NFS, in my experience, is much better optimized for the kind of large-scale operations that virtualization clusters require. I know Microsoft is making great strides in improving the performance of SMB, but I don’t know if it’s there yet.

It’s Just Microsoft, Right?

So is there any way to design around this? Could I, in theory, deploy Exchange this way and still get support from my virtualization vendor? A lot of people I talk to point to a whitepaper that VMware published in 2009 that showed the relative performance of Exchange 2007 over iSCSI, FC, and NFS. They use this paper as “proof” that Exchange over NFS is supported.

Not so much, at least not with VMware. The original restriction may come from the Exchange product group (other Microsoft workloads are supported in this configuration), but the other vendors certainly know the limitation and honor it in their guidance. Look at VMware’s Exchange 2010 best practices at http://www.vmware.com/files/pdf/Exchange_2010_on_VMware_-_Best_Practices_Guide.pdf on page 13:

It is important to note that there are several different shared-storage options available to ESX (iSCSI, Fibre Channel, NAS, etc.); however, Microsoft does not currently support NFS for the Mailbox Server role (clustered or standalone). For Mailbox servers that belong to a Database Availability Group, only Fibre Channel is currently supported; iSCSI can be used for standalone mailbox servers. To see the most recent list of compatibilities please consult the latest VMware Compatibility Guides.

According to this document, VMware is even slightly more restrictive! If you’re going to use RDMs (this section is talking about RDMs, so don’t take the iSCSI/FC statement as a limit on guest-level volume mounts), VMware is saying that you can’t use iSCSI RDMs, only FC RDMs.

Now, I believe – and there is good evidence to support me – that this guidance as written is actually slightly wrong:

• The HT queue database is also an ESE database and is subject to the same limitations; this is pretty clear on a thorough read-through of the Exchange 2010 requirements in TechNet. Many people leave the HT queue database on the same volume they install Exchange to, which means that volume also cannot be presented via NFS. If you follow best practices, you move this queue database to a separate volume (which should be an RDM or guest-mounted iSCSI/FC LUN).
• NetApp, one of the big storage vendors that supports the NFS-mounted VMware data store configuration, only supports Exchange databases mounted via FC/iSCSI LUNs using SnapManager for Exchange (SME) as shown in NetApp TR-3845. Additionally, in the join NetApp-VMware-Cisco performance whitepaper on virtualizing Microsoft workloads, the only configuration tested for Exchange 2010 is FC LUNs (TR-3785).
• It is my understanding that the product group’s definition of Exchange files doesn’t just extend to ESE files and transaction logs, but to all of the Exchange binaries and associated files. I have not yet been able to find a published source to document this interpretation, but I am working on it.
• I am not aware of any Microsoft-related restriction about iSCSI + DAG. This VMware Exchange 2010 best practices document (published in 2010) is the only source I’ve seen mention this restriction, and in fact, the latest VMware Microsoft clustering support matrix (published in June 2011) lists no such restriction. Microsoft’s guidelines seem to imply that block storage is block storage is block storage when it comes to “SCSI pass-through storage”). I have queries in to nail this one down because I’ve been asking in various communities for well over a year with no clear resolution other than, “That’s the way VMware is doing it.”

Okay, So Now What?

When I’m designing layouts for customers who are used to deploying Windows VMs via NFS-mounted VMDKs, I have a couple of options. My preferred option, if they’re also using RDMs, is to just have them provision one more RDM for the system drive and avoid NFS entirely for Exchange servers. That way, if my customer does have to call Microsoft support, we don’t have to worry about the issue at all.

However, that’s not always possible. My customer may have strict VM provisioning processes in place, have limited non-NFS storage to provision, or have some other reason why they need to use NFS-based VMDKs. In this case, I have found the following base layout to work well:

Volume	Type	Notes
C:	VMDK or RDM	Can be on any type of supported data store. Should be sized to include static page file of size PhysicalRAM + 10 MB.
E:	RDM or guest iSCSI/FC	iSCSI/FC    All Exchange binaries installed here. Move IIS files here (scripts out on Internet to do this for you). Create an E:\Exchdata directory and use NTFS mount points to mount each of the data volumes the guest will mount.
Data volumes	RDM or guest iSCSI/FC	Any volume holding mailbox/PF database EDB or logs, or HT queue EDB or logs. Should mount these separately, NTFS mount points recommended. Format these NTFS volumes with 64K block size, not default.

Note that we have several implicit best practices in use here:

• Static page file, properly sized for a 64-bit operating system with a large amount of physical RAM. Doing this ensures that you have enough virtual memory for the Exchange memory profile AND that you can write a kernel memory crash dump to disk in the event of a blue screen. (If the page file is not sized properly, or is not on C:, the full dump cannot be written to disk.)
• Exchange binaries not installed on the system drive. This makes restores much easier. Since Exchange uses IIS heavily, I recommend moving the IIS data files (the inetpub and children folders) off of the system drive and onto the Exchange volume. This helps reduce the rate of change on the system drive and offers other benefits such as making it easier to properly configure anti-virus exclusions.
• The use of NTFS mount points (which mount the volume to a directory) instead of separate drive letters. For large DAGs, you can easily have a large number of volumes per MB role, making the use of drive letters a limitation on scalability. NTFS mount points work just like Unix mount points and work terribly well – they’ve been supported since Exchange 2003 and recommended since the late Exchange 2003 era for larger clusters. In Exchange 2007 and 2010 continuous replication environments (CCR, SCR, DAG), all copies must have the same pathnames.
• Using NTFS 64K block allocations for any volumes that hold ESE databases. While not technically necessary for log partitions, doing so does not hurt performance.

So Why Is This Even A Problem?

This is the money question, isn’t it? Windows itself is supported under this configuration. Even SQL Server is. Why not Exchange?

At heart, it comes down to this: the Exchange ESE database engine is a very finely-tuned piece of software, honed for over 15 years. During that time, with only one exception (the Windows Storage Server 2003 Feature Pack 1, which allowed storage solutions running WSS 2003 + FP1 to host Exchange database files over NAS protocols), Exchange has never supported putting Exchange database files over file-level storage. I’m not enough of an expert on ESE to whip up a true detailed answer, but here is what I understand about it.

Unlike SQL Server, ESE is not a general purpose database engine. SQL is optimized to run relational databases of all types. The Exchange flavor of ESE is optimized for just one type of data: Exchange. As a result, ESE has far more intimate knowledge about the data than any SQL Server instance can. ESE provides a lot of performance boosts for I/O hungry Exchange databases and it can do so precisely because it can make certain assumptions. One of those assumptions is that it’s talking to block-level storage.

When a host process commits writes to storage, there’s a very real difference in the semantics of the write operation between block-level protocols and file-level protocols. Exchange, in particular, depends dramatically on precise control over block-level writes – which file protocols like NFS and SMB can mask. The cases under which this can cause data corruption for Exchange are admittedly corner cases, but they do exist and they can cause impressive damage.

Cleaning Up

What should we do about it if we have an Exchange deployment that is in violation of these support guidelines?

Ideally, we fix it. Microsoft’s support stance is very clear on this point, and in the unlikely event that data loss occurs in this configuration, Microsoft support is going to point at the virtualization/storage vendors and say, “Get them to fix it.” I am not personally aware of any cases of a configuration like this causing data loss or corruption, but I am not the Exchange Product Group – they get access to an amazing amount of data.

At the very least, you need to understand and document that you are in an unsupported configuration so that you can make appropriate plans to get into support as you roll out new servers or upgrade to future versions of Exchange. This is where getting a good Exchange consultant to do an Exchange health check can help you get what you need and provide the support you need with your management – we will document this in black and white and help provide the outside validation you might need to get things put right.

One request for the commenters: if all you’re going to do is say, “Well we run this way and have no problems,” don’t bother. I know and stipulate that there are many environments out there running in violating of this support boundary that have not (yet) run into issues. I’ve never said it won’t work. There are a lot of things we can do, but that doesn’t mean we should do them. At the same time, at the end of the day – if you know the issues and potential risks, you have to make the design decision that’s right for your organization. Just make sure it’s an informed (and documented, and signed-off!) decision.

Whidbey Island

The first thing I did was do a little research to locate a candidate list of reasonable bed and breakfasts for us to stay at. Steph had never before been to one and, frankly, hotels are boring. Ideally, I wanted one that was based out of a Victorian house, since Steph loves them. Potential bed & breakfasts of course would have to be able to handle the no-dairy/no-gluten restrictions. I really wanted to find one on Whidbey Island, which is close off-shore in the Sound, separated from Fidalgo Island by Deception Pass.

Why did I want to go to Whidbey Island for our overnight?

• Islands are picturesque as hell. On the right island, you’re always close to the water, which I love.
• We’d only been there once previously, during a quick drive-around last September when we got our new car.
• Whidbey is one of the bigger islands in the Sound. It hosts several towns and has a high enough population to still offer some great experiences even during the depths of the off-season.
• You can drive to it (by the bridge to Fidalgo Island, then by the bridge over Deception Pass to the north end of the island) or take the Mukilteo-Clinton ferry at the south end of the island. Transportation flexibility in winter months is a good thing.
• Our most direct route to Whidbey Island is the ferry route, which runs every half-hour and is a short 20 minute ride. This helps achieve Steph’s goal of riding every ferry route in the Sound at least once. It also indulges my love of being out on the water.

Once I had a couple of candidates and knew what their check-in times were, I could work backwards for travel times and ferry crossings and determine the window of time in which we’d need to leave. This gave me the all-important time: my cut-off for the work day. Armed with this time, I set up an out-of-office calendar appointment and clearly communicated with my co-workers and clients that I had a hard stop at 3pm. As I set up each part of the weekend reservations, I sent Steph appropriate meeting requests in our shared Outlook/Exchange calendar. This let her know what my plans were and gave her the links and information she’d need to poke around and do her own reading. It seemed to work, because I quickly got acceptance notices and by Wednesday, Steph was practically bouncing off the walls in anticipation!

Whidbey Island (from the Whidbey Island Visitors Guide website)

Once Friday came, Steph was obviously eager to be off on our adventure. I think she was packed to go by 10am. At any rate, I was promptly done with work by 3pm, took a few minutes to pack, and we were out the door by 3:45pm as I’d planned. We took a quick detour to run a necessary errand, then headed for the Mukilteo ferry terminal. We arrived in time to queue up and watch (but not participate in) the loading of the 5pm ferry crossing as the sun set; it would be our turn in 30 minutes. The ride across the Sound was quick and cold in the gloaming, and we made our way north up the island until arriving at Coupeville.

The Blue Goose Inn

After doing some homework and reading reviews, it became clear that my #1 choice was going to be The Blue Goose Inn in Coupeville, overlooking Penn Cove in central Whidbey Island. Proprietors Sue and Marty McDaniel offer a fantastic getaway experience out of two lovely restored Victorian historical homes, and during a good portion of the year also operate a pub on-premises (sadly, it was closed during our visit). When I called to inquire, Sue assured me that the dietary restrictions would be no problem. A few minutes later, I had chosen the Captain’s Suite because of the king-size bed, the soaking tub, and the view of Penn Cove; we had our reservation!

Stephanie in front of The Blue Goose Inn in Coupeville, WA

Even though we arrived after sunset, Stephanie could see enough details that she was delighted by the choice. As we walked in the front door and were greeted by Sue and Marty, we immediately felt welcome. As I’d taken care of payment over the phone when I made the reservations, there was no paperwork to take care of; we chatted for a few minutes, they approved of my choice of venue for dinner, gave us our room key to the Captain’s Suite in the Coupe House, explained the accommodations that were available besides our room, and sent us on our way. We were not disappointed; the room was lovely and tastefully appointed with beautiful and functional furniture. In many older homes, drafts can be a problem, especially on a cold, windy night; this was not a problem here! The room was comfortable without being stuffy or unpleasant. We quickly unpacked, rested for a bit, and prepared for dinner.

Once we’d returned from a fabulous dinner, we again relaxed and settled in for the evening. Other than our Windows phones, we didn’t crack open any computers, so I don’t know how the complimentary Wi-Fi access was. We can both report, however, that the soaking tub was every bit as luxurious as it was claimed. The king bed (which towered off the ground) was one of the most comfortable beds I’ve ever slept in away from home.

Looking NNW over Penn Cove from the Captain’s Suite

In the morning, we woke up, brewed tea (for her) and coffee (for me, which is not my normal morning habit), got ready, and packed. As promised, the view of Penn Cove was beautiful and not marred at all by the brief but vigorous attack of hail and rain we enjoyed. Just before 9am, we placed our bags in our car and headed back into the main house for breakfast, where Marty greeted us by name and showed us to our place in the dining room with the rest of the guests. What a treat!

• Tea and coffee were on offer and Marty was quick to refill any cups that looked like they were thinking about becoming empty.
• The first course was a green mango fool. Now, I’m not a mango person…or at least, I didn’t think I was. I had a tiny bite of this and it was quite simply divine. I would have promptly devoured my whole serving, but I wasn’t quite awake yet and the morning’s cup of coffee kept me from being hungry yet. Stephanie also got to enjoy this, minus the cream.
• Our next course was buttermilk scones with currants. Again, these were very tasty, and again, my stomach wasn’t quite open for business yet. Sue made sure that Steph was supplied with gluten/dairy-free banana muffins, which Steph devoured.
• The final course was a three-cheese omelet and a serving of oven-roasted seasoned Yukon Gold potatoes; Steph got scrambled eggs. Now, Steph’s not a scrambled egg person, but you’d never have known that — just as you’d never have known that I never eat breakfast potatoes unless they’re hash browns. The eggs came in separate porcelain oval bowls that kept them hot and tasty.

We lingered over our breakfast until the other guests left. At that point, we chatted a few minutes more with Marty (and said goodbye to Sue when she stuck her head out of the kitchen). After purchasing a Blue Goose Inn mug for me, we promised we’d be back during pub season, then hit the road back to the ferry terminal and points east. We had to head home, unpack, relax, and get the family ready for the afternoon’s plans: a visit to the Seattle Art Museum.

Christopher’s at Whidbey

Since Friday evening dinner wasn’t provided by The Blue Goose, this was the other major logistical challenge I faced in my planning. Dining is now much more exciting than it was back when I was the pickiest eater in the family, and it can be a significant source of stress for Steph. This was supposed to be a relaxing night away and I didn’t want her to have to worry about anything. Was I up to the task? As I said before, one of the reasons I chose Whidbey Island is that there are several towns on the island. Even if I couldn’t find anything near our lodgings, I was confident I’d be able to find a nice place for an intimate evening meal that could offer Steph not just one dinner option, but a choice of meals. They would also need to have food I’d eat — I still don’t like too much food with my food, if you know what I mean.

Coupeville turned out to be perfect because it’s also home to Christopher’s on Whidbey, a small and unassuming restaurant that boasts exquisite food and wine at amazingly affordable prices. During my planning, I’d called them up, explained our requirements, and in a few moments had an 8pm dinner reservation set up. They assured me that not only would Stephanie’s needs be taken care of, but that she would have a number of items to choose from. They asked me all the right questions to give me confidence that they actually did understand how to properly cook her meals without overlooking anything or putting her in danger of cross-contamination.

It was just a short drive from the Blue Goose to Christopher’s; if the weather had been better and we had still had light, we’d have walked the few blocks. When we arrived, the interior of the restaurant was well-lit, warm, and comfortably elegant without being pretentious or snobby. They greeted us by name, reassured me that they had Steph’s dietary restrictions on file, and showed us to a quiet table in the corner.

Albrecht 2008 Pinot Blanc from Alsace, France

They had an interesting and eclectic wine selection, with offerings from a number of sources. Unlike many wine lists, they seemed to focus on offering affordable, enjoyable wines, mainly from local and regional wineries. Stephanie and I both favor white wines, and I noticed they offered a pinot blanc from Alsace. I’ve heard good things about Alsace wine but have never had it, so I enquired about it; apparently, this was a good thing, because this wine turned out to be a favorite of their wine expert. We ordered a bottle and found it to be delicate and satisfying both chilled and warm; it boasted a fantastic balance of dry vs. sweet with an unassuming and crisp fruity taste. A lot of whites taste like alcohol mixed with simple syrup; this one barely tasted like alcohol at all, and went well with both our dishes. While we waited for our entrees, Steph enjoyed a salad and I attacked a basket of bread with butter.

Stephanie chose the king salmon with raspberry barbeque sauce with greens and mixed vegetables. I went with something a little less adventuresome: linguine alfredo with chicken; in my defense, I don’t get cream-based sauces at home any more thanks to our Glorious New Dietary Regime. Our food was served rather quickly and was presented with a simple elegance that could easily have double the price tag in another establishment. I’ll let Stephanie speak for her meal if she chooses, but I will note that she told me at least once that she could eat it every day and be happy. My linguine was simply fantastic; the pasta was perfectly al dente, the sauce was light and creamy and in perfect proportion to enhance the pasta without smothering it, and the chicken was tender and full of flavor. It was easily the best pasta I’ve had in my life, and the entire meal rates up in my top three dining experiences. The service, of course, was quick, cheerful, and unassuming. We will happily come back and acquaint ourselves with the rest of the menu.

Picasso at the Seattle Art Museum

Upon arriving back at home around 12:30pm on Saturday, we unpacked, grabbed an informal lunch with the family, and planned out the rest of the day. For Christmas, the kids had purchased Stephanie a family membership in the Seattle Art Museum, in part so we could all head to the Picasso exhibit they have running through January 17. We had our tickets to get into the Picasso exhibit for 5pm, and with the Seahawks kicking off in Seattle at 1:30pm, we decided to wait for traffic to die down and head into town later in the afternoon. That gave us time to locate several alternatives for dinner after we’d been to the museum.

Once we got into Seattle and were parked at the garage underneath the SAM — a much trickier proposition now that we have a Ford Freestyle — we went up to Member Services and got our temporary membership cards. At that point, we had about 75 minutes to fill before we could enter the Picasso exhibit. We therefore broke up into groups and wandered around the museum’s various levels. Much of what I saw made little impression on me; a few of the pieces provoked a strong response (usually strong incredulity). I very much enjoyed the European and Italian galleries; in particular, they had a recreation of an Italian room, full of dark carved wood, that I found particularly intriguing.

Soon enough, 5pm approached and we queued up to enter the Picasso exhibit. I’m afraid I’m the wrong person to comment on it — I find most of Picasso’s work to be unapproachable. I tended to concentrate, instead, on the other people viewing the exhibit. There were a lot of very serious people there who apparently found all sorts of serious things to ponder. They were no fun. I liked watching the people who were totally blown away by what they were seeing; even if I didn’t share their reaction, I couldn’t help but be happy they were having a great time. These people invariably talked about how the art made them feel; the former types tended to pontificate on how it should make others feel and think. That’s an interesting lesson, don’t you think?

Once we had our fill of Picasso — or at least of walking around on the hard floors and dueling our way through the maddening crowds — we headed down to the waterfront to the Old Spaghetti Factory. I hadn’t dined here in many years — back when Stephanie and I were first married and I was working down on Pier 70. I’d really enjoyed it then and was looking forward to introducing my kids, especially because they offered gluten-free/dairy-free options. Instead, Stephanie and I found it to be one of the most disappointing dining experiences we’ve ever had. Maybe we were spoiled by still being on a high from the previous evening’s dining, but the restaurant felt crowded and dark, our table was noisy and drafty, and our server, while personable enough, couldn’t hit the right balance between competence and comedy. I can make better pasta than the half-hearted attempt I received. The best thing we can say is that Mom enjoyed it, as did the kids, although even the kids say that Steph would have made a better meal.

Wrapping Up

So, now it’s time for me to get off the computer and go spend the rest of the day with my family. I think we’ve got a board game or two on deck, maybe a family movie. Or, I could always pull out the copy of Enchanter’s Endgame that we’ve slowly been working through and read another chapter out loud. At any rate, we’ll have a good evening and get prepared to throw ourselves back into school, work, and life come Monday morning.

Right now, I’d like to introduce you to a friend of mine by the name of Chris. Chris and I became acquainted lo these many moons ago when I got involved in the community for the online PC game Starsiege: Tribes back at the end of the 90s. A week after we met, Chris was in a horrible motorcycle accident that changed his life forever. It’s a miracle he’s still alive. Stephanie and I have kept touch with him and over the years, have had the privilege of having him fly out from Vermont for three extended visits with our family, including two memorable Christmas holidays. He’s been placed in our lives for a reason, and we’ve drawn him into our family-of-choice.

Chris at the Gangers for Christmas 2007

Chris’s medical condition is deteriorating; his doctor now estimates that he has approximately five years at the outside until he will need to live in assisted care. We were able to help him out a couple of years ago by putting him up on the awesome Select Comfort air bed that Steph had scrounged up for our guest room. The difference it made during his four-week visit that year was amazing — by the end of the visit, he was regularly going without an entire pain medication dose and was still more active and healthy than he’d been since the accident. His doctor worked all year to get the State of Vermont health services to purchase a Select Comfort bed for him — wrote the prescription, jumped through hoops to show how the cost of the bed would easily repay itself in the reduced medication costs, etc. — and some bureaucratic organization killed the whole idea. Why? Good question — we still don’t know. After a year of struggling, we sent the bed home with him after the next Christmas visit. (Screw you, nameless Vermont functionaries!)

We’ve been working on getting him moved from Vermont to Washington — specifically out to be near us — but it’s been an uphill battle. It has been extremely frustrating hearing him tell us over and over how he gets a good phone interview for a perfect part-time job but then once they meet him in person, game over. Now Chris has a plan. It may not be the best plan, but it’s better than what we’ve been able to come up with and we’re going to help.

Chris working on my Lego Star Destroyer

Those of you who read my blog, whether directly, through some feed, through Twitter, or through Facebook: I’m hoping that you might be able and willing to give some help as well. Please go read his site and background — we’re going to scrounge up the pictures we have of him and send them so he can include them in updates and allow folks to get to know him. If you can, donate. If you can, spread this further. We’d love to get Chris relocated this spring and summer once the weather turns good and get him out here where we can provide in-person assistance. It won’t take much — $1, $2, $maybe even $5 and then pass the word on.

One of the biggest differences I’m seeing when deploying Exchange 2010 compared to previous versions is that for just about all of my customers, load balancing is becoming a critical part of the process. In Exchange 2003 FE/BE, load balancing was a luxury unheard of for all but the largest organizations with the deepest pockets. Only a handful of outfits offered load balancing products, and they were expensive. For Exchange 2007 and the dedicated CAS role, it started becoming more common.

For Exchange 2003 and 2007, you could get all the same benefits of load balancing (as far as Exchange was concerned) by deploying an ISA server or ISA server cluster using Windows Network Load Balancing (WNLB). ISA included the concept of a “web farm” so it would round-robin incoming HTTP connections to your available FE servers (and Exchange 2007 CAS servers). Generally, your internal clients would directly talk to their mailbox servers, so this worked well. Hardware load balancers were typically used as a replacement for publishing with an ISA reverse proxy (and more rarely to load balance the ISA array instead of WNLB). Load balancers could perform SSL offloading, pre-authentication, and many of the same tasks people were formerly using ISA for. Some small shops deployed WNLB for Exchange 2003 FEs and Exchange 2007 CAS roles.

In Exchange 2010, everything changes. Outlook RPC connections now go to the CAS servers in the site, not the MB server that hosts the active copy of the database. Mailbox databases now have an affiliation with either a specific CAS server or a site-specific RPC client access array, which you can see using the –RpcClientAccessServer parameter of the Get-MailboxDatabase cmdlet. If you have two or more servers, I recommend you set up the RPC client access array as part of the initial deployment and get some sort of load balancer in place.

Load Balancing Options

At Trace3, we’re an F5 reseller, and F5 is one of the few load balancer companies out there that has really made an effort to understand and optimize Exchange 2010 deployments. However, I’m not on the sales side; I have customers using a variety of load balancing solutions for their Exchange deployments. At the end of the day, we want the customer to do what’s right for them. For some customers, that’s an F5. Others require a different solution. In those cases, we have to get creative – sometimes they don’t have budget, sometimes the networking team has their own plans, and on some rare occasions, the plans we made going in turned out not to be a good fit after all and now we have to come up with something on the fly.

If you’re not in a position to use a high-end hardware load balancer like an F5 BIG-IP or a Cisco ACE solution, and can’t look at some of the lower-cost (and correspondingly lower-feature) solutions that are now on the market, there are few alternatives:

• WNLB. To be honest, I have attempted to use this in several environments now and even when I spent time going over the pros and cons, it failed to meet expectations. If you’re virtualizing Exchange (like many of my customers) and are trying to avoid single points of failure, WNLB is so clearly not the way to go. I no longer recommend this to my customers.
• DNS round robin. This method at least has the advantage of in theory driving traffic to all of the CAS instances. However, in practice it gets in the way of quickly resolving problems when they come up. It’s better than nothing, but not by much.
• DAG cluster IP. Some clever people came up with this option for instances where you are deploying multi-role servers with MB+HT+CAS on all servers and configuring them in a DAG. DAG = cluster, these smart people think, and clusters have a cluster IP address. Why can’t we just use that as the IP address of the RPC client access array? Sure enough, this works, but it’s not tested or supported by Microsoft and it isn’t a perfect solution. It’s not load balancing at all; the server holding the cluster IP address gets all the CAS traffic. Server sizing is important!

The fact of the matter is, there are no great alternatives if you’re not going to use hardware load balancing. You’re going to have to compromise something.

Introducing Devin’s Load Balancer

For many of my customers, we end up looking something like this:

• The CAS/HT roles are co-located on one set of servers, while MB (and the DAG) is on another. This rules out the DAG cluster IP option.
• They don’t want users to complain excessively when something goes wrong with one of the CAS/HT servers. This rules out DNS round robin.
• They don’t have the budget for a hardware solution yet, or one is already in the works but not ready because of schedule. They need a temporary, low-impact solution. This effectively rules out WNLB.

I’ve come up with a quick and dirty fix I call Devin’s Load Balancer or, as I commonly call it, the DLB. It looks like this:

1. Pick one CAS server that can handle all the traffic for the site. This is our target server.
2. Pick an IP address for the RPC client access array for the site. Create the DNS A record for the RPC client access array FQDN, pointing to the IP address.
3. Create the RPC client access array in EMS, setting the name, FQDN, and site.
4. On the main network interface of the target server, add the IP address. If this IP address is on the same subnet as the main IP address, there is no need to create a secondary interface! Just add it as a secondary IP address/subnet mask.
5. Make sure the appropriate mailbox databases are associated with the RPC client access array.
6. Optionally, point the internal HTTP load balance array DNS A record to this IP address as well (or publish this IP address using ISA).

You may have noticed that this sends all traffic to the target server; it doesn’t really load balance. DLB also stands for Doesn’t Load Balance!

This configuration, despite its flaws, gives me what I believe are several important benefits:

• It’s extremely easy to switchover/failover. If something happens to my target server, I simply add the RPC client access array IP address as a secondary IP address to my next CAS instance. There are no DNS cache entries to wait to expire. There are are no switch configurations to modify. There are no DNS records I have to update. If this is a planned switchover, client get disrupted but can immediately connect. I can make the update as soon as I get warning that something happened and my clients can reconnect without any further action on their part.
• It isolates what I do with the other CAS instances. Windows and Exchange no longer have any clue they’re in a load balanced pseudo-configuration. With WNLB, if I make any changes to the LB cluster (like add or remove a member), all connections to the cluster IP addresses are dropped!
• It makes it very easy to upgrade to a true load balancing solution. I set the true solution up in parallel with an alternate, temporary IP address. I use local HOSTS file entries on my test machines while I’m getting everything tested and validated. And then I simply take the RPC client access array IP address off the target server and put it on the load balancer. Existing connections are dropped, but new ones immediately connect with no timeouts – and now we’re really load balancing.

Note that you do not need the CAS SSL certificate to contain the FQDN of the RPC client access array as a SAN entry. RPC doesn’t use SSL for encryption (it’s not based on HTTP).

Even in a deployment where the customer is putting all roles into single-server configuration, if there’s any thought at all that they might want to expand to an HA configuration in the future, I now am in the habit of configuring this. The RPC client access array is now configured and somewhat isolated from the CAS configuration, so now my future upgrades are easier and less disruptive.

Back at the end 0f August, I had the opportunity, thanks to the @OReillyMedia twitter feed, to get my hands on a review copy of Cooking for Geeks (CfG) in e-book format. As part of the review agreement, I was supposed to:

• Select a recipe from the book,
• Prepare it,
• Photograph it,
• Write a review and post it,
• Post the photograph on the O’Reilly Facebook page,
• and all by September 6th.

Oops. Obviously, I’ve missed the precise timing here, but a bit belated, here’s the review I owe.

Why this cooking book?

There’s a lot of information on cooking out there. Stephanie has a metric ton of cookbooks and collected recipes in our house, and there are large chunks of old-growth forest bound up in the various cookbooks you can find in various stores. Thanks to the celebrity chef craze on TV, cooking (never an unpopular subject) has grown leaps and bounds beyond the good old Betty Crocker Cookbook that many of us grew up with[1]. Popular TV chefs now write and sell cookbooks on just about any specialty and niche you can imagine. I’ve even indulged in the recipe fetish myself once or twice, most noticeably to snag and perfect my favorite dish, the Cheesecake Factory’s Spicy Cashew Chicken dish.

What caught my attention (other than this being an O’Reilly book) about CfG was that my household has been slowly and steadily moving into the exciting world of food allergens. We recently flung ourselves off the cliffs of insanity this summer when blood tests revealed that Steph and Treanna tested positive for gluten antibodies. Add that to the existing dairy-free regime, and it was clear that menu planning at Chez Ganger had just started a new, exciting, but potentially very limited and boring chapter.

We’ve got a lot of friend who are gluten-free, dairy-free, vegetarian, vegan, some other regime, or even combinations of the above, so Steph’s no stranger to the issues involved. What is doable as an occasional thing, though, can become overwhelming when it’s a sudden lifestyle change that comes hard on the heels of a long, exhausting summer – just in time for the new school year. Understandably, Steph was struggling to cope – and we weren’t exactly the most helpful crew she could hope for.

After a few weeks of the same basic dishes being served over and over again, I was ready for any lifesaver that I could find. That’s when the fateful tweet caught my eye. After a few rounds of back and forth e-mail, I discovered that CfG included a chapter on cooking to accommodate allergens. The rest, as they say, is fate.

Torturing Chickens For Fun and Noms

Although I could go into great detail about the recipe my family ended up selecting – butterflied roasted chicken – my wife has already done so. Like a good writer, I will steal her efforts link to her blog post instead. She even took pictures! Go, read and salivate!

Back already?

Under the Cover

CfG is written by Jeff Potter, whose geek credentials appear to be genuine. The book has a fantastic companion site, which is essentially a link fest to the related blog and Twitter stream (as well as to the various places you can go on the Internet to purchase a copy of the book).

My lovely wife handled the “cooking” and “presentation” parts well, so I’m going to move on to our thoughts about the book itself:

• Content. If you want a book that explores the science and the art behind cooking, this is your book. It’s not a college textbook; it’s a great middle school or high school-level overview of the science of cooking that seems more interested in sharing Jeff’s love of cooking with you rather than creating cooking’s equivalent of the CCIE. Jeff writes with a very informal personable voice and isn’t afraid to show off his mastery of the physics behind good and bad dishes, sharing them in a way that’s part Bill Nye the Science Guy and part Ferris Bueller. I have never before laughed while reading a book on cooking. However, if you’re expecting a cookbook, check your expectations at the door. If this book has a weakness, it’s that talking about all this food will make you want a lot of recipes to try out, and I was surprised by how relatively few recipes there actually are. What is there provides an interesting cross-section across different types of dishes and ingredients, but it’s not a comprehensive reference guide. This is not “Cooking in a Nutshell” or cooking’s Camel Book; it is instead a not-to-scale map of the CfG theme park. If you find something that entrances you, you should be able to walk away with enough exposure to be able to knowledgeably pick out some other more detailed work for given area. CfG is the culinary equivalent of Jerome K. Jerome’s immortal Three Men in a Boat (To Say Nothing of the Dog); you’re going to get a fantastic lazy summer day punt trip down the river of Jeff’s epicurean experiences.
• Format. We used the PDF format (like all of O’Reilly’s e-books, unencumbered by DRM). Steph already made a comment about how useful she found the e-book format. With a sturdy tablet, I think an e-book cookbook would be great in the kitchen, especially if there were some great application that could handle browsing and organizing recipes from multiple sources. As I already said, though, this book is not a cookbook and I’d probably just make a quick copy of (or retype) the recipes I was interested in so that I didn’t have to use the physical book in the kitchen. Having said that, though, we’re going to purchase a physical copy of the book to facilitate quick browsing. If you’ve already made the switch to casual e-reading (we have not yet), you probably won’t have this same issue.
• Organization. Whether you like the book’s organization will depend on what you wanted out of it. If you wanted cooking’s Camel Book, you will find the book to be dismayingly unorganized. The structure of the book (and the recipes within) are based around the physics of cooking. Here, Jeff reveals himself to be a Lego Master of building blocks – you will find yourself introduced to one scientific concept after another, and each chapter will build on that knowledge by concentrating on a particular theme or technique rather than on a specific type of food or course. It really will help you to think of it as a novel (a romance, actually, between Jeff and food) and read the book from cover to cover rather than jump around in typical O’Reilly reference format. This is passion, not profession; calling, not career.
• Utility. I’m pretty much a dunce when it comes to cooking, so I found this book to be extremely useful. I hate following the typical magical thinking approach to cooking: put ingredient A into slot B and pull on tab C for 30 minutes until you screw it all up because you didn’t know that your masterpiece was afraid of loud noises. I want to know why I’m putting nasty old cream of tartar into my mixing bowl; what purpose does it serve? How can I usefully strike out into the scary wilderness of trying to adapt existing favorite recipes to a gluten-free, dairy-free existence? CfG doesn’t answer all my questions, but it answers a hell of a lot more of them than any other cooking book I’ve picked up. It didn’t talk down to me, but it didn’t assume I was already a lifelong member of the Secret and Worshipful Order of Basters, Bakers, and Broilers. What it didn’t do, though, is give me a large number of variations on a theme to go and try. At times the recipe selection – while ecletic and representative – felt somewhat sparse and even unrelated to what was being talked about in the main text. It seemed like someone on the team had written a badly behaved random recipe widget[2] to insert a recipe every so often. I would love, in the second edition, to see a little bit more connection between the theory and the practice, even though I recognize this isn’t a textbook.

We found our payoff in the chapter on cooking around allergens. Of all the chapters, this is the one that most felt like a reference work — a concise but thorough reference work. Jeff explains why (for example) taking gluten out of a recipe and merely substituting some non-gluten flour is probably not going to produce edible results, and then explains some of the common approaches for dealing with the problem. He’s trusting us, the readers, to be able and willing to do some experimentation and find our own way without having a GPS to lead us by the nose. While it’s initially tempting to have the comfort of specific substitution steps, in the end, CfG will help you know how to make substitutions on your own and quickly dial in to an acceptable solution rather than sit around waiting for someone to write the HOWTO.

In the end, Jeff’s approach is empowerment. We liked it a lot; thank you, Jeff and O’Reilly!

[1] Not only did I grow up with one and spend a lot of time browsing it, Steph has one. I’ll have you know, however, that I’ve only flipped through it once for auld lang syne.

[2] Probably written in Ruby or PHP.

Easy Installation and Upgrade Slipstreaming

One thing that I love about Exchange service packs is that from Exchange 2007 on, they’re full installations in their own right. Ready to deploy a brand new Exchange 2010 SP1 server? Just run setup from the SP1 binaries – no more fiddling around with the original binaries, then applying your service packs. Of course, the Update Rollups now take the place of that, but there’s a mechanism to slipstream them into the installer (and here is the Exchange 2007 version of this article).

Note: If you do make use of the slipstream capabilities, remember that Update Rollups are both version-dependent (tied to the particular RTM/SP release level) and are cumulative. SP1 UR4 is not the same thing as RTM UR4! However, RTM UR4 will include RTM UR3, RTM UR2, and RTM UR1…just as SP1 UR4 will contain SP1 UR3, SP1 UR2, and SP1 UR1.

The articles I linked to say not to slipstream the Update Rollups with a service pack, and I’ve heard some confusion about what this means. It’s simple: you can use the Updates folder mechanism to slipstream the Update Rollups when you are performing a clean install. You cannot use the slipstream mechanism when you are applying a service pack to an existing Exchange installation. In the latter situation, apply the service pack, then the latest Update Rollup.

It’s too early for any Update Rollups for Exchange 2010 SP1 to exist at the time of writing, but if there were (for the sake of illustration, let’s say that SP1 UR X just came out), consider these two scenarios:

• You have an existing Exchange 2010 RTM UR4 environment and want to upgrade directly to SP1 UR1. You would do this in two steps on each machine: run the SP1 installer, then run the latest SP1 UR X installer.
• You now want to add a new Exchange 2010 server into your environment and want it to be at the same patch level. You could perform the installation in a single step from the SP1 binaries by making sure the latest SP1 UR X installer was in the Updates folder.

If these scenarios seem overly complicated, just remember back to the Exchange 2003 days…and before.

Third Party Applications

This might surprise you, but in all of the current Exchange 2010 projects I’m working on, I’ve not even raised the question of upgrading to SP1 yet. Why would I not do that? Simple – all of these environments have dependencies on third-party software that is not yet certified for Exchange 2010 SP1. In some cases, the software has barely just been certified for Exchange 2010 RTM! If the customer brings it up, I always encourage them to start examining SP1 in the lab, but for most production environments, supportability is a key requirement.

Make sure you’re not going to break any applications you care about before you go applying service packs! Exchange service packs always make changes – some easy to see, some harder to spot. You may need to upgrade your third-party applications, or you may simply need to make configuration changes ahead of time – but if you blindly apply service packs, you’ll find these things out the hard way. If you have a critical issue or lack of functionality that the Exchange 2010 SP1 will address, get it tested in your lab and make sure things will work.

Key applications I encourage my customers to test include:

• Mobile device connectivity (like BES)
• Backup/restore/DR applications
• Archival packages
• Security/Anti-virus/mail hygiene packages (Microsoft isn’t immune from this; SP1 breaks their own Exchange 2010 Edge Transport/Forefront Threat Management Gateway integration and they don’t have a fix for it at this time)
• Non-standard desktop clients (Entourage and IMAP clients, I’m looking at you)
• Anything that uses EWS or MAPI to monitor mailboxes

Applications that use SMTP submission are typically pretty safe, and there are other applications that you might be okay living without if something does break. Figure out what you can live with, test them (or wait for certifications), and go from there.

Complications and Gotchas

Unfortunately, not every service pack goes smoothly. For Exchange 2010 SP1, one of the big gotchas that early adopters are giving strong feedback about is the number of hotfixes you must download and apply to Windows and the .NET Framework before applying SP1 (a variable number, depending on which base OS your Exchange 2010 server is running).

Having to install hotfixes wouldn’t be that bad if the installer told you, “Hey, click here and here and here to download and install the missing hotfixes.” Exchange has historically not done that (citing boundaries between Microsoft product groups) even though other Microsoft applications don’t seem to be quite as hobbled. However, this instance of (lack of) integration is particularly egregious because of two factors.

Factor #1: hotfix naming conventions. Back in the days of Windows 2000, you knew whether a hotfix was meant for your system, because whether you were running Workstation or Server, it was Windows 2000. Windows XP and Windows 2003 broke that naming link between desktop and server operating systems, often confusingly so once 64-bit versions of each were introduced (32-bit XP and 32-bit 2003 had their own patch versions, but 64-bit XP applied 64-bit 2003 hotfixes).

Then we got a few more twists to deal with. For example, did you know that Windows Vista and Windows Server 2008 are the same codebase under the hood? Or that Windows 7 and Windows Server 2008 R2, likewise, are BFFs? It’s true. Likewise, the logic behind the naming of Windows Server 2003 R2 and Windows Server 2008 R2 were very different; Windows Server 2003 R2 was basically Windows Server 2003 with a SP and few additional components, while Windows Server 2008 R2 has some substantially different code under the hood than Windows Server 2008 with SP. (I would guess that Windows Server 2008 R2 got the R2 moniker to capitalize on Windows 2008’s success, while Windows 7 got a new name to differentiate itself from the perceived train wreck that Vista had become, but that’s speculation on my part.)

At any rate, figuring out which hotfixes you need – and which versions of those hotfixes – is less than easy. Just remember that you’re always downloading the 64-bit patch, and that Windows 2008=Vista while Windows 2008 R2=Windows 7 and you should be fine.

Factor #2: hotfix release channels. None of these hotfixes show up under Windows Update. There’s no easy installer or tool to run that gets them for you. In fact, at least two of the hotfixes must be obtained directly from Microsoft Customer Support Services. All of these hotfixes include scary legal boilerplate about not being fully regression tested and thereby not supported unless you were directly told to install them by CSS. This has caused quite a bit of angst out in the Exchange community, enough so that various people are collecting the various hotfixes and making them available off their own websites in one easy package to download[1].

I know that these people mean well and are trying to save others from a frustrating experience, but in this case, the help offered is a bad idea. That same hotfix boilerplate means that everyone who downloads those hotfixes agree not to redistribute those hotfixes. There’s no exception for good intentions. If you think this is bogus, let me give you two things to think about:

• You need to be able to verify that your hotfixes are legitimate and haven’t been tampered with. Do you really want to trust production mission-critical systems to hotfixes you scrounged from some random Exchange pro you only know through blog postings? Even if the pro is trustworthy, is their web site? Quite frankly, I trust Microsoft’s web security team to prevent, detect, and mitigate hotfix-affecting intrusions far more quickly and efficiently than some random Exchange professional’s web host. I’m not disparaging any of my colleagues out there, but let’s face it – we have a lot more things to stay focused on. Few of us (if any) have the time and resources the Microsoft security guys do.
• Hotfixes in bundles grow stale. When you link to a KB article or Microsoft Download offering to get a hotfix, you’re getting the most recent version of that hotfix. Yes, hotfixes may be updated behind the scenes as issues are uncovered and testing results come in. In the case of the direct-from-CSS hotfixes, you can get them for free through a relatively simple process. As part of that process, Microsoft collects your contact info so they can alert you if any issues later come up with the hotfix that may affect you. Downloading a stale hotfix from a random bundle increases the chances of getting an old hotfix version that may cause issues in your environment, costing you a support incident. How many of these people are going to update their bundles as new hotfix versions become available? How quickly will they do it – and how will you know?

The Exchange product team has gotten an overwhelming amount of feedback on this issue, and they’ve responded on their blog. Not only do they give you a handy table rounding up links to get the hotfixes, they also collect a number of other potential gotchas and advice to learn from from before beginning your SP1 deployment. Go check it out, then start deploying SP1 in your lab.

Good luck, and have fun! SP1 includes some killer new functionality, so take a look and enjoy!

[1] If you’re about to deploy a number of servers in a short period of time, of course you should cache these downloaded hotfixes for your team’s own use. Just make sure that that you check back occasionally for updated versions of the hotfixes. The rule of thumb I’d use is about a week – if I’m hitting my own hotfix cache and it’s older than a week, it’s worth a couple of minutes to make sure it’s still current.

In order for this to be true, you have to have the following conditions:

1. You have no other Exchange 2010 servers in the AD site. This implies that at least one of your DAG nodes is multi-role — remember that you need to have a CAS role and an HT role in the same site as your MB roles, preferably two or more of each for redundancy and load. If you have another Exchange 2010 server, then it’s already got the correct permissions — let Exchange manage the FSW automatically.
2. If the site in question is part of a DAG that stretches sites, there are more DAG nodes in this site than in the second site. If you’re trying to place the FSW in the site with fewer members, you’re asking for trouble[1].
3. You have no other Windows 2003 or 2008 servers in the site that you consider suitable for Exchange’s automatic FSW provisioning[2]. By this, I mean you’re not willing to the the Exchange Trusted Subsystem security group to the server’s local Administrators group so that Exchange can create, manage, and repair the FSW on its own. If your only other server in the site is a DC, I can understand not wanting to add the group to the Domain Admins group.

If that’s the case, and you’re dead set on doing it this way, you will have to manually create the FSW yourself. A FSW consists of two pieces: the directory, and the file share. The process for doing this is not documented anywhere on TechNet that I could find with a quick search, but happily, one Rune Bakkens blogs the following process:

To pre-create the FSW share you need the following:
- Create a folder etc. D:\FilesWitness\DAGNAME
- Give the owner permission to Exchange Trusted Subsystem
- Give the Exchange Trusted Subsystem Full Control (NTFS)
- Share the folder with the following DAGNAME.FQDN (If you try a different share name,
it won’t work. This is somehow required)
- Give the DAGNAME$ computeraccount Full Control (Share)

When you’ve done this, you can run the set-databaseavailabilitygroup -witnessserver CLUSTERSERVER – witnessdirectory D:\Filewitness\DAGNAME

You’ll get the following warning message:

WARNING: Specified witness server Cluster.fqdn is not an Exchange server, or part of the Exchange Servers security group.
WARNING: Insufficient permission to access file shares on witness server Cluster.fqdn. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the set-databaseavailabilitygroup cmdlet to try the operation again. Error: Access is denied

This is expected, since the cmdlet tries to create the folder and share, but don’t have the permissions to do this.

When this is done, the FSW should be configured correct. To verify this, the following files should be created:

- VerifyShareWriteAccess
- Witness

Just for the record, I have not tested this process yet. However, I’ve had to do some recent FSW troubleshooting lately and this matches with what I’ve seen for naming conventions and permissions, so I’m fairly confident this should get you most of the way there. Thank you, Rune!

Don’t worry, I haven’t forgotten the next installment of my Exchange 2010 storage series. It’s coming, honest!

[1] Consider the following two-site DAG scenarios:

• If there’s an odd number of MB nodes, Exchange won’t use the FSW.
• An even number (n) of nodes in each site. The FSW is necessary for there to even be a quorum (you have 2n+1 nodes so a simple majority is n+1). If you lose the FSW and one other node — no matter where that node is — you’ll lose quorum. If you lose the link between sites, you lose quorum no matter where the FSW is.
• A number (n) nodes in site A, with at least one fewer nodes (m) in site B. If n+m is odd, you have an odd number of nodes — our first case. Even if m is only 1 fewer than n, putting the FSW in site B is meaningless — if you lose site A, B will never have quorum (in this case, m+1 = n, and n is only half — one less than quorum).

I am confident in this case that if I’ve stuffed up the math here, someone will come along to correct me. I’m pretty sure I’m right, though, and now I’ll have to write up another post to show why. Yay for you!

[2] You do have at least one other Windows server in that site, though, right — like your DC? Exchange doesn’t like not having a DC in the local site — and that DC should also be a GC.

Patriotism is holding my elected officials, their political appointees, and the news media accountable for the choices and actions they take in my name. As a patriot, I have a responsibility to ensure that my representatives are conducting the business of government according to the values and principles they represented during election time. I need accurate and timely information on their performance and actions. I need to understand the difference between news and entertainment and know when each is appropriate.

Patriotism is acknowledging my country’s flaws with integrity and honesty instead of trying to cover them up or excuse them. When my government and policies fail – and being human institutions, they will fail – I will be tempted to downplay or minimize the impact of these failures. Instead, I must face these failures and their consequences forthrightly, make every reasonable effort to keep them from occurring again, and encourage my fellow Americans to do the same.

Patriotism is respecting the offices and institutions of my government even when expressing my disagreement with its policies and actions. Whether I am Democrat, Independent, Libertarian, Republican, some other party, or a member of none, I choose to discuss government and politics with civility and grace. I do not have to vilify political opponents in order to successfully engage their ideas and point out the failures of their actions. I can condemn bad choices and actions without hatred or unnecessary anger towards those who make them.

Patriotism is placing untainted personal ethics and morality ahead of my politics. I will not spread racism, classism, sexism, or other institutionalized forms of hatred. I have a responsibility to ensure that the voice of every American can be heard and that America provides as level of a playing field as possible. I have a personal stake in making America an ideal of compassionate, reasoned behavior to Americans and to the people of the world. I understand that my country will not be truly great if her citizens are not also great.

Patriotism is patient and compassionate. It is not jealous or blind. It does not covet or boast. Patriotism builds up and exhorts. It does not destroy or belittle. It does not promote lies or avoid the truth. Patriotism does not demand perfection, but asks you to always give your best.

May we all strive to be better patriots.

I’m no guru or saint, so I can’t answer the question for you, but for me it turns out the answer comes from a combination of two life experiences: my six and a half years at 3Sharp, and the nearly two years I’ve been studying karate. At 3Sharp, I learned how to do a lot of things that were beyond my initial comfort zone, developing deep technical presentations (and delivering them to large audiences), scoping and producing large technical projects such as books and whitepapers, and doing a large variety of work from hands-on consulting to research projects.

I’ve talked in previous posts about the physical benefits I’ve seen from karate. However, two weeks ago I tested for my 5th kyu belt (the second of my three green belts) and that experience made me aware of some deep changes in my personality and character. The step from 6th kyu to 5th kyu was particularly hard for me, and it took some time to sort out the two reasons why.

The obvious cause was schedule. I took two months off of karate at the beginning of the year, due to a combination of factors. That’s a hard gap to come back from; I had problems after the three week hiatus I took because of the MCM class. After two months, I just didn’t feel that my presence in class was doing any good until I had the privilege of watching two of my friends from the Mukilteo dojo earn their black belts one Saturday morning in February. I walked away from that experience feeling a new level of commitment to karate. After all, I told myself at that point, if I study hard, I’ll get to 5th kyu sooner or later, and that’s half-way to black belt!

The other cause was technical. The test kata for 5th kyu (Pinan Shodan) is the karateka’s first introduction[1] to a well-known and complicated set of katas, and while most of it seems to be straightforward, there’s a lot boiling up from underneath the surface. Carlos Sensei began introducing us to a series of drills based around Pinan Shodan that unpack a lot of useful theory and practice from the first eight moves of the kata. There’s this very difficult pivot/kick/double punch move right in there (I dub it UberHardMove) that is a key element of the kata, and I was having a hard time getting the pivot, kick, and punches all coordinated together and working the right way without falling on my ass. In fact, I had such a difficult time with it that I can remember sometime around the end of December thinking that maybe I’d found the wall beyond which my lack of coordination was not going to let me pass. In addition, there’s some pretty gnarly tuite that goes along with all of this and I found that I felt horribly weak on my tuite all around, let alone with the techniques I was supposed to able to demonstrate some proficiency at.

What ended up happening, though, was that the two-month time-out did me unexpected good. I didn’t go to class during that period, but I kept practicing karate around the house. (Just ask Steph and the kids; they’ll tell you that it can be difficult to get me to knock it off and stop interfering with whatever they’re trying to do.) And what I did during that time was to take UberHardMove and break it down into components, the way I had previously been shown as a blue belt[2]. I combined that with specific suggestions given to me by both Carlos Sensei and Liam Sensei and picked UberHardMove down to bare bones.

When I finally came back to class, I came back finally believing that the whole concept of me one day earning my black belt wasn’t the world’s best joke. I came back believing that I’d already invested nearly two years and I was willing to invest even more. I didn’t have to be perfect; I gave myself permission to suck. I knew that I was going to make stupid mistakes that I wouldn’t make (like mixing up techniques in lower level katas) if I’d been in class the whole time. I knew that my endurance was going to be awful. I knew that there was a lot of rust to scrub off and deal with and that it wasn’t going to happen immediately. I knew that I needed to let my instructors know that I desperately needed help with my tuite techniques. I knew that I was going to have to have them explain the same things about UberHardMove multiple times until I finally grokked it. In short, I accepted failure without accepting being a failure.

That was March. I tested near the end of May. Somewhere in there, I became proficient with my tuite. I learned a measure of peace with UberHardMove; I’m still not great at it, but I mastered it enough to move on to the next lessons[3]. Perfection is in fact is a bad word in our household. We think the concept of perfection is one of the worst lies that the Adversary ever got humans to accept.

When you stop trying to be perfect – when you give yourself permission to have flaws and failings and determine to be honest about them and learn from them rather than try to cover them up – something amazing tends to happen. You accept “doing your best” instead of “doing it better than everyone else.” You accept “that’s enough for now” instead of “that’s not good enough yet.” You develop a sense of faith that over time, your progress will trend upwards. With that faith, you can draw valuable lessons from your mistakes and missteps. You stop fighting the basic physical and neurological limits of how your body and mind acquire new proficiencies and start working within your limits to expand them instead of struggling against them to tear them down with brute force. You acquire patience – new and fledgling, but the seed of something that starts to affect how you deal with all of your life.

I’m no paragon of patience, but I can see clear changes. For example, I’ve been spending far less time playing Call of Duty on the Xbox in the last month or so. I have a better understanding of how that experience has been frustrating instead of fun and relaxing and I’m less willing to give in to that anymore.

I don’t know where this will go ultimately or at what pace. I can honestly say, though, that I’m okay with that. Will I get my black belt? I don’t know; there are many circumstances that could prevent or delay that. However, I certainly want to, and I finally know I’m capable of doing it, so I wouldn’t bet against me. But I also know that’s just another waypoint on the journey. It’s not an end. It’s a marker where I can say, “See what I’ve done so far? That’s pretty cool. Now I’ve learned enough that I can get serious about learning this stuff and helping pass it on to others.”

Two months ago, I’d have said I couldn’t wait for that day. You know what? That’s not true. I can wait. I will wait. And I will do so profitably.

[1] In our style, at least. There are other styles that place another Pinan kata before Pinan Shodan.

[2] In a nice twist of synchronicity, the person who showed me was at the time was a helpful brown belt from Mukilteo who ended up being one of the two black belts I got to watch test. He has continued to be an amazing source of inspiration for me through what is now a large number of discouraging situations. Hi, Max!

[3] It’s not going away; I still practice it, and I know that it will get better as I learn more. In fact, those final four moves in Pinan Nidan where I’m in a cat stance might be helpful here, hmmm…

In part 1, I began the process of talking about how I think the new Exchange 2010 storage options will play out in live Exchange deployments over the next several years. The first essay in this series discussed what is I believe the fundamental question at the heart an Exchange 2010 storage design: at what level will you ensure the redundancy of your Exchange mailbox databases? The traditional approach has used RAID at the disk level, but Exchange 2010 DAGs allow you to deploy mailbox databases in JBOD configurations. While I firmly believe that’s the central question, answering it requires us to dig under the hood of storage.

With Exchange 2010, Microsoft specifically designed Exchange mailbox servers to be capable of using the lowest common denominator of server storage: a directly attached storage (DAS) array of 7200 RPM SATA disks in a Just a Box of Disks (JBOD) configuration (what I call DJS). Understanding why they’ve made this shift requires us to understand more about the disk drive technology. In this essay, part 2 of this series, let’s talk about disk technology and find out how Fibre Channel (FC), Serially Attached SCSI (SAS), and Serial Advanced Technology Attachment (SATA) disk drives are the same – and more importantly, what slight differences they have and what that means for your Exchange systems.

So here’s the first dirty little secret: for the most part, all disks are the same. Regardless of what type of bus they use, what form factor they are, what capacity they are, and what speed they rotate at, all modern disks use the same construction and principles:

• They all have one or more thin rotating platters coated with magnetic media; the exact number varies by form factor and capacity. Platters look like mini CD-ROM disks, but unlike CDs, platters are typically double-sided. Platters have a rotational speed measured in revolutions per minute (RPMs).
• Each side of a platter has an associated read-write head. These heads are on a single-track arm that moves in toward the hub of the platter or out towards the rim. The heads do not touch the platter, but float very close to the surface. It takes a measurable fraction of a second for the head to relocate from one position to another; this is called its seek time.
• The circle described by the head’s position on the platter is called a track. In a multi-platter disk, the heads move in synchronization (there’s no independent tracking per platter or side). As a result, each head is on the same track at the same time, describing a cylinder.
• Each drive unit has embedded electronics that implement the bus protocol, control the rotational speed of the platters, and translate I/O requests into the appropriate commands to the heads. Even though there are different flavors, they all perform the same basic functions.

If you would like a more in-depth primer on how disks work, I recommend starting with this article. I’ll wait for you.

Good? Great! So that’s how all drives are the same. It’s time to dig into the differences. They’re relatively small, but small differences have a way of piling up. Take a look at Table 1 which summarizes the differences between various FC, SATA, and SAS disks, compared with legacy PATA 133 (commonly but mistakenly referred to as IDE) and SCSI Ultra 320 disks:

Table 1: Disk parameter differences by disk bus type

As of this writing, the most common drive types you’ll see for servers are SATA-II, SAS 300, and FC over copper. Note that while SCSI Ultra 320 drives in theory have a maximum data transfer higher than either SATA-II or SAS 300, in reality that bandwidth is shared among all the devices connected to the SCSI bus; both SATA and SAS have a one-to-one connection between disk and controller, removing contention. Also remember that SATA is only a half-duplex protocol, while SAS is a full-duplex protocol. SAS and FC disks use the full SCSI command set to allow better performance when multiple I/O requests are queued for the drive, whereas SATA uses the ATA command set. Both SAS and SATA implement tagged queuing, although they use two different standards (each of which has its pros and cons).

The second big difference is the average access time of the drive, which is the sum of multiple factors:

• The average seek time of the heads. The actuator motors that move the heads from track to track are largely the same from drive to drive and thus the time contributed to the drive’s average seek time by just the head movements is roughly the same from drive to drive. What varies is the length of the head move; is it moving to a neighboring track, or is it moving across the entire surface? We can average out small track changes with large track changes to come up with idealized numbers.
• The average latency of the platter. How fast the platters are spinning determines how quickly a given sector containing the data to be read (or where new data will be written) will move into position under the head once it’s in the proper track. This is a simple calculation based on the RPM of the platter and the observed average drive latency. We can assume that a given sector will move into position, on average, in no more than half a rotation. This gives us 30 seconds out of each minute of rotation, or 30,000 ms, into which we can divide the drive’s actual rotation.
• The overhead caused by the various electronics and queuing mechanisms of the drive electronics, including any power saving measures such as reducing the spin rate of the drive platters. Although electricity is pretty fast and on-board electronics are relatively small circuits, there may be other factors (depending on the drive type) that may introduce delays into the process of fulfilling the I/O request received from the host server.

What has the biggest impact is how fast the platter is spinning, as shown in Table 2:

Table 2: Average latency caused by rotation speed

(As an exercise, do the same math on the disk speeds for the average laptop drives. This helps explain why laptop drives are so much slower than even low-end 7,200 RPM SATA desktop drives.)

Rather than painfully take you through the result of all of these tables and calculations step by step, I’m simply going to refer you to work that’s already been done. Once we know the various averages and performance metrics, we can figure out how many I/O operations per second (IOPS) a given drive can sustain on average, according to the type, RPMs, and nature of the I/O (sequential or random). Since Microsoft has already done that work for us as part of the Exchange 2010 Mailbox Role Calculator (version 6.3 as of this writing, I’m going to simply use the values there. Let’s take a look at how all this plays out in Table 3 by selecting some representative values.

Table 3: Drive IOPS by type and RPM

There are three things to note about Table 3.

1. These numbers come from Microsoft’s Exchange 2010 Mailbox Sizing Calculator and are validated across vendors through extensive testing in an Exchange environment. While there may be minor variances between drive model and manufacturers and these number may seem pessimistic according to calculated IOPS number published for individual drives, these are good figures to use in the real world. Using calculated IOPS numbers can lead both to a range of figures, depending on the specific drive model and manufacturer, as well as to overestimating the amount of IOPS the drive will actually provide to Exchange.
2. For the most part, SAS and FC are indistinguishable from the IOPs point of view. Regardless of the difference between the electrical interfaces, the drive mechanisms and I/O behaviors are comparable.
3. Sequential IOPS are not listed; they will be quite a bit higher than the random IOPS (that same 7,200RPM SATA drive can provide 300+ IOPS for sequential operations). The reason is simple; although a lot of Exchange 2010 I/O has been converted from random to sequential, there’s still some random I/O going on. That’s going to be the limiting factor.

The IOPS listed are per-drive IOPS. When you’re measuring your drive system, remember that the various RAID configurations have their own IOPS overhead factor that will consume a certain number

There are of course some other factors that we need to consider, such as form factor and storage capacity. We can address these according to some generalizations:

• Since SAS and FC tend to have the same performance characteristics, the storage enclosure tends to differentiate between which technology is used. SAS enclosures can often be used for SATA drives as well, giving more flexibility to the operator. SAN vendors are increasingly offering SAS/SATA disk shelves for their systems because paying the FC toll can be a deal-breaker for new storage systems.
• SATA disks tend to have a larger storage capacity than SAS or FC disks. There are reasons for this, but the easiest one to understand is that SAS, being traditionally a consumer technology, has a lower duty cycle and therefore lower quality control specifications that must be met.
• SATA disks tend to be offered with lower RPMs than SAS and FC disks. Again, we can acknowledge that quality control plays a part here – the faster a platter spins, the more stringently the drive components need to meet their specifications for a longer period of time.
• 2.5” drives tend to have lower capacity than their 3.5” counterparts. This makes sense – they have smaller platters (and may have fewer platters in the drive).
• 2.5” drives tend to use less power and generate less heat than equivalent 3.5” drives. This too makes sense – the smaller platters have less mass, requiring less energy to sustain rotation.
• 2.5” drives tend to permit a higher drive density in a given storage chassis while using only fractionally more power. Again, this makes sense based on the previous two points; I can physically fit more drives into a given space, sometimes dramatically so.

Let’s look at an example. A Supermicro SC826 chassis holds 12 3.5” drives with a minimum of 800W power while the equivalent Supermicro SC216 chassis holds 24 2.5” drives with a minimum of 900W of power in the same 2Us of rack space. Doubling the number of drives makes up for the capacity difference between the 2.5” and 3.5” drives, provides twice as many spindles and allows a greater aggregate IOPS for the array, and only requires 12.5% more power.

The careful reader has noted that I’ve had very little to say about capacity in this essay, other than the observation above that SATA drives tend to have larger capacities, and that 3.5” drives tend to be larger than 2.5” drives. From what I’ve seen in the field, the majority of shops are just now looking at 2.5” drive shelves, so it’s safe to assume 3.5” is the norm. As a result, the 3.5” 7,200 RPM SATA drive represents the lowest common denominator for server storage, and that’s why the Exchange product team chose that drive as the performance bar for DJS configurations.

Exchange has been limited by performance (IOPS) requirements for most of its lifetime; by going after DJS, the product team has been able to take advantage of the fact that the capacity of these drives is the first to grow. This is why I think that Microsoft is betting that you’re going to want to simplify your deployment, aim for big, cheap, slow disks, and let Exchange DAGs do the work of replicating your data.

Now that we’ve talked about RAID vs. JBOD and SATA vs. SAS/FC, we’ll need to examine the final topic: SAN vs. DAS. Look for that discussion in Part 3, which will be forthcoming.

1 The LORD is my sensei; I shall not fear.

2 He makes me work out with white belts; he leads me through katas.

3 He perfects my form. He leads me in the path of new techniques for the sake of advancement.

4 Though I walk through the valley of the shadow of death, I fear no evil, for his teachings are with me; my kama and bo staff comfort me.

5 He prepares testing for me in the presence of my fellow karateka; he adorns my waist with new obi, my gi fits better.

6 Surely discipline and health shall follow me all the days of my studies, and I will dwell in the dojo of the LORD forever.

To kick it off, I’ve updated my From Whence Redundancy? Exchange 2010 Storage Essays, Part 1 post with some new material on database reseed times and reposted it there in its entirety. Don’t worry, I’ve also updated it here.

Table 1: Maximum databases per Exchange editions

However, the Exchange Server edition is not directly tied to the Windows Server edition:

• For Exchange 2003 failover cluster mailbox servers, Exchange 2007 SCC/CCR environments [1], and  Exchange 2010 DAG environments, you need Windows Server Enterprise Edition in order to get the MSCS cluster component framework.
• For Exchange 2003 servers running purely as bridgeheads or front-end servers, or Exchange 2007/2010 HT, CAS, ET, and UM servers, you only need Windows Server Standard Edition.

I’ve seen some discussion around the fact that Exchange 2010 will install on Windows Server 2008 Datacenter Edition and Windows Server 2008 R2 Datacenter Edition, even though it’s not supported there and is not listed in the Operating System requirements section of the TechNet documentation.

HOWEVER…if we look at the Prerequisites for Exchange 2010 Server section of the Exchange Server 2010 Licensing site, we now see that Datacenter edition is, in fact listed as shown in Figure 1:

Figure 1: Exchange 2010 server license comparison

This is pretty cool, and the appropriate TechNet documentation is in the process of being updated to reflect this. What this means is that you can deploy Exchange 2010 on Windows Server Datacenter Edition; the differences between editions of Windows Server 2008 R2 are found here.[2] If you take a quick scan through the various feature comparison charts in the sidebar, you might wonder why anyone would want to install Exchange 2010 on Windows Server Datacenter Edition; it’s more costly and seems to provide the same benefits. However, take a look at the technical specifications comparison; this is, I believe, the meat of the matter:

• Both editions give you a maximum of 2 TB – more than you can realistically throw at Exchange 2010.
• Enterprise Edition gives you support for a maximum eight (8) x64 CPU sockets, while Datacenter Edition gives you sixty-four (64). With quad-core CPUs, this means a total of 32 cores under Enterprise vs. 256 cores under Datacenter.
• With the appropriate hardware, you can hot-add memory in Enterprise Edition. However, you can’t perform a hot-replace, nor can you hot-add or hot-replace CPUs under Enterprise. With Datacenter, you can hot-add and hot-remove both memory and CPUs.

These seem to be compelling in many scenarios at first glance, unless you’re familiar with the recommended maximum configurations for Exchange 2010 server sizing. IIRC, the maximum CPUs that are recommended for most Exchange 2010 server configurations (including multirole servers) would be 24 cores – which fits into the 8 socket limitation of Enterprise Edition while using quad core CPUs.

With both Intel and AMD now offering hexa-core (6 core) CPUs, you can move up to 48 cores in Enterprise Edition. This is more than enough for any practical deployment of Exchange Server 2010 I can think of at this time, unless future service packs drastically change the CPU performance factors. Both Enterprise and Datacenter give you a ceiling of 2TB of RAM, which is far greater than required by even the most aggressively gigantic mailbox load I’d want to place on a single server. I’m having a difficult time seeing how anyone could realistically build out an Exchange 2010 server that goes beyond the performance and scalability limits of Enterprise Edition in any meaningful way.

In fact, I can think of only three reasons someone would want to run Exchange 2010 on Windows Server Datacenter Edition:

• You have spare Datacenter Edition licenses, aren’t going to use them, and don’t want to buy more Enterprise Edition licenses. This must be a tough place to be in, but it can happen under certain scenarios.
• You have a very high server availability requirements and require the hot-add/hot-replace capabilities. This will get costly – the server hardware that supports this isn’t cheap – but if you need it, you need it.
• You’re already running a big beefy box with Datacenter and virtualization[3]. The box has spare capacity, so you want to make use of it.

The first two make sense. The last one, though, I’d be somewhat leery of doing. Seriously, think about this – I’m spending money on monstrous hardware with awesome fault tolerance capabilities, I’ve forked over for an OS license[4] that gives me the right to unlimited virtual machines, and now I’m going to clutter up my disaster recovery operations by mixing Exchange and other applications (including virtualization) in the same host OS instance? That may be great for a lab environment, but I’d have a long conversation with any customer who wanted to do this under production. Seriously, just spin up a new VM, use Windows Server Enterprise Edition, and go to town. The loss of hardware configuration flexibility I get from going virtual is less than I gain by compartmentalizing my Exchange server to its own machine, along with the ability to move that virtual machine to any virtualization host I have.

So, there you have it: Exchange 2010 can now be run on Windows Server Datacenter Edition, which means yay! for options. But in the end, I don’t expect this to make a difference for any of the deployments I’m like to be working on. This is a great move for a small handful of customers who really need this.

[1] MSCS is not required for Exchange 2007 SCR, although manual target activation can be easier in some scenarios if your target is configured as a single passive node cluster.

[2] From what I can tell, the same specs seem to be valid for Windows Server 2008, with the caveat that Windows Server 2008 R2 doesn’t offer a 32-bit version so the chart doesn’t give that information. However, since Exchange 2010 is x64 only, this is a moot point.

[3] This is often an attractive option, since you can hosted an unlimited number of Windows Server virtual machines without having to buy further Windows Server licenses for them.

[4] Remember that Datacenter is not licensed at a flat cost per server like Enterprise is; it’s licensed per socket. The beefier the machine you run it on, the more you pay.

Pat Robertson’s comments on Haiti basically boil down to “they got what was coming to them.” Mr. Robertson, I think you forgot Matthew 25:34-46 (KJV):

³⁴Then shall the King say unto them on his right hand, Come, ye blessed of my Father, inherit the kingdom prepared for you from the foundation of the world: ³⁵For I was an hungred, and ye gave me meat: I was thirsty, and ye gave me drink: I was a stranger, and ye took me in: ³⁶Naked, and ye clothed me: I was sick, and ye visited me: I was in prison, and ye came unto me. ³⁷Then shall the righteous answer him, saying, Lord, when saw we thee an hungred, and fed thee? or thirsty, and gave thee drink? ³⁸When saw we thee a stranger, and took thee in? or naked, and clothed thee? ³⁹Or when saw we thee sick, or in prison, and came unto thee? ⁴⁰And the King shall answer and say unto them, Verily I say unto you, Inasmuch as ye have done it unto one of the least of these my brethren, ye have done it unto me.

⁴¹Then shall he say also unto them on the left hand, Depart from me, ye cursed, into everlasting fire, prepared for the devil and his angels: ⁴²For I was an hungred, and ye gave me no meat: I was thirsty, and ye gave me no drink: ⁴³I was a stranger, and ye took me not in: naked, and ye clothed me not: sick, and in prison, and ye visited me not. ⁴⁴Then shall they also answer him, saying, Lord, when saw we thee an hungred, or athirst, or a stranger, or naked, or sick, or in prison, and did not minister unto thee? ⁴⁵Then shall he answer them, saying, Verily I say unto you, Inasmuch as ye did it not to one of the least of these, ye did it not to me. ⁴⁶And these shall go away into everlasting punishment: but the righteous into life eternal.

Rush Limbaugh may have forgotten the above as well. His claims that Obama is using humanitarian aid for political profit definitely seem to have forgotten Matthew 7:15-20:

15 Beware of false prophets, which come to you in sheep’s clothing, but inwardly they are ravening wolves. 16 Ye shall know them by their fruits. Do men gather grapes of thorns, or figs of thistles? 17 Even so every good tree bringeth forth good fruit; but a corrupt tree bringeth forth evil fruit. 18 A good tree cannot bring forth evil fruit, neither can a corrupt tree bring forth good fruit. 19 Every tree that bringeth not forth good fruit is hewn down, and cast into the fire. 20 Wherefore by their fruits ye shall know them.

If that last passage seems a bit murky, here’s a quote from C. S. Lewis’s The Last Battle (the last book of the Chronicles of Narnia) that I have always loved. The speaker is a Calormene soldier, Emeth, who has had a life-changing encounter with Aslan during the last hours of Narnia:

He answered, Child, all the service thou hast done to Tash, I account as service done to me. Then by reasons of my great desire for wisdom and understanding, I overcame my fear and questioned the Glorious One and said, Lord, is it then true, as the Ape said, that thou and Tash are one? The Lion growled so that the earth shook (but his wrath was not against me) and said, It is false. Not because he and I are one, but because we are opposites, I take to me the services which thou hast done to him. For I and he are of such different kinds that no service which is vile can be done to me, and none which is not vile can be done to him. Therefore if any man swear by Tash and keep his oath for the oath’s sake, it is by me that he had truly sworn, though he know it not, and it is I who reward him. And if any man do a cruelty in my name, then, though he says the name Aslan, it is Tash whom he serves and by Tash his deed is accepted. Dost thou understand, Child?

By their fruits ye shall know them…whatever their claims.

That’s exactly where Google wants you to focus.

Let’s take a closer look at their blog post.

Paragraph 1:

In mid-December, we detected a highly sophisticated and targeted attack on our corporate infrastructure originating from China that resulted in the theft of intellectual property from Google.

Paragraph 2:

As part of our investigation we have discovered that at least twenty other large companies from a wide range of businesses–including the Internet, finance, technology, media and chemical sectors–have been similarly targeted.

Whoa. That’s some heavy-league stuff right there. Coordinated, targeted commercial espionage across a variety of vertical industries. Google first accuses China of stealing its intellectual property, then says that they weren’t the only ones. Mind you, industry experts – including the United States government– have been saying the same thing for years. Cries of ‘China hacked us!” happen relatively frequently in the IT security industry, enough so that it blends into the background noise after awhile.

My question is why, exactly, Google thought this wouldn’t happen to them? They’re a big fat juicy target on many levels. Gmail with thousands upon thousands of juicy mailboxes? Check! Search engine code and data that allows sophisticated monitoring and manipulation of Internet queries? Check! Cloud-based office documents that just might contain some competitive value? Check!

My second question is, why, exactly, is Google trying to shift the focus of the story from the IP theft (which by their own press report was successful) and cloak their actions in the “oh, noes, China tried to grab dissidents’ email” moral veil they’re using?

Paragraph 3:

Second, we have evidence to suggest that a primary goal of the attackers was accessing the Gmail accounts of Chinese human rights activists. Based on our investigation to date we believe their attack did not achieve that objective. Only two Gmail accounts appear to have been accessed, and that activity was limited to account information (such as the date the account was created) and subject line, rather than the content of emails themselves.

Two accounts, people, and the attempt wasn’t even fully successful. And the moral outrage shimmering from the screen in Paragraph 4, when Google says that “dozens” of accounts were accessed by third parties not through any sort of security flaw in Google, but rather through what is probably malware, is enough to knock you over.

Really, Google? You’re just now tumbling to the fact that people’s GMail accounts are getting hacked through malware?

I don’t buy the moral outrage. I think the meat of the matter is back in paragraph 1. I believe that the rest of the outrage is a smokescreen to repaint Google into the moral high ground for their actions, when from the sidelines here it certainly looks like Google chose knowingly to play with fire and is now suddenly outraged that they, too, got burned.

Google, you have enough people willing to play along with your attempt to be the victim. I’m not one of them. You compromised human rights principles in 2006 and knowingly put your users into harm’s way. “Do no evil,” my ass.

Over the next couple of months, I’d like to slowly sketch out some of the thoughts and impressions that I’ve been gathering about Exchange 2010 storage over the last year or so and combine them with the specific insights that I’m gaining at my new job. In this inaugural post, I want to tackle what I have come to view as the fundamental question that will drive the heart of your Exchange 2010 storage strategy: will you use a RAID configuration or will you use a JBOD configuration?

In the interests of full disclosure, the company I work for now is a strong NetApp reseller, so of course my work environment is conducive to designing Exchange in ways that make it easy to sell the strengths of NetApp kit. However, part of the reason I picked this job is precisely because I agree with how they address Exchange storage and how I think the Exchange storage paradigm is going to shake out in the next 3-5 years as more people start deploying Exchange 2010.

In Exchange 2010, Microsoft re-designed the Exchange storage system to target what we can now consider to be the lowest common denominator of server storage: a directly attached storage (DAS) array of 7200 RPM SATA disks in a Just a Box of Disks (JBOD) configuration. This DAS/JBOD/SATA (what I will now call DJS) configuration has been an unworkable configuration for Exchange for almost its entire lifetime:

• The DAS piece certainly worked for the initial versions of Exchange; that’s what almost all storage was back then. Big centralized SANs weren’t part of the commodity IT server world, reserved instead for the mainframe world. Server administrators managed server storage. The question was what kind of bus you used to attach the array to the server. However, as Exchange moved to clustering, it required some sort of shared storage. While a shared SCSI bus was possible, it not only felt like a hack, but also didn’t scale well beyond two nodes.
• SATA, of course, wasn’t around back in 1996; you had either IDE or SCSI. SCSI was the serious server administrator’s choice, providing better I/O performance for server applications, as well as faster bus speeds. SATA, and its big brother SAS, both are derived from the lessons that years of SCSI deployments have provided. Even for Exchange 2007, though, SATA’s poor random I/O performance made it unsuitable for Exchange storage. You had to use either SAS or FC drives.
• RAID has been a requirement for Exchange deployments, historically, for two reasons: to combine enough drive spindles together for acceptable I/O performance (back when disks were smaller than mailbox databases), and to ensure basic data redundancy. Redundancy was especially important once Exchange began supporting shared storage clustering and required both aggregate I/O performance only achievable with expensive disks and interfaces as well as the reduced chance of a storage failure being a single point of failure.

If you look at the marketing material for Exchange 2010, you would certainly be forgiven for thinking that DJS is the only smart way to deploy Exchange 2010, with SAN, RAID, and non-SATA systems supported only for those companies caught in the mire of legacy deployments. However, this isn’t at all true. There are a growing number of Exchange experts (and not just those of us who either work for storage vendors or resell their products) who think that while DJS is certainly an interesting option, it’s not one that’s a good match for every customer.

In order to understand why DJS is truly possible in Exchange 2010, and more importantly begin to understand where DJS configurations are a good fit and what underlying conditions and assumptions you need to meet in order to get the most value from DJS, we need to separate these three dimensions and discuss them separately.

While I will go into more detail on all three dimensions at later date, I want to focus on the JBOD vs.. RAID question now. If you need some summaries, then check out fellow Exchange MVP (and NetApp consultant) John Fullbright’s post on the economics of DAS vs. SAN as well as Microsoft’s Matt Gossage and his TechEd 2009 session on Exchange 2010 storage. Although there are good arguments for diving into drive technology or storage connection debates, I’ve come to believe that the central philosophy question you must answer in your Exchange 2010 design is at what level you will keep your data redundant. Until Exchange 2007, you had only one option: keeping your data redundant at the disk controller level. Using RAID technologies, you had two copies of your data[1]. Because you had a second copy of the data, shared storage clustering solutions could be used to provide availability for the mailbox service.

With Exchange 2007’s continuous replication features, you could add in data redundancy at the application level and avoid the dependency of shared storage; CCR creates two copies, and SCR can be used to create one or more additional copies off-site. However, given the realities of Exchange storage, for all but the smallest deployments, you had to use RAID to provide the required number of disk spindles for performance. With CCR, this really meant you were creating four copies; with SCR, you were creating an additional two copies for each target replica you created.

This is where Exchange 2010 throws a wrench into the works. By virtue of a re-architected storage engine, it’s possible under specific circumstances to design a mailbox database that will fit on a single drive while still providing acceptable performance. The reworked continuous replication options, now simplified into the DAG functionality, create additional copies on the application level. If you hit that sweet spot of the 1:1 database to disk ratio, then you only have a single copy of the data per replica and can get an n-1 level of redundancy, where n is the number of replicas you have. This is clearly far more efficient for disk usage…or is it? The full answer is complex, the simple answer is, “In some cases.”

In order to get the 1:1 database to disk ratio, you have to follow several guidelines:

1. Have at least three replicas of the database in the DAG, regardless of which sites they are in. Doing so allows you to place both the EDB and transaction log files on the same physical drive, rather than separating them as you did in previous versions of Exchange.
2. Ensure that you have at least two replicas per site. The reason for this is that unlike Exchange 2007, you can reseed a failed replica from another passive copy. This allows you to avoid reseeding over your WAN, which is something you do not want to do.
3. Size your mailbox databases to include no more users than will fit in the drive’s performance envelope. Although Exchange 2010 converts many of the random I/O patterns to sequential, giving better performance, not all has been converted, so you still have to plan against the random I/O specs.
4. Ensure that write transactions can get written successfully to disk. Use a battery-backed caching controller for your storage array to ensure the best possible performance from the disks. Use write caching for the physical disks, which means ensuring each server hosting a replica has a UPS.

At this point, you probably have disk capacity to spare, which is why Exchange 2010 allows the creation of archive mailboxes in the same mailbox database. All of the user’s data is kept at the same level of redundancy, and the archived data – which is less frequently accessed than the mainline data – is stored without additional significant disk or I/O penalty. This all seems to indicate that JBOD is the way to go, yes? Two copies in the main site, two off-site DR copies, and I’m using cheaper storage with larger mailboxes and only four copies of my data instead of the minimum of six I’d have with CCR+SCR (or the equivalent DAG setup) on RAID configurations.

Not so fast. Microsoft’s claims around DJS configurations usually talk about the up-front capital expenditures. There’s more to a solid design than just the up-front storage price tag, and even if the DJS solution does provide savings in your situation, that is only the start. You also need to think about the lifetime of your storage and all the operational costs. For instance, what happens when one of those 1:1 drives fails?

Well, if you bought a really cheap DAS array, your first indication will be when Exchange starts throwing errors and the active copy moves to one of the other replicas. (You are monitoring your Exchange servers, right?) More expensive DAS arrays usually directly let you know that a disk failed. Either way, you have to replace the disk. Again, with a cheap white-box array, you’re on your own to buy replacement disks, while a good DAS vendor will provide replacements within the warranty/maintenance period. Once the disk is replaced, you have to re-establish the database replica. This brings us to the wonderful manual process known as database reseeding, which is not only a manual task, but can take quite a significant amount of time – especially if you made use of archival mailboxes and stuffed that DJS configuration full of data. Let’s take a closer look at what this means to you.

[Begin 4/13 update]

There’s a dearth of hard information out there about what types of reseed throughputs we can achieve in the real world, and my initial version of this post where I assumed 20GB/hour as an “educated guess” earned me a bit of ribbing in some quarters. In my initial example, I said that if we can reseed 20GB of data per hour (from a local passive copy to avoid the I/O hit to the active copy), that’s 10 hours for a 200GB database, 30 hours for a 600GB database, or 60 hours –two and a half days! – for a 1.2 TB database[2].

According to the Top 10 Exchange Storage Myths post on the Exchange team blog, 20GB/hour is way too low; in their internal deployments, they’re seeing between 35-70GB per hour. How would these speeds affect reseed times in my examples above? Well, let’s look at Table 1:

Table 1: Example Exchange 2010 Mailbox Database reseed times

As you can see, reseed time can be a key variable in a DJS design. In some cases, depending on your business needs, these times could make or break whether this is a good design. I’ve done some talking around and found out that reseed times in the field are all over the charts. I had several people talk to me at the MVP Summit and ask me under what conditions I’d seen 20GB/hour, as that was too high. Astrid McClean and Matt Gossage of Microsoft had a great discussion with me and obviously felt that 20GB/hour is way too low.

Since then, I’ve received a lot of feedback and like I said, it’s all over the map. However, I’ve yet to hear anyone outside of Microsoft publicly state a reseed throughput higher than 20GB/hour. What this says to me is that getting the proper network design in place to support a good reseed rate hasn’t been a big point in deployments so far, and that in order to make a DJS design work, this may need to be an additional consideration.

If your replication network is designed to handle the amount of traffic required for normal DAG replication and doesn’t have sufficient throughput to handle reseed operations, you may be hurting yourself in the unlikely event of suffering multiple simultaneous replica failures on the same mailbox database.

This is a bigger concern for shops that have a small tolerance for any given drive failure. In most environments, one of the unspoken effects of a DJS DAG design is that you are trading number of replicas – and database-level failover – for replica rebuild time. If you’re reduced from four replicas down to three, or three down to two during the time it takes to detect the disk failure, replace the disk, and complete the reseed, you’ll probably be okay with that taking a longer period of time as long as you have sufficient replicas.

All during the reseed time, you have one fewer replica of that database to protect you. If your business processes and requirements don’t give you that amount of leeway, you either have to design smaller databases (and waste the disk capacity, which brings us right back to the good old bad days of Exchange 2000/2003 storage design) or use RAID.

[End 4/13 update]

Now, with a RAID solution, we don’t have that same problem. We still have a RAID volume rebuild penalty, but that’s happening inside the disk shelf at the controller, not across our network between Exchange servers. And with a well-designed RAID solution such as generic RAID 10 (1+0) or NetApp’s RAID-DP, you can actually survive the loss of more disks at the same time. Plus, a RAID solution gives me the flexibility to populate my databases with smaller or larger mailboxes as I need, and aggregate out the capacity and performance across my disks and databases. Sure, I don’t get that nice 1:1 disk to database ratio, but I have a lot more administrative flexibility and can survive disk loss without automatically having to begin the reseed dance.

Don’t get me wrong – I’m wildly enthusiastic that I as an Exchange architect have the option of designing to JBOD configurations. I like having choices, because that helps me make the right decisions to meet my customers’ needs. And that, in the end, is the point of a well-designed Exchange deployment – to meet your needs. Not the needs of Microsoft, and not the needs of your storage or server vendors. While I’m fairly confident that starting with a default NetApp storage solution is the right choice for many of the environments I’ll be facing, I also know how to ask the questions that lead me to consider DJS instead. There’s still a place for RAID at the Exchange storage table.

In further installments over the next few months, I’ll begin to address the SATA vs. SAS/FC and DAS vs. SAN arguments as well. I’ll then try to wrap it up with a practical and realistic set of design examples that pull all the pieces together.

[1] RAID-1 (mirroring) and RAID-10 (striping and mirroring) both create two physical copies of the data. RAID-5 does not, but it allows the loss of a single drive failure — effectively giving you a virtual second copy of the data.

[2] Curious why picked these database sizes?  200GB is the recommended maximum size for Exchange 2007 (due to backup limitations), and 600GB/1.2TB are the realistic recommended maximums you can get from 1TB and 2TB disks today in a DJS replica-per-disk deployment; you need to leave room for the content index, transaction logs, and free space.

I wish I could change the RAM, CPU configuration on running VMs in #VMWare and have the changes apply on next reboot.

This prompted one of my nieces, a lovely and intelligent young lady in high school, to ask me to say that in English.

I pondered just hand waving it, but I was loathe to do so. Like I said, she’s intelligent. I firmly believe that kids live up to your expectations; if you talk down to them and treat them like they’re dumb because that’s what you expect, they’re happy to be that way. On the other hand, if you expect them to be able to understand concepts with the proper explanations, even if they may not immediately grasp the fine points, I’ve found that kids are actually quite able to do so – better than many adults, truth be told.

So, this is my answer:

The physical machinery of computers is called hardware. The programs that run on them (Windows, games, etc.) is software.
VMware is software that allows you to create virtual machines. That is, instead of buying (for example) 10 computers to do different tasks and have most of them have unused memory and processor power, you buy one or two really beefy computers and run VMWare. That allows you to create a virtual machine in software, so those two computers become 10. I don’t have to buy quite as much hardware because each virtual machine only uses the resources it needs, leaving the rest for the other virtual machines.

However, one of the problems with VMWare currently is that if you find you’ve given a virtual machine too much memory or processor (or not enough), you have to shut it down, make the change, then start it back up. I want the software to be smart enough to take the change *now* and automatically apply it when it can, such as when the virtual machine is rebooting. For a physical computer, it makes sense — I have to power it down, crack the case open, put memory in, etc. — but for a virtual computer, it should be able to be done in software.

Think of it this way: hardware is like a closet. You can build a big closet or a small closet or a medium closet, but each closet holds a finite amount of stuff. Software is the stuff you put in the closet — clothes, shoes, linens, etc. You can dump a bunch of stuff into a big closet, but doing so makes it cluttered and hard to use. So if you use multiple smaller closets, you’re wasting space because you probably won’t fill every one exactly.

In this metaphor, virtualization is like a closet organizer system. You can add a clothing rod here to hang dresses and blouses on, and underneath that add a shelf or two for shoes, while to the side you have more shelves for pants and towels and other stuff. You waste a little bit of your closet space for the organizer, but you keep everything organized and clutter-free, which means you’re better off and take less time to keep everything up.

Of course, this metaphor fails on my original point, because it totally makes sense you have to take all the stuff off shelves before moving those shelves around. In the world of software, though, it doesn’t necessarily make sense — it’s just the right people didn’t think of it at the right time.

Clear?

I came close to busting out Visio and starting to diagram some of this. I decided not to.

Edit: I don’t have to diagram it! Thank you, Ikea, and your lovely KOMPLEMENT wardrobe organizer line!

First, we have confirmation that his naughty and nice database has been hacked.

Now, there are credible rumors that the North Pole CIO has been covering up a years-long, systemic problem with Santa losing mobile devices. According to unidentified sources, the list of allegations includes:

• Lack of priority for safeguarding key data, especially through mobile systems. Recent refits for the sled have focused on tracking transponders for “greater publicity”, but no corresponding upgrades to mobile IT systems. These systems are specifically characterized as “obsolete 286 systems running DOS and home-brew Paradox applications written by some dentist in his spare time.”
• Habitual problems with smartphones. In order to ensure inexpensive world-wide access, Santa’s system includes the use of multiple handsets from strategically selected regional carriers. “In the last several years, Santa has yet to come back from his Christmas Eve run without having lost at least three of his devices,” one insider claims, “and of course we don’t have remote wipe capabilities. That would require him spending money.”
• Lax information and network practices, including no formal security policies or processes. Remote accesses aren’t even protected via SSL, according to sources, since “anyone who’s so cheap they haven’t updated stock PR footage of elves making wooden toys isn’t likely to shell out for a respected SSL certificate or PKI infrastructure.”

It will take time to gather confirmation of these claims, but if they are true, it shows a shocking disregard for basic security best practices at the North Pole.

At any rate, I’m back with some cool Exchange blogging. I’ve been getting a chance to dive into a “All-Devin, All-Exchange, All The Time” groove and it’s been a lot of fun, some of the details of which I hope to be able to share with you in upcoming months. In the process, I’ve been building a brand new Exchange 2010 lab environment and ran smack into a myth that seems to be making the rounds among people who are deploying Exchange 2010. This myth gives bum advice for those of you who are deploying an Exchange 2010 DAG and not using an Exchange 2010 Hub Transport as your File Share Witness (FSW). I call it the Exchange Trusted Subsystem Myth, and the first hint of it I see seems to be on this blog post. However, that same advice seems to have gotten around the net, as evidenced by this almost word-for-word copy or this posting that links to the first one. Like many myths, this one is pernicious not because it’s completely wrong, but because it works even though it’s wrong.

If you follow the Exchange product group’s deployment assumptions, you’ll never run into the circumstance this myth addresses; the FSW is placed on an Exchange 2010 HT role in the organization. Although you can specify the FSW location (server and directory) or let Exchange pick a server and directory or you, the FSW share isn’t created during the configuration of the DAG (as documented by fellow Exchange MVP Elan Shudnow and the “Witness Server Requirements” section of the Planning for High Availability and Site Resilience TechNet topic). Since it’s being created on an Exchange server as the second member of the DAG is joined, Exchange has all the permissions it needs on the system to create the share. If you elect to put the share on a non-Exchange server, then Exchange doesn’t have permissions to do it. Hence the myth:

1. Add the FSW server’s machine account to the Exchange Trusted Subsystem group.
2. Add the Exchange Trusted Subsystem group to the FSW server’s local Administrators group.

The sad part is, only the second action is necessary. True, doing the above will make the FSW work, but it will also open a much wider hole in your security than you need or want. Let me show you from my shiny new lab! In this configuration, I have three Exchange systems: EX10MB01, EX10MB02, and EX10MB03. All three systems have the Mailbox, Client Access, and Hub Transport roles. Because of this, I want to put the FSW on a separate machine. I could have used a generic member server, but I specifically wanted to debunk the myth, so I picked my DC EX10DC01 with malice aforethought.

• In Figure 1, I show adding the Exchange Trusted Subsystem group to the Builtin/Administrators group on EX10DC01. If this weren’t a domain controller, I could add it to the local Administrators group instead, but DCs require tinkering. [1]

Figure 1: Membership of the Builtin/Administrators group on EX10DC01

• In Figure 2, I show the membership of the Builtin/Administrators group on EX10DC01. No funny business up my sleeve!

Figure 2: Membership of the Exchange Trusted Subsystem group

• I now create the DAG object, specifying EX10DC01 as my FSW server and the C:\EX10DAG01 directory so we can see if it ever gets created (and when).
• In Figure 3, I show the root of the C:\ drive on EX10DC01 after adding the second Exchange 2010 server to the DAG. Now, the directory and share are created, without requiring the server’s machine account to be added to the Exchange Trusted Subsystem group.

Figure 3: The FSW created on EX10DC01

I suspect that this bad advice came about through a combination of circumstances, including an improper understanding of Exchange caching of Active Directory information and when the FSW is actually created. However it came about, though, it needs to be stopped, because any administrator that configures their Exchange organization is opening a big fat hole in the Exchange security model.

So, why is adding the machine account to the Exchange Trusted Subsystem group a security hole? The answer lies in Exchange 2010’s shift to Role Based Access Control (RBAC). In previous versions of Exchange, you delegated permissions directly to Active Directory and Exchange objects, allowing users to perform actions directly from their security context. If they had the appropriate permissions, their actions succeeded.

In Exchange 2010 RBAC, this model goes away; you now delegate permissions by telling RBAC what options given groups, policies, or users can perform, then assigning group memberships or policies as needed. When the EMS cmdlets run, they do so as the local machine account; since the local machine is an Exchange 2010 server, this account has been added to the Exchange Trusted Subsystem group. This group has been delegated the appropriate access entries in Active Directory and Exchange databases objects, as described in the Understanding Split Permissions TechNet topic. For a comprehensive overview of RBAC and how all the pieces fit together, read the Understanding Role Based Access Control TechNet topic.

By improperly adding a non-Exchange server to this group, you’re now giving that server account the ability to read and change any Exchange-related object or property in Active Directory or Exchange databases. Obviously, this is a hole, especially given the relative ease with which one local administrator can get a command line prompt running as one of the local system accounts.

So please, do us all a favor: if you ever hear or see someone passing around this myth, please, link them here.

Busted!

[1] Yes, it is also granting much broader permissions than necessary to make a DC the FSW node. Now the Exchange Trusted Subsystem group is a member of the Domain Admins group. This is probably not what you want to do, so really, don’t do this outside of a demo lab.

I love living in the future. First, though, watch this video that Alaric and I made.

I was a Boy Scout for close to three years. I started as a Boy Scout; I missed Cub Scouts, including Webelos Scout. When I was in Scouting, we had to go door-to-door to do our fundraisers, or spend a lot of time with our relatives over the phone. I hated doing it, for reasons that didn’t become clear until much later in life when I began grappling with autism and Asperger’s. However, I have a lot of good memories of Scouting; it did a lot for me and it was a valuable part of my childhood.

Steph and I wanted Alaric to experience Scouting. Even though the modern BSA has some characteristics that I don’t agree with, I’ve come to the decision that first and foremost, Scouting is about the boys. Scouting needs intelligent, reasonable adults of all persuasions to help drive the program. By being part of Scouting, Alaric will learn and do things Steph and I can’t give him on our own; by having us there with him, Alaric will learn how to deal with people from differing backgrounds in a diplomatic and productive manner.

Over the summer, Alaric has really seen what a good thing Scouting is. He even got me to go to Scout Camp with him for four days in July, and I must admit I even had fun. It was a great experience for both of us, including facing down and conquering some challenges.

Unlike many Scout packs and troops, Alaric’s pack works on the schedule of the school year. As a result, they do their major fundraising push at the beginning of the school year with a number of activities. Alaric’s already helped out pulling Hire-A-Scout wagons at the local auto swap meet and had a great time. However, the major source of operating funds is the traditional Trail’s End popcorn fundraiser. Trail’s End, if you don’t know, has been the go-to-source for Scout fundraising for a long time, and they offer some of the best popcorn on the planet.

Over the past few weeks, we’ve been rather hectic and busy and haven’t really had time to coach Alaric on his first door-to-door sales campaign. (Poor guy seems to have the same issues I did when I was his age, so it was pretty painful.) This last week, I came up with what is I hope a brainstorm: harness the power of the Internet to get Alaric’s sales pitch out there. So, you get to enjoy the results: the following video where Alaric and I pitch popcorn to YOU, the faithful reader. And because this is the future, Trail’s End even got with the program: they now allow you to purchase online, supporting a specific Scout, and have the product shipped directly to your door!

Go to Trail’s End to support Alaric’s fundraising for his pack

Thank you for your support!

I think that the Ecclesiastes 3:1 says it best; in the King James Version, the poet says, “To every thing there is a season, and a time to every purpose under the heaven.” It has been my privilege to use this blog to talk about Exchange, data protection, and all the other topics I’ve talked about since my first post here five years ago (holy crap, has it really been five years???) With 3Sharp’s gracious permission and blessing, I’ll be duplicating all of the content I’ve posted here over on my personal blog, Devin on Earth. If you have a link or bookmark for this blog or are following me via RSS, please take a moment to update it now (Devin on Earth RSS feed). I’ve got a few new posts cooking, but this will be my last post here.

Thank you to 3Sharp and the best damn co-workers I could ever hope to work with over the years. Thank you, my readers. You all have helped me grow and solidify my skills, and I hope I returned the favor. I look forward to continuing the journey with many of you, even if I’m not sure yet where it will take me.

Yes, that’s right, zombies. The RAHR-BRAINS-RAHR shambling undead kind, not the “mystery objects in Active Directory” kind.

Well, now you can see what I was up to.

I was working with long-time fellow 3Sharpie David Gerhardt on creating a series of 60-second vignettes for the upcoming Office 2010 application suite. Each vignette focuses on a single new area of functionality in one of the Office products. I got to work with OneNote 2010.

Here’s where the story gets good.

I got brought into the project somewhat late, after a bunch of initial planning and prep work had been done. The people who had been working on the project had decided that they didn’t want to do the same boring business-related content in their OneNote 2010 vignettes; oh, no! Instead, they hit upon the wonderful idea of using a Zombie Plan as the base document. Now, I don’t really like zombies, but this seemed like a great way to spice up a project!

The rest, as they say, is history. Check out the results (posted both at GetSharp and somewhere out on YouTube) for yourself:

• OneNote 2010 Advanced Wiki Features
• OneNote 2010 Linked Notes
• OneNote 2010 Quick Filing
• OneNote 2010 Search UI

One of the best parts of this project, other than getting a chance to learn about some of the wildly cool stuff the OneNote team is doing to enhance an already wonderful product, was the music selection. We worked a deal with local artist Dave Pezzner to use some of his short music clips for these videos. Dave is immensely talented and provided a wide selection of material, so I enjoyed being able to pick and choose just the right music for each video. It did occur to me how cool it would be if I could use Jonathan Coulton’s fantastic song Re: Your Brains, but somehow I think his people lost my query email. Such is life – and I think Mr. Pezzner’s music provided just the right accompaniment to the Zombie Plan content.

Enjoy!