Windows "shared storage" from 2003(SCSI 2) to 2008 & R2(SCSI 3) clusters

2012-08-31T04:29:00.000-07:00

Windows "shared storage" from 2003(SCSI 2) to 2008 & R2(SCSI 3) clusters

Hello Everyone , After working several years in Microsoft we have seen a lot of changes in the shared storage architecture . I bet the folks who have worked on 2000 clusters storage and moved on to 2003 and henceforth on 2008 R2 clusters will find the shared storage architecture a lot different and much better and reliable indeed !!!

The Cluster service only uses the SCSI protocol to manage disks on the shared bus.

Note This does not mean that all disks will be of type SCSI, specifying the hardware interface known as SCSI, but rather that the storage unit must be able to properly interpret and process the SCSI protocol and commands.

The following lists of commands are the additional SCSI protocol features that will be used when disks are in a clustered environment.

· Reserve: This command is issued by a host bus adapter to obtain or maintain ownership of a SCSI device. A device that is reserved refuses all commands from all other host bus adapters except the one that initially reserved it, the initiator.

· Release: This command is issued by the owning host bus adapter when a disk resource is taken offline; it frees a SCSI device for another host bus adapter to reserve.

· Reset: This command breaks the reservation on a target device. This command can either be a bus reset (for the entire bus) or, using the storport drivers (introduces in WIn 2k3 Sp2) a targeted reset for a particular device on the bus.

The following procedure describes how a server cluster starts and gains control of the shared disks. This scenario assumes that only one node is being turned on at a time:

When the computer is started, the Cluster Disk Driver (Clusdisk.sys) reads the following local registry key to obtain a list of the signatures of the shared disks under cluster management:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ClusDisk\Parameters \Signatures

After the list is obtained, the cluster service attempts to scan all of the devices on the shared SCSI bus to find matching disk signatures.

When the first node in the cluster starts, the cluster disk driver first marks all LUNs (LUN: logical unit number, a unique identifier used on a SCSI bus to distinguish between devices that share the same bus) matching the Signatures key as offline volumes. Note that this is not the same as taking a cluster resource offline. The volume is marked offline to prevent multiple nodes from having write access to the volumes simultaneously. If the cluster is a shared disk cluster, one of the disks is designated as quorum disk by the cluster service. Quorum disk is the first resource brought online when cluster service attempts to form a cluster.

When the cluster service on the forming node starts, it first tries to bring online the physical device designated as quorum disk. It executes the disk arbitration algorithm on the quorum disk to gain ownership. On successful arbitration, cluster service sends a request to clusdisk to start sending periodic reserves to the disk (to maintain ownership). Then cluster service sends a request to clusdisk to unblock access to the quorum disk and mounts the volumes on the disk. Successful mounting of the volume(s), completes the online procedure and the cluster service then continues with the cluster form process. The request is passed from the cluster disk driver to the Microsoft storage driver stack and finally to the driver specific to the HBA that communicates to the disks. It may also be passed to any multipath software running in the storage stack. For more information regarding storage stacks and driver models, please click the following links:
After the storage controller/device driver reports that the device has been successfully reserved, the cluster service ensures that the drive can be read from and written to. Once the disk has passed all of these tests, the disk resource are marked as online and the cluster service then continues to bring all other resources online.

Each node in the cluster renews reservations for any LUNs it owns every three seconds. If the nodes of a cluster lose network communication with each other (for example, if there is no communication over the private or public network), the nodes begin a process known as arbitration to determine ownership of the quorum disk. The node that wins ownership of the quorum disk resources in total communication loss between cluster nodes will remain functional. Any nodes that cannot communicate and cannot maintain or acquire ownership of the quorum disk will terminate the cluster service and any resources that node was hosting will be moved to another node in the cluster.

1. The node that currently owns the quorum disk is the defending node. The defender assumes that it is defending against any cluster nodes that it cannot communicate with and for which it did not receive a shutdown notification. The defender continually renews its reservation to the quorum by requesting a SCSI reserve be placed on the LUN every three seconds.

2. All other nodes (nodes that do not own the quorum disk and cannot communicate with the node that owns the quorum resource) become challenging nodes.

3. When the challenger detects the loss of all communications, it immediately requests a bus-wide SCSI reset to break any existing reservations.

4. Seven seconds after the SCSI reset requested, the challenger tries to reserve the quorum disk. If the defender node is online and functioning, it will have already reserved the quorum disk as it typically does every three seconds. The challenger detects that it cannot reserve the quorum, and terminates the cluster service. If the defender is not functioning properly, the challenger can successfully reserve the quorum disk. After ten seconds, the challenger brings the quorum online and takes ownership of all resources in the cluster. If the defending node loses ownership of the quorum device, then the cluster service on the defending node terminates immediately.

When a cluster node takes a disk resource offline, it requests that the SCSI reserve be released and then the drive will once again be unavailable to the operating system. Anytime a disk resource is offline in a cluster, the volume that the resource points to (the disk with the matching signature) will be inaccessible to the operating system on any of the cluster nodes.

Now after Induction of 2008 clusters we moved on from SCSI Reserve and Release Mechanism to Shared Persistent reservation key it has reduced issues which we use to face like data corruption because of complete bus reset and flushing the Disk IO .

Here is the difference between SCSI 3 and SCSI 2 Commands :

One of the areas where Windows 2003 Clustering historically had the most challenges, and where most support calls were logged was with storage problems. The reason for that was that SCSI-2 commands were used for managing and manipulating storage; in particular the reserve, reset and release commands. These commands are used to basically 'lock' the shared storage for the active node, and move it over to the other node in case of failure.

The reason behind these often occurring issues was that the SCSI Primary Commands-2 specification was going through multiple layers to finally get to the storage array. For example, this is what would happen when a typical Cluster SCSI such as a 'LUN Reset' would be requested and sent into the IO stack:

The Clustered disk device sends the reset LUN request.
This goes to the Multipath software, who translates it to a different SCSI command
This then ends up at the (SCSI) Port driver, which may not understand a targeted SCSI LUN reset command, but knows about a SCSI BUS reset. (Dangerous!)
Down the stack, the HBA receives this SCSI Bus reset. Let's assume we are using an iSCSI SAN; it does not know about SCSI Bus resets, but does know a SCSI cold or warm reset, which is similar and translates it into a cold reset and sends that down the stack.
This cold reset is optional in the iSCSI specs, and since they are disruptive the storage array decides not to do it, but simply sends back a "status success"

What it comes down to is this: What started out as a targeted LUN reset being requested by the Clustering service, ends up being translated through many layers of translation to a cold reset, only to be ignored. There is so much lost in translation that in the real world, the cluster never failed-over and changed to the other node! Of course the example is a worse-case scenario; but with the SCSI-2 commands this was essentially possible.

An extra challenge is that each layer introduces potential disruptions to the environment when changing, for example: a simple firmware update may result in one of the layers doing a slightly different translation, and with that rendering your cluster fail-over unusable.

SCSI Primary Commands -3

In Windows 2008 Clustering the change was made to only support SCSI Primary Commands-3 specification The advantage being that now "Persistent Reservations" can be used (see page 62 of the earlier linked .pdf file).

A quote from the SPC-3 specs: "The PERSISTENT RESERVE OUT and PERSISTENT RESERVE IN commands provide the basic mechanism for dynamic contention resolution in systems with multiple initiator ports accessing a logical unit."

This means that the command requested by the top level Cluster Disk service is –exactly- the same as the request sent down the stack and received by the Storage Array. There is no room for interpretation or anything being lost in translation.

This way a large number of storage issues with Clustering under Windows 2003 have now been fixed with Windows 2008 Clustering. This makes installation as well as managing a clustered environment a whole lot easier, at the expense however of unsuspecting admins who want to migrate to Windows 2008 not having checked their shared storage to be SPC-3 compatible. Also, as I have mentioned in my iSCSI article: a lot of the iSCSI software target offerings are not yet SPC-3 compatible, which means quite a few labs being built for clustering will also encounter this issue.

The conclusion; with moving to SPC-3 Microsoft has made what used to be an area of frustration in Clustering: shared storage, almost effortless and with that they will have reduced the number of support calls on this area. The funny thing is that I'm pretty sure that this SCSI -3 persistent reservation error- messages in the Cluster configuration validation wizard will bump that number of support calls right up again. Especially since some people will want to understand why the hardware that used to run smoothly with their Windows 2003 cluster, is no longer supported!

http://blogs.technet.com/b/askcore/archive/2011/06/02/my-disk-is-read-only-help.aspx (Read only attribute)

http://technet.microsoft.com/en-us/library/cc773485(WS.10).aspx (Node cleanup)

http://scsi3pr.blogspot.in/2010/11/scsi-3-persistent-group-reservation.html (Overview of SCSI Commands )
In the next blog I will try to explain the Windows storage architecture and a bit about NTFS file system .

Please provide your feedback and any inputs are welcome to improve the quality of blogs .

Proyoung

Windows Storage

Windows "shared storage" from 2003(SCSI 2) to 2008 & R2(SCSI 3) clusters

Windows "shared storage" from 2003(SCSI 2) to 2008 & R2(SCSI 3) clusters