In this particular case study, I’m working with a customer with two data centers, New York and Washington DC with a 50 Mb/s line between the two data centers. EMC RecoverPoint is the replication technology of choice, and the customer is doing bi-directional replication. The Washington DC site has about 4TB of data that needs to replicate to New York, and the New York site had about 10TB of data that needs to replicate to Washington DC.

In a perfect world (no latency, no packet loss, 100% utilization of the link), it would take roughly 7 days to replicate the 4TB and roughly 18 days to replicate the 10TB. That’s almost a month to move the data with the link fully saturated. Unfortunately, that link was also used for other business uses, e.g. VoIP traffic, internal application traffic, server monitoring traffic, etc. Thus the CIO mandated that we find another method to perform the initial synchronization, as using the link (even throttled) was not an option for this duration.

The Approach
The EMC RecoverPoint Release 3.4 Administrator Guide (P/N 300-012-256) documents a method for performing first-time initialization from backup. The primary kicker here though, is that the backup must be a block-level backup, not a file-level backup. This is because the target RecoverPoint image will be seeded with that block-level backup and then RecoverPoint will perform a full volume sweep to synchronize the incremental changes since the block-level backup.

Most companies, however, do not perform block-level backups of their servers. Rather, they perform file-level backups, which then gets catalogued for easy restores. Below is a summary of the process I used to perform the RecoverPoint initial synchronization using dd as the block-level backup.

Pre-requisites/Setup

• Downloaded dd on a Windows utility server
  • http://www.chrysocome.net/downloads/dd-0.6beta3.zip
  • This is the tool we will use for the block-level backups.

    dd if=[vol_source] of=[vol_target] bs=512k

  • Note: I did some very rudimentary performance tests to see what block size would be optimal for these backups. I found 512k to be the sweet spot.
• Configured clones for all volumes that will be seeded with RecoverPoint. The main reason for this is two fold:
  • I didn’t want to impact the performance of the production volume while dd reads from the source volume to create the backup.
  • dd cannot operate against volumes with open files. Thus, we’d need to bring down the applications for the duration of the dd backup. When performing a dd against a mounted clone and against the PHYSICALDRIVE address, I did not get open file errors. Below is an example of the errors you will see with dd if there are open files.

    C:\Utilities>dd if=\\.\H: of=z:\testvolume.img bs=512k
    rawwrite dd for windows version 0.6beta3.
    Written by John Newbigin <jn@it.swin.edu.au>
    This program is covered by terms of the GPL Version 2.
    Error opening input file: 32 The process cannot access the file because it is being used by another process

  • The source volumes were on 15k drives and the target volumes were on 7.2k SATA drives. I was able to copy roughly 5 GB/min (+/-0.5) with this process.

The Process 

1. For new source volumes, confirm that all the data has first been migrated before proceeding.
2. Configure the consistency group(s) for the volumes in scope
   • When finishing the consistency group, do not start the transfer. Leave the transfer paused.
3. Right-click the consistency group, select “Clear Markers”
   • This will let RP know that the remote site is known to be identical to its corresponding production volume. Thus a full volume sweep is not required.
   • When the dialog box pops up, select both copies.
   • Note: had to do this via command line because the GUI was only letting me clear the markers in the DR location. The command line without the copy=XYZ option allows you to clear all markers on both sides.

     clear_markers group=RPSyncTest

4. Create the block-level copy with dd
5. Transfer the copy to the secondary site. In our case, we shipped the USB drives to the secondary site.
6. Enable image access on the secondary volume
   • Select the latest image
   • After access goes to logged access, enable direct access
7. Restore the backup to the secondary volume
   • Remember, you already did the clear markers before you did the first dd copy. If you do it again, it will mess up tracking where the replication should resume.
   • No need to give the drive a drive letter or format it, as you can access it via the \\.\PHYSICALDRIVE2 address.
8. Disable image access and start the transfer
   • Check the “Start data transfer immediately” checkbox to resume replication
   • Monitor the consistency group. The traffic you see will be the changes to the source volume since the block-level dd copy was made. The duration should be significantly less than if it was a full copy, depending on how much data has changed since the original dd backup.

The Results
Below are some of the results from the initial synchronization process. Note that between the dd on the source and the reverse dd on the secondary volume, roughly two days elapsed.

• 330GB consistency group 1
  • At 50 Mb/s, it would have taken roughly 15 hours to perform a full sync.
  • Initial synchronization took 58 minutes, transferring roughly 21GB (6.36%).
  • We saved roughly 14 hours and 309GB of transfer.
• 330GB consistency group 2
  • Initial synchronization took 43 minutes, transferring roughly 10GB (3.03%).
  • We saved a little over 14 hours and 320GB of transfer.
• 330GB consistency group 3
  • Initial synchronization took 50 minutes, transferring roughly 18GB (5.45%).
  • We saved a little over 14 hours and 312GB of transfer.
• 330GB consistency group 4
  • Initial synchronization took 53 minutes, transferring roughly 19GB (5.76%).
  • We saved a little over 14 hours and 311GB of transfer.

The Commands

plink -l admin -pw admin 192.168.10.10 "enable_group group=RPSyncGroup start_transfer=no"
plink -l admin -pw admin 192.168.10.10 "clear_markers group=RPSyncGroup"
dd if=\\.\[PHYSICALDRIVE##] of=z:\[PHYSICALDRIVE##.img] bs=512k
[transfer the images to the secondary site via USB drive]
plink -l admin -pw admin 192.168.10.10 "enable_image_access group=RPSyncGroup copy=DR_RPSyncGroup image=latest"
plink -l admin -pw admin 192.168.10.10 "set_image_access_mode group=RPSyncGroup copy=DR_RPSyncGroup mode=direct"
dd if=z:\[PHYSICALDRIVE##.img] of=\\.\[PHYSICALDRIVE##] bs=512k
plink -l admin -pw admin 192.168.10.10 "disable_image_access group=RPSyncGroup copy=DR_RPSyncGroup start_transfer=no"
plink -l admin -pw admin 192.168.10.10 "start_transfer group=RPSyncGroup"
[monitor initial synchronization traffic]

1. Downloaded the latest package from http://www.phpmyadmin.net/home_page/downloads.php onto my laptop (192.168.1.119).
2. Used WinSCP to copy the file to my home directory.
3. Logged in and sudo’ed to root.
4. Copied the file from my home directory to /var/www/html, untarred the package, and renamed the directory to phpmyadmin.
5. I then went to access the server at http://192.168.1.135/phpmyadmin. I then encountered the following 403 error.

Forbidden
You don’t have permission to access /phpmyadmin on this server.

The error logs (/var/log/httpd/error_log) showed the following:

[Thu Apr 19 06:28:22 2012] [error] [client 192.168.1.119] (13)Permission denied: access to /phpmyadmin/ denied

I then sought the counsel of Google. Many web sites talk about either permissions on the directory/files or the httpd.conf configuration. My issue was none of those. It had to do with selinux which apparently comes built into the minimal CentOS 6.2 install.

[root@sandbox conf]# yum list | grep selinux
libselinux.i686 2.0.94-5.2.el6 @anaconda-CentOS-201112130233.i386/6.2
libselinux-utils.i686 2.0.94-5.2.el6 @anaconda-CentOS-201112130233.i386/6.2
selinux-policy.noarch 3.7.19-126.el6 @anaconda-CentOS-201112130233.i386/6.2
selinux-policy-targeted.noarch 3.7.19-126.el6 @anaconda-CentOS-201112130233.i386/6.2
ipa-server-selinux.i686 2.1.3-9.el6 base
libselinux-devel.i686 2.0.94-5.2.el6 base
libselinux-python.i686 2.0.94-5.2.el6 base
libselinux-ruby.i686 2.0.94-5.2.el6 base
libselinux-static.i686 2.0.94-5.2.el6 base
pki-selinux.noarch 9.0.3-21.el6_2 updates
selinux-policy.noarch 3.7.19-126.el6_2.10 updates
selinux-policy-doc.noarch 3.7.19-126.el6_2.10 updates
selinux-policy-minimum.noarch 3.7.19-126.el6_2.10 updates
selinux-policy-mls.noarch 3.7.19-126.el6_2.10 updates
selinux-policy-targeted.noarch 3.7.19-126.el6_2.10 updates

The problem was that the phpmyadmin package that I copied via WinSCP took the wrong context, which therefore didn’t have the appropriate permissions for apache to display.

[root@sandbox html]# ls -Z
-rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 info.php
drwxr-xr-x. root root unconfined_u:object_r:user_tmp_t:s0 phpmyadmin

To fix this, I needed to do the following:

chcon -R -t httpd_sys_content_t phpmyadmin

Note: be sure to use the -R to recursively apply that context against all files. Otherwise you will get a server misconfiguration error.

[root@sandbox html]# ls -Z
-rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 info.php
drwxr-xr-x. root root unconfined_u:object_r:httpd_sys_content_t:s0 phpmyadmin

In retrospect, had I downloaded the file via wget directly into the /var/www/html directory, it would have already taken the proper context, and I would not have had the issue.

• Configure enough resources for your workload
• Use resources evenly for best overall performance
  • Spread across all available components
  • Includes FE, BE and disks
  • Path management can help FE
  • FAST/Optimizer can help BE

Commonly asked questions

• What size system do I need?
  • Each resource has a limit of I/Os per second and MBs per second
    • Disks
    • Back-end controllers (DAs)
    • Front-end controllers (Fibre, FICON, GigE)
    • SRDF controllers
    • Slices (CPU complexes)
  • Configure enough components to support workload peaks
  • Use those resources as uniformly as possible
  • CPU utilization
    • As a rule of thumb, a limit of no more than 50-70% utilization is good if response time is critical
    • A higher utilization can be tolerated if only IOPS or total throughput matters
  • Memory considerations
    • Ideal to have same size memory boards and same memory between engines
    • Imbalance will make little or no difference with OLTP type workloads
    • Imbalance will create more accesses to boards or engines with large amount of memory, creating a skewed distribution over the hardware resources
  • Front-end connections
    • Go wide before you go deep
      • Use all 0 ports on director first and then the 1 ports
      • Spread across directors first, then on same director
      • Two active ports on one FA slice do not generally do more I/Os
    • Ratios (random read hit normalized at 1)
      • Random read hit 1
      • Random read miss 1/2
      • Random Overwrite I/O’s 1/2
      • Random new write 1/4
    • Worst connection for a host with 8 connections
      • All on one director
      • Instead do one connection per director
  • Disks
    • Performance will scale linearly as you add drives
      • You can see up to 510 IOPS per drive when benchmarking at 8KB, but 150 IOPS is a reasonable design number for real world situations
    • Note that with higher IOPS comes higher response times as well as queues will grow
    • Until some back-end director limit is reaches
    • With smaller I/O sizes (<32KB), the limit reaches is the CPU limit
    • With largest I/O sizes (>32KB), we can reach a throughput limit in the plumbing instead
  • Engine Scaling
    • Scales nearly linear, though not quite.
    • From 1 to 8 engines, it’s 6.8 to 7.8x WRT to IOPS (8KB I/O)
    • From 1 to 8 engines, it’s 4.2 to 7.1x WRT to bandwidth (64KB I/O)
    • Scaling from 1 to 8 shows worst numbers. 4 to 8 showed better numbers.
• What’s the optimum size of a hyper or number per disk?
  • General rule of thumb, fewer larger hypers will give better overall system performance.
    • There is a system overhead to manage a logical volume so it makes sense that more logical volumes could lead to more overhead.
  • Frequently legacy hyper size is carried forward because of migration
  • Virtual Provisioning will make the size of the hyper on the physical disk
    • You can create very large hypers for the TDATs and still present small LUNs to the host
  • There can be a case of having too few hypers per drive
    • Because it could limit concurrency
    • Set a minimum of 4 to 8 hypers
    • Not an issue with large drives or protections other than R1
• What is the optimum queue depth?
  • Single threaded (or 1 I/O at a time), the I/O rate is simply the inverse of the service time.
    • For a 5.8ms service time your maximum IOPS is 172.
    • Same drive with 128 I/Os queued can get nearly 500 IOPS
  • We need 1-4 I/Os queued to the disk to achieve the maximum throughput with reasonable latencies
    • Lower queue lengths if response time is CRITICAL
  • Higher if total IOPS is more important than response time
  • With VP, the LUN could be spread over 1000s of drives
    • Queue depth of 32 per VP LUN is probably a reasonable start
  • As IOPS go up, response time will exponentially get worse
• What is the optimum number of members in a meta volume?
  • 255 maximum supported
  • Reasonable sizes for meta member counts are something like 4, 8, 16, 32
  • Even numbers are preferred
    • Powers of 2 fit nicely into back-end configurations
    • Powers of 2 not important for VP thin metas
  • Getting enough I/O into a very large meta can be a problem
    • 32-way R5 7+1 meta volume would need at least 256 I/Os queued to have 1 I/O per physical disk
• Should I use meta volumes or host-based striping? Or both?
  • Avoid too many levels of striping (plaid)
  • One large meta volume may outperform serveral smaller meta volumes that are grouped in a host stripe
  • In many cases, host-based striping is preferred over meta volumes
    • One reason is because there will be more host-based queues for concurrency that the host can manage before even getting to the array.
  • However, meta volumes can reduce complexity at the host level
  • So it all depends
  • 24-way meta versus 6 host x 4-way meta – average read response time was better with host-based stripe
• Striped or Concatenated Metas?
  • In most cases, striped meta volumes will give you better performance than concatenated
    • Because they reside on more spindles
    • Some exceptions exist where concatenated may be better
      • If you don’t have enough drives for all the meta members to be on separate drives (wrapping)
      • If you plan to re-stripe many meta volumes again at the host-level
      • If you are making a very large R5/R6 meta and your workload is largely sequential
    • Concatenated meta volumes can be placed on the same RAID group
    • Don’t place striped meta volumes on the same RAID group (wrapped)
  • Virtual Provisioning
    • Back-end is already striped over the virtual provisioning pool so why re-stripe the thin volume (TDEV)
    • May be performance reasons to have a striped meta on VP
    • Device WP “disconnect” between front-end and backend
      • 5874 Q210SR, 5773 future SR fixes this
    • Number of random read requests we can send to a single device
      • Single device can have 8 outstanding reads per slice per device (TDEV on FA slice)
    • Number of outstanding SRDF/S writes per device
      • Single device can have 1 outstanding write per path per device
    • If it is important to be able to expand a meta, choose concatenated
• What stripe and I/O size should I choose?
  • For most host-based striping, 128KB or 256KB is good
  • May want to consider a smaller stripe size for database logs, 64KB or smaller may be advised by a Symmetrix performance guru
  • I/O sizes about 64KB or 128KB show little to no performance boost (flattens out). 256KB may actually decrease throughput. This is because everything is managed internally at 64KB chunks.
• Segregation
  • For the most optimal system performance, you should not segregate applications/BCVs/Clones onto separate physical disks/DAs or engines
  • For the most predictable system performance, you should segregate
  • Tiers should share DA resources so that one tier will not consume resources for another tier
• What disk drive class should I choose?
  • EFD provide the best response time and maximum IOPS of all drives
  • 15k provide 30% faster performance than 10k (random read miss)
  • 15k provide 56% faster than SATA, 10k provide 39% faster than SATA (random read miss)
  • SATA still does well in sequential read (with single threaded and larger block sizes) (basically good in single stream, bad with multi-thread and therefore disk seeks)
• What RAID protection should I choose?
  • Performance of reads similar across all protection types (number of drives is what matters)
  • Major difference with random write performance
    • Mirrored: 1 host write = 2 writes
    • R5: 1 host write = 2 reads + 2 writes
    • R6: 1 host write = 3 reads + 3 writes
  • Cost is also a factor
    • R5/R6 are best at 12.5% and 25% protection overhead
    • R1 has 50% protection overhead
• How much cache do I need?
  • Easiest method is to utilize the Dynamic Cache Partition White If (DCPwi) tool
  • Put like devices together in cache partitions
  • Start analysis mode and collect DCP stats
• How do I know when I’m getting close to limits?
  • Watch for growth trends in your workload with SPA
  • Look out for increasing response time (host-based tools like iostat, sar, RMF)
  • Monitor utilization metrics in WLA/STP
  • Better to be pro-active than waiting to hit th ewall
  • Any utilizations well over 50% should be considered a possible source of future issues with growth

• Utilization = 100% * busy time in period / (idle + busy) time in period
• Throughput = total number of visitors in periods / period in length in seconds
• Average Busy Queue Length = sum of queue upon arrive of visitor x / total number of visitors
• Queue length = ABQL * utilization/100%
• Response time = queue length / throughput (Little’s Law)

For low LUN throughput (<32 IOPS), response time might be inaccurate

• Response time here is calculated, lazy writes will skew the LUN busy counter
• RBA actually measures the response time

Dual SP ownership of a disk

• Can also impact response time
• Each SP only knows about its own ABQL, throughput and utilization for the disk
• At poll time, they exchange views. The utilization is max(SPA,SPB)
• ABQL is computed from the sum of the sum
• And SP throughput is the sum of SPA and SPB throughput

Be wary of confusing SP response time in Analyzer with the average response time of all LUNs on that SP

• Response time is calculated and based on utliization
• A LUN is busy (not resting) as long as something is queued to it
• An SP is busy (not resting) as long as it is not in the OS idle loop
• While a disk is busy getting a LUN request, the LUN is still busy
• While a disk is busy getting a LUN request, the SP might be idle
• The SP response time is generally smaller than the average response time of all the LUNs on that SP
• Host response time is approximated by LUN response time

Recall from last year:

• Rules of Thumb
• Multiplier (CPUM)
• CX4-960 – 1.00
• CX4-480 – 0.65
• CX4-240 – 0.55
• CX4-120 – 0.30
• CX3-80 – 0.50
• A – CPUM x 50k reads/s standard lun
• B – CPUM x 16k write/s R5
• C – CPUM x 20k writes/s R10
• D – CPUM x 40k reads/s, Snaps, MV/s, clone source
• E – CPUM x 7.5k writes/s MV/s
• F – CPUM x 6k writes/s, clone-in-sync
• G – CPUM x 2.5k writes/s, Snap COFW
• H- CPUM x 6k writes/s, Snap non-COFW
• Data logging % = Number of LUNs / Max LUNs * 10%
• One SP’s utilization will be the sum of the proportional contributions of each I/O type
• Use 4KB for IOPS and 512KB for Bandwidth
• I = CPUM x 1500MB/s read
• J = CPUM x 600MB/s write (cache on)
• Note: ASAP rebuilt, background verify, mirror syncs count against this number
• Example: CX4-960, RAID 5, 9000 IOPS, 2:1 R:W, 8KB –> 38% utilization
• 6000 read IOPs, 3000 write IOPs, 48MB/s read, 23MB/s write, RAID 5, CX4-960
• 6000/50000 + 3000/16000 + 48/1500 + 24/600 = 12% + 19% + 3.2% + 4.0% = 38.2% SP utilization

His formula is low

• Configuration polling
  • Pre-FLARE 26.31 configuration polling is another low priority internal function that affects utilization
  • Go to http://ipaddress/setup
  • Set Update Parameters in the Setup Menu and pick 300s. Update Interval to 300s.
  • Performance Interval (for statistics logging) is ok at 60. This does nothing compared to configuration polling and data logging.
  • Also include the -np (no poll) option whenever possible in CLI scripts
• Data logging
  • 7-10% differential comes from default data logging settings in older FLARE revisions with a lot of LUNs
  • Throughput was still unaffected because Analyzer threads run at a lower priority than I/O threads
  • Navisphere commands could be sluggish because they would be at the same level
  • Fix it by changing from 60/60 or 60/120 to 300/300.
  • Data logging poll rate is the lower of the two.
  • This will signficantly reduce pre-FLARE 29 utilization
• Navisphere operations, especially without -np (no pool)
• Background verify, rebuild, LUN migration, zeroing operations
• Snap, Clone, Mirror, SAN Copy overhead
• Disk or bus bottlenecks
• Heavy flushing

His formula is too high

• Coalesced backend writes
• Pre-fetch
• Nature of the load

In FLARE 26.31, FLARE 28, FLARE 29, FLARE 30

• Delta polling was introduced in FLARE 28 and back-revved to FLARE 26.31
• Significantly reduces Navisphere overhead
• FLARE 30, CLI commands without -np are given more processor time
• FLARE 29, data logging utilization has been reduced 80%
• FLARE 30 introduces fully provisioning virtual LUNs in pools of storage (thick LUNs)
• H6099 document
• NDU now uses % PrivilegedTime not % Processor Time as shown by Analyzer, 65% is safe (instead of 50%).

What will happen with SP utilization in the presence of EMC Flash Cache?

• 64KB is the base element for analysis for migration into Flash Cache
• There is a considerable amount of promotions (HDD > EFD) that will cost SP utilization. After the bulk of those initial promotions occur, it will be about 8-10% increased SP utilization for Flash Cache after warmup.

• Cannot directly perform Vmotion since storage is not shared
• Must first perform storage Vmotion

Vmotion with VPLEX

• Enables direct Vmotion between data centers
• Storage Vmotion is no longer required
• Replicate the data once then move the VMs at will

Use Cases

• Data Center Load Balancing
  • Optimize resources across several data centers
• Disaster Avoidance and Data Center Maintenance
  • Evacuate data center ahead of a probable disaster
  • Move applicatoin to remote data center to perform maintenance on local data center
• Zero-downtime Data Center moves
  • Move VMs and data to new data center then decommission old data center

Three Basic Configurations

• Common configurations
  • Maximum supported distance, 100km (with 5ms latency)
  • ESX hosts in both data centers have common IP subnets (stretched layer 2 network)
  • ESX servers can participate in local HA and DRS-enabled clusters
• VMFS volume built on a VPLEX distributed device
• VMFS volume is then shared between ESX servers in two locations
• Scenario 1 (distributed device)
  • Best practice
  • Continuous data protection and transparently protects against storage failures in either location
  • Continuous IO on biased cluster after WAN link failure
  • Continuous IO on biased cluster after non-biased site failure
  • Suspend IO on non-biased cluster after biased site failure
• Scenario 2 (built on remote device)
  • Not highly available, only good for temporary use when VM must move immediately
• Scenario 3 (temporary distributed device)
  • Storage Vmotion to a distribute device while in transit to the remote site
  • Then Storage Vmotion back to local storage in the remote site
  • Do this to regain some array functionality that VPLEX might not have

Failure Cases

• N+1 configuration handles director failures transparently
• Any WAN or remote cluster failure while Vmotion is in progress simply results in Vmotion being aborted

Rule-set Best Practices

• Manage your rule-sets very carefully
  • Be aware of which cluster will win in the event of a failure
• Place related VMs on the same data store so that they will move together
• For any given data store, move all VMs at the same time
• For the most critical applications, dedicate a data store to the VM

The iSCSI Story

• Transport SCSI over standard Ethernet
• Reliability through TCP
• SCSI has limited distance, iSCSI extended the distance

Non-Ethernet Convergence Options

• Infiniband
  • Used broadly for High Performance Computing (HPC) environment
  • Low cost and ultra-low latency geared for server to server cluster
  • Separate use from general network (Ethernet) or storage (FC or Ethernet)
• PCIe
  • Extension of the server bus to an I/O aggregation box
  • Not a standard, small players
  • Still using Etherhet and FC network and storage from an aggregation box

Maturation of 10Gb Ethernet

• Allows replacement of n x 1Gb with must smaller number of 10Gb adapters
• Single network allows for easier mobility for virtualization/cloud deployments
• Simplifies server, network and storage infrastructure

Standards

• 40Gb and 100Gb Ethernet (IEEE) standards will be completed in June 2010
• 16Gb FC (T11) standard is targeted for completion at the end of 2010
• 32Gb is in the works
• Server Adoption of FC ~3+ years, of Ethernet ~5+ years

Protocols and Standards
Fibre Channel over Ethernet

• Developed by T11, International Committee for Information Technology Standards (INCITS) T11 Fibre channel Interfaces Technical Committee
• FC-BB-5 standard ratified in June 2009

Converged Enhanced Ethernet

• Developed by IEEE Data Center Bridging (DCB) Task Group
• Commonly referred to as Lossless Ethernet
• IEEE standards targeting ratification mid-2010

iSCSI and FCoE Framing

• iSCSI is SCSI functionality transported using TCP/IP for delivery and routing
• FCoE is FC frames encapsulated in Layer 2 Ethernet frames over Lossless Ethernet

FCoE Frame Formats

• Ethernet frames give a 1:1 encapsulation of FC frames, no segmenting FC frames across multiple Ethernet frames

FC-BB-6

• Next step
• Not required for multi-hope FCoE or other current deployments
• Likely to support point-to-point configuration which allows two FCoE devices to communicate without going through an FCF (or switch)
• Today initiator cannot talk directly to a target without a switch in between. FC-BB-6 is investigating this.

Lossless Ethernet

• IEEE 802.1 Data Center Bridging (DCB) is the standards task group
• Converged Enhanced Ethernet (CEE) is the industry consensus term
• Link level enhancements (Priority Flow Control, Enhanced Transmission Selection, Data Center Bridging Exchange Protocol) are shipping in products today
  • PAUSE and Priority Flow Control
    • Classic 802.3x PAUSE is rarely implemented since it stops all traffic
    • New PAUSE known as PFC that can halt traffic according to priority tag while allowing traffic at other priority levels to continue. This creates lossless virtual lanes.
  • Enhanced Transmission Selection
    • Maintain low latency treatment of certain traffic classes
  • Data Center Bridging Exchange Protocol
    • Auto-negotiation for devices as they determine the link parameters
• The CEE cloud or DCB-enabled LAN is only for the portion of your network that requires lossless Ethernet

Beyond Link Level

• End-to-end
• Congestion notification
  • IEEE 802.1Qau ratified
  • Allows a switch to notify attached ports to slow down transmission due to heavy traffic
• Layer 2 multipathing
  • IETF TRILL – Transparent Interconnection of Lots of Links
  • Used with STP to provide more efficient bridging and bandwidth aggregation
  • Focuses on bridging capability that will increase bandwidth by allowing and aggregating multiple network paths

Solution Evolution
iSCSI

• iSCSI was >15% of revenue ($1.8B in 2009) and >20% capacity in SAN market in 2009
• 10Gb iSCSI solutions are available
• iSCSI natively routable (IP)
• iSCSI solutions are much smaller scale than FC

FCoE

• FCoE with direct attach of server to Converged Network Switch at top of rack or end of row
• Tightly controlled in 2009
• First solutions are with FcoE aware Ethernet switch (FIP snooping)
• Blade > FCoE switch > FC switch > Storage

Rack Area Network (RAN)

• Cisco UCS
• Vblock
• VCE (virtual computing environment)

Timeline

• End of year, you will see FCoE native in CX and DMX

• Memory buffer
  • Buffers hold index of all locations
  • Buffers incoming writes
  • Buffer resiliency
    • Power capacitors maintain power to the buffer in the event of system power failure
    • Contents are then written to the persistent store if power fails
• Pages
  • Cells are addressed by pages
  • 73GB and 200GB use 4KB pages
  • 400GB use 16KB pages
  • Page contents are contiguous address space, like SP cache pages
  • Two 2KB IO in a 4KB flash page but must be contiguous WRT LBA
• Blocks
  • NAND storage is mapped like a filesystem
  • Pages are grouped together into blocks
    • Not to be confused with SCSI or filesystem blocks
    • Multiple page sin a block jumpled together
    • Addresses of pages in a block do not have to be contiguous
  • Writes to NAND are done at block level
  • Block images are held in buffer until the block is full, then written to previously erased block on disk
  • There must be an erased block available for the write
• Channels
  • Paths to physical devices (chips)
  • Flash drives have multiple channels, discrete devices can be read from or written to simultaneously
  • Large I/O is striped across the channels

Page States

• Flash as Mapped Device
  • Workload can affect page state
  • Page state can affect availability of blocks
  • Availability of free (erased) blocks determines write performance
• Valid state: contains good data (referenced by host and flash)
• Invalid state: contains stale data
• Erased state: block is not in use
• Pages become randomized due to random writes
• Valid or invalid if referenced by flash meta data
• For example
  • A file that occupied two blocks on the chip gets written to
  • The first block gets written to the buffer and the block in the NAND gets marks as invalid

Reserve Capacity

• Some percentage of capacity is reserved and not included as user addressable capacity
• The capacity will be used to provide ready blocks for incoming writes
• Sustained heavy writes can saturate a Flash drive
• Now the drive will need to perform erase operations in idle cycles

Erasing Blocks

• The drive will erase blocks during idle periods
• To be erased, a block must have all invalid pages
  • Every valid page in a block must first be written to another block
  • That requires additional activity
    • Read in pages to buffered block
    • Erased old locales in NAND
    • Write out consolidated block to NAND
    • Basically this is defragmentation (housekeeping)

Consolidation

• Do flash drives slow down over time?
• Free space is a factor but so is time because it gets more and more fragmented over time
• Total capacity utilization can affect the response time of sustained writes
  • Higher capacity utilization results in more valid pages in each block
  • Over time, distribution of valid pages becomes more random and capacity utilization increases
  • If blocks have a high percentage of valid pages, it is more difficult to consolidate and erase a block
  • The drive therefore needs more time to do housekeeping

Issues

• >20% random write workload can have pretty significant affect on flash drive importance

Backfill

• Small writes and backfill, aka write amplification
• Writing an I/O smaller than the page requires read-modify-write
• This therefore doubles the workload on the drive
• This makes 73GB and 200GB flash drives better as they use 4KB page sizes and don’t suffer from this penalty as much as 400GB drives do with 16KB page sizes

Flash and Write Cache

• Original guidance: flash does not need SP cache
• New guidance: flash can help SP cache in many cases
• Experience: many uses of flash + SP cache in the field
• OK to use the SP cache for flash drives now and is a benefit in many cases

Best Practices

• Best use
  • High random read rates
  • Smaller I/O
  • I/O patterns that are not optimal for cached FC implementations
• Databases: 4-15 flash drives typical
  • Indexes and busy tables
    • Biggest disk-for-disk increase in ready-heavy tables (10-20x)
  • Temp space
    • But turn on SP write cache because of the write/re-read/write nature of temp databases
  • Some clients using Flash for write-heavy loads
    • Use SP cache for better response time
    • Flash flushes cache faster
• Really big databases are a little different
  • Up to 30 flash drives
  • These bypass SP write cache to maximize write throughput
• Oracle ASM 11gR2
  • Users can differentiate groups as FAST, AVERAGE, SLOW
• Messaging (Exchange, Notes)
  • Database to flash and all users benefit
  • Use R5 for Exchange on flash
    • Turn on SP write cache
    • Writes flush to R5 on flash faster than R10 on FC
    • Reads are likely better distributed than from R10 on flash
    • Flash rebuilds faster than FC and impact is less
• Ok to use
  • Databases
    • Oracle Flash Recovery: SATA do fine here, more economical
    • Redo logs: FC is sufficient, cost less
    • Archive logs: FC even SATA do fine
  • Media
    • Editing configurations are the best fit for flash in media
    • Some advantage to multi-stream access
    • FC will give more predictable write performance at a micro level due to flash’s internal structure
  • Any time power/cooling is an issue

• Storage federation platform that extends storage beyond the boundaries of the data center
• Distributed, peer-to-peer, fault-tolerant storage system

What problem does it solve?

• Simultaneous access to storage devices from two distinct locations
• Planned data mobility within, across and between data centers

What’s unique?

• Clusters can scale out and scale up
• Distributed coherent cache enabling distributed storage access
• Designed for asynchronous distances and multiple clusters

VPLEX Director

• 8x 8Gb/s FC front-end
• 8x 8Gb/s backend
• Dual quad-core processors (2.33Ghz)
• 32GB memory, about 25GB for cache
• 4x 8Gb/s FC com ports (intra- and inter-cluster)
• 4x 1Gb/s Ethernet com ports (inter-cluster)
• Standard EMC hardware
• GeoSynchrony is the operating system
• Looks like a VMAX engine

VPLEX Cluster

• Rack containing 1, 2, or 4 engines
• Plus a management server (and standby)
• Intra-director fibre channel switches
• Power and battery backup for each engine
• Cabling for management and intra-cluster com
• VPLEX Local is one cluster
• VPLEX Metro is two clusters

Major Subsystems

• Front-end – storage view
• Cache – distributed coherent cache
• Device Virtualization – virtual volumes
• Back-end – storage volumes

Lease Rollover

• Can be used to migrate data from one array to a new array
• Import new storage array/volumes
• Production volumes are mirrored
• Original array can be pulled out while the virtual devices now point to the new array

Application Migration

• Enable application at the remote site
• Cache activity now functions at source and remote site
• Disable application at source site
• Cache activity migrates to the remote site

Cluster failure and cluster partition

• Cluster failure and partition is hard
  • What does A know and when does it notice it can talk to B any more?
  • Easy if B’s really dead (then it can’t mess us up), but what if it’s still alive?
  • How do A and B decide what to do?
• Cluster Bias
  • Each distributed volume has a bias that favors one site over another
• What happens to all the writes to the other side?

A Closer Look at Distributed Mirrors

• Each disributed mirror has two dirty region logs
  • One records writes that couldn’t be committed locally
  • One records writes that couldn’t be committed to the remote site
• Each distributed mirror also has a bias and timeout for applying that bias

Failure of cluster without bias

1. IO goes to cluster A and B
2. Something bad happens to A. A can’t talk to B and thus starts a timer. IO pauses briefly
3. The timer expires, IO resumes for hosts at A. Writes are logged to the DRL.
4. Eventually B is repaired and brought back online. A and B reconnect.
5. A begins to resynchronze B based on DRL. Distributed mirror and cache coherency present A’s version of the data
6. Resynchronization finishes, DRLs are now empty

Failure of cluster with bias

1. IO goes to cluster A and B
2. B can’t talk to A. IO pauses indefinitely.
3. Admin instructs B to resume IO, and starts passive application at B. Writes are logged in B’s DRL
4. A is repaired and brough back online. And B reconnect. A notices B continued in spite of bias settings.
5. B begins to resynchronze A based on DRL. Distributed mirror and cache coherency present B’s version of the data
6. Resynchronization finishes, DRLs are now empty

3 Grouping Objects

• Storage groups: symaccess -sid 1234 create -type storage -name mystorage -devs 050:053
• Port groups: symaccess -sid 1234 -type port -name myports -dirport 7e:0, 8:e0
• Initiator groups: symaccess -sid 1234 create -type initiator -name myinit -file initiator_list

Must define a masking view

• Associates devices in the storagr group to the port groups to the intiator group
• Uses dynamic lun addressing, same lun address for all paths
• symaccess -sid 1234 create view -name dbserver -storgrp mystorage -portgrp myports -initgrp myinit
• Simplifies updates/changes

AccessLogix Database

• ACLX gatekeeper device (instead of the VCM database)

Limited symmask Compatibility Mode

• Both the VCM and ACLX database contain masking entires, resides in SFS, accessed through gatekeeper device
• However, data structures are completely different
• VCM database entries are based on one-to-one relationship between initiators and front-end ports
• ACLX database entries are based on many-to-many relationship between groups of initiators and ports
• Compatibility mode will take the traditional VCM symmask command but it still convert it to the auto-provisioning groups above but with one entry in each group
• Restrictions: once you start using symmask, you have to continue using symmask. If you use symaccess, symmask will no longer work because symmask is looking for one-to-one relationship
• Best practice is to transition over to auto-provisioning groups

Storage Templates

• Define capacity, protection, drive type, and other criteria
• Specify template rather than specific devices when initially provisioning storage or adding new capacity
• If the devices do not exist, it can optionally create the device