Computer Techno

Cloud storage is for blocks too, not just files

2017-10-24T14:24:00.000+08:00

One of the misconceptions about cloud storage is that it is only useful for storing files. This assumption comes from the popularity of file-sharing services and the fact that it’s difficult to conceive of data objects that aren’t files because that’s what people use.

But Windows Azure Storage works very well for storing fingerprints and fingerprints definitely are not files—they are logical packages of contiguous blocks of data. Blocks are an addressing mechanism that operating systems use to calculate where to put data to maintain system performance. CiS systems exchange blocks, not files, with servers. On the cloud side, CiS systems exchange objects with Windows Azure Storage—and fingerprints, chock full of blocks, are the objects used by the Microsoft HCS solution. Just as the operating system translates files into blocks in order to store them on storage devices, CiS translates blocks into fingerprints so they can be stored in the cloud.

Source of Information : Rethinking Enterprise Storage

A big breakthrough: Cloud snapshots

2017-10-23T14:13:00.000+08:00

The Microsoft HCS solution incorporates elements from backup, dedupe, and snapshot technologies to create a highly automated data protection system based on cloud snapshots. A cloud snapshot is like a storage snapshot but where the snapshot data is stored in Windows Azure Storage instead of in a storage array. Cloud snapshots provide system administrators with a tool they already know and love—snapshots—and extend them across the hybrid cloud boundary.

Fingerprints in the cloud
The data objects that are stored as snapshots in the cloud are called fingerprints. Fingerprints are logical data containers that are created early in the data lifecycle when data is moved out of the input queue in the CiS system. While CiS systems store and serve block data to servers, they manage the data internally as fingerprints.

Just as backup processes work by copying newly written data to tapes or disk, cloud snapshots
work by copying newly made fingerprints to Windows Azure Storage. One of the biggest differences between backup and cloud snapshots is that backup transforms the data by copying it into a different data format, whereas cloud snapshots copy fingerprints as-is without changing the data format. This means that fingerprints in Windows Azure Storage can be directly accessed by the CiS system and used for any storage management purpose.

Cloud snapshots work like incremental-only backups insofar that fingerprints only need to be uploaded once to Windows Azure Storage. Replication services in Windows Azure Storage makes multiple copies of the data as protection against failures. With most backup systems, there are many different backup data sets that need to be tracked and managed, but with cloud snapshots, there is only a single repository of fingerprints. In addition, there is no need to create synthetic full tapes because all the fingerprints needed to be recovered are located in the same Windows Azure Storage bucket.

Scheduling cloud snapshots
IT teams can flexibly configure their CiS systems to perform automated cloud snapshots to meet a broad range of requirements. Unlike tape backup systems that necessarily tie data expiration to tape rotation schedules, cloud snapshots can be assigned any expiration period. For instance, if the IT team decides they want to keep all cloud snapshot data for a minimum of three months, they can do it without having to worry about which tapes to use. Also, if the IT team wants to upload data more frequently, they can run cloud snapshots several times a day.

Efficiency improvements with cloud snapshots
Cloud snapshots eliminate tape problems and operator errors because there are no tapes to manage, lose, or go bad. No tapes need to be loaded for the next backup operation, no tapes are transferred off site, there are no tape names and labels to worry about, and no courier services need to be engaged. The arcane best practices that were developed for tape backup no longer apply to cloud snapshots. This is an enormous time saver for the IT team and removes them from the drudgery of managing tapes, tape equipment, and backup processes. Data protection with cloud snapshots also eliminates the need to make full or synthetic full tapes. The incremental-only approach of cloud snapshots means that a minimal amount of data is copied and transferred. In addition, the fact that data is deduped on-premises before it is snapshotted means the amount of data that is uploaded is minimized.

Comparing cloud snapshots
The biggest difference between cloud snapshots with the Microsoft HCS solution and other backup products is the integration with Windows Azure Storage. Cloud snapshots improve data protection in three important ways:

1. Off-site automation. Cloud snapshots automatically copy data off site to Windows Azure Storage.

2. Access to off-site data. Cloud snapshot data stored off site is quickly accessed on premises.

3. Unlimited data storage and retention. The amount of backup data that can be retained on Windows Azure Storage is virtually unlimited.

Remote replication can be used to enhance disk-based backup and snapshot solutions by automating off-site data protection. The biggest difference between cloud snapshots and replication-empowered solutions is that replication has the added expense of remote systems and facilities overhead, including the cost of managing disk capacities and replication links.

Remote office data protection
Cloud snapshots are also effective for automating data protection in remote and branch offices (ROBOs). These locations often do not have skilled IT team members on site to manage backup, and as a result, it is common for companies with many ROBOs to have significant gaps in their data protection.

Installing the Microsoft HCS solution in ROBO locations allows the IT team to completely automate data protection in Windows Azure Storage. This highlights another important architectural advantage—the many to one (N:1) relationship of on-premises locations to cloud storage. This design makes it possible for a Microsoft HCS solution at a corporate data center to access data from any of the ROBO locations. In addition, alerts from CiS systems running in the ROBOs can be sent to the IT team so they can remotely troubleshoot any problems that arise.

The role of local snapshots
CiS systems also provide local snapshots that are stored on the CiS system. Although local and cloud snapshots are managed independently, the first step in performing a cloud snapshot is running a local snapshot. In other words, all the data that is snapped to the cloud is also snapped locally first. The IT team can schedule local snapshots to run on a regular schedule—many times a day and on demand.

Looking beyond disaster protection
Snapshot technology is based on a system of pointers that provide access to all the versions of data stored by the system. The Microsoft HCS solution has pointers that provides access to all the fingerprints stored on the CiS system and in Windows Azure Storage.

The fingerprints and pointers in a Microsoft HCS solution are useful for much more than disaster protection and accessing point-in-time copies of data. Together they form a hybrid data management system that spans the hybrid cloud boundary. A set of pointers accompanies every cloud snapshot that is uploaded to Windows Azure Storage, referencing the fingerprints that are stored there. The system of pointers and fingerprints in the cloud is a portable data volume that uses Windows Azure Storage for both protection and portability.

This hybrid data management system enables additional data and storage management functions beyond backup and disaster recovery. For example, data tiering and archiving both take advantage of it to manage data growth and drive storage efficiency.

Source of Information : Rethinking Enterprise Storage

For the love of snapshots

2017-10-22T14:11:00.000+08:00

Snapshot technology is an alternative to backup that was first made popular by NetApp in their storage systems. Snapshots are a system of pointers to internal storage locations that maintain access to older versions of data. Snapshots are commonly described as making point-in-time copies of data. With snapshots, storage administrators are able to recreate data as it existed at various times in the past.

Snapshot technology is widely appreciated by IT teams everywhere for having saved them innumerable hours that they would have spent restoring data from backup tapes. It’s no wonder that snapshot technology has become a key element of storage infrastructures and is one of the most heavily utilized features on most business-class storage systems.

While IT teams have largely replaced backups with snapshots for restoring historical versions of data, the two technologies are often used together in backup scenarios. Snapshots are used to capture updates to data and then backup processes capture the data from the snapshot. This keeps backups from interfering with active production applications and their data.

One problem with snapshots is that they consume additional storage capacity on primary storage that has to be planned for. The amount of snapshot data depends on the breadth of changed data and the frequency of snapshots. As data growth consumes more and more capacity the amount of snapshot data also tends to increase and IT teams may be surprised to discover they are running out of primary storage capacity. A remedy for this is deleting snapshot data, but that means fewer versions of data are available to restore than expected.

In many cases, that may not be a huge problem, but there could be times when not being able to restore previous versions of data could cause problems for the IT team. Otherwise, the ease that snapshot capacity can be returned to free space depends on the storage system and may not be as simple as expected.

Source of Information : Rethinking Enterprise Storage

Dedupe makes a big difference

2017-10-21T14:08:00.000+08:00

A breakthrough in virtual tape technology came when dedupe technology was integrated with VTLs. Like previous-generation VTLs, dedupe VTLs require backup software products to generate backup data, but the dedupe function eliminates redundant data from backup streams. This translates directly into backup storage capacity savings and makes them much more cost-competitive with tape systems. Not only that, but dedupe VTLs improve backup performance by simultaneously backing up a larger number of servers and by keeping more backup copies readily available online. Many organizations
happily replaced their tape backup systems with dedupe VTLs.

While dedupe VTLs have transformed backup for many IT teams, it has done relatively little to make disaster recovery easier. In most cases, tape copies still need to be made for off-site protection and the challenges of restoring data from tape are the same whether they were generated by tape drives or a dedupe VTL. However, like incremental-only backup solutions, some dedupe VTLs can also replicate data off site to another remote dedupe VTL, eliminating the need to make off-site tape copies—with the familiar caveats that remote replication adds additional systems and facilities costs as well as being more complicated to manage.

Dedupe variations: source and primary dedupe
Due to the success of dedupe backup systems, most people associate dedupe technology with target-side backup protection, but the technology can be successfully implemented other ways as well. Source dedupe implements dedupe technology before sending it over the network to be backed up. The main advantage of source dedupe is that it consumes far less bandwidth to transfer data and the main disadvantage is that it takes more processing resources on the server where the dedupe process runs.

Primary dedupe is the application of dedupe technology for primary production data, as opposed to being limited to backup data. The main advantage of primary dedupe is that it reduces the amount of capacity consumed on primary storage— which tends to be the most expensive storage in the data center. The main disadvantage of primary dedupe is the performance impact of running dedupe on production data.

Source of Information : Rethinking Enterprise Storage

Incremental-only backup

2017-10-20T13:39:00.000+08:00

The incremental-only approach to backup makes a single full backup copy and thereafter makes incremental backup copies to capture newly written data. If synthetic full tapes are not made, this approach leads to horrendously long and troublesome restores because every tape that was ever made might be needed for recovery. This implies copies need to be made of every tape in case they fail and also requires them to be stored in different locations, which means it might be necessary to have multiple copies at each location to account for media failures and so on and so forth. (It’s funny what disaster paranoia will lead you to think about.)

That’s why backup vendors developed disk-based, incremental-only backup systems that automatically copy backup data from a backup system at one site to another system at a remote location. When a disaster happens at the primary site, a full recovery can be made at the remote site from backup data in the remote system.

Incremental-only backup solutions integrate database, replication, and backup software along with the redundant hardware systems and facilities overhead at the remote site. Like other disk-based backup systems, they have capacity limitations that restrict the amount of backup data that can be kept, requiring management diligence and planning. Incremental-only backup systems are effective for solving backup problems, but, if the IT team also wants to reduce the cost of storage, incremental-only systems probably don’t fit the bill.

Source of Information : Rethinking Enterprise Storage

Virtual tape

2017-10-19T13:35:00.000+08:00

The desire to reduce the dependency on tape for recovery gave rise to the development of virtual tape libraries (VTLs) that use disk drives for storing backup data by emulating tapes and tape hardware. Off-site storage of backup data is accomplished by copying virtual tapes onto physical tapes and transporting them to an off-site facility. This backup design is called disk-to-disk-to-tape, or D2D2T—where the first disk (D) is in file server disk storage, the second disk (D) is in a virtual tape system, and tape refers to tape drives and media.

VTLs significantly improve the automation of backup processes and provide good backup performance, but are more expensive than tape backup systems. Because the storage capacity of virtual tape products is limited, it might not be possible to backup as many servers or retain as much backup data as desired. For cost, capacity, and performance reasons, VTLs were mostly used in niche environments until dedupe technology was integrated with them and made them more widely applicable.

Source of Information : Rethinking Enterprise Storage

The many complications and risks of tape

2017-10-18T10:47:00.000+08:00

Magnetic tape technology was adopted for backup many years ago because it met most of the physical storage requirements, primarily by being portable so that it could be transported to an off-site facility. This gave rise to a sizeable ecosystem of related backup technologies and services, including tape media, tape drives, autoloaders, large scale libraries, device and subsystem firmware, peripheral interfaces, protocols, cables, backup software with numerous agents and options, off-site storage service providers, courier services, and a wide variety of consulting practices to help companies of all sizes understand how to implement and use it all effectively.

Tape media
Tape complexity starts with its physical construction. In one respect, it is almost miraculous that tape engineers have been able to design and manufacture media that meets so many challenging and conflicting requirements. Magnetic tape is a long ribbon of multiple laminated layers, including a microscopically jagged layer of extremely small metallic particles that record the data and a super-smooth base layer of polyester-like material that gives the media its strength and flexibility. It must be able to tolerate being wound and unwound and pulled and positioned through a high-tension alignment mechanism without losing the integrity of its dimensions. Manufacturing data grade magnetic tapes involves sophisticated chemistry, magnetics, materials, and processes.

Unfortunately, there are many environmental threats to tape, mostly because metals tend to oxidize and break apart. Tape manufacturers are moving to increase the environmental range that their products can withstand, but historically, they have recommended storing them in a fairly narrow humidity and temperature range. There is no question that the IT teams with the most success using tape take care to restrict its exposure to increased temperatures and humidity. Also, as the density of tape increases, vibration during transport has become a factor, resulting in new packaging and handling requirements. Given that tapes are stored in warehouses prior to being purchased and that they are regularly transported by courier services and stored off-site, there are environmental variables beyond the IT team’s control—and that makes people suspicious of its reliability.

Tape’s metallic layer is abrasive to tape recording heads and constantly causes wear and tear to them. Over time the heads wear out, sometimes much faster than expected. It can be very difficult to determine if the problem is head wear, tape defects, or dirty tape heads. Sometimes the only remedy is to replace both the tape heads and all the tapes. The time, effort, and cost involved in managing
wear-and-tear issues can be a sizeable burden on the IT group with no possible return on that investment to the organization. Tape aficionados are very careful about the tapes they buy and how they care for them, but many IT leaders no longer think it is worthwhile to maintain tapes and tape equipment.

Media management and rotation
Transporting tapes also exposes them to the risk of being lost, misplaced, or stolen. The exposure
to the organization from lost tapes can be extremely negative, especially if they contain customer account information, financial data, or logon credentials. Businesses that have lost tapes in-transit have not only had to pay for extensive customer notification and education programs, but they have also suffered the loss of reputation.

Backup software determines the order that tapes are used, as well as the generation of tape names. Unfortunately, tapes are sometimes mislabeled which can lead to incomplete backup coverage, as well as making restores and recoveries more challenging. It sounds like a simpleproblem to solve, but when you consider that multiple tapes may have been used as part of a single backup job and that some tapes (or copies of tapes) are off site and cannot be physically checked, it turns out that there is not always a fast way to clear up any confusion.

Tape rotation is the schedule that is used by backup software to determine which tapes should be used for the next backup operation. If an administrator improperly loads the wrong tape in a tape drive, the backup software may not run, which means new data is not protected.

Conversely, the backup software may choose to overwrite existing data on the tape, making it impossible to recover any of it. A similar problem occurs when a backup administrator erroneously deletes tape records from the backup system’s database or erases the wrong tapes. Backup only works correctly when the database used to track data on tape accurately reflects the data that is recorded on tapes.

These sorts of problems are well-known to backup administrators and are more common that one might think. Backup administration and tape management tends to be repetitive, uninteresting work which sets the stage for operator oversights and errors. This is the reality of tape backup and it is why automated data protection with the Microsoft HCS solution from

Microsoft is such an important breakthrough. It removes the responsibility for error- prone processes from people who would rather be doing something else. When you look at all the problems with tape, it is highly questionable as an infrastructure technology. Infrastructures should be dependable above all else and yet, that is the consistent weakness of tape technology in nearly all its facets.

Synthetic full backups
An alternative to making full backup copies is to make what are called synthetic full copies, which aggregate data from multiple tapes or disk-based backups onto a tape (or tapes) that contains all the data that would be captured if a full backup were to be run. They reduce the time needed to complete backup processing, but they still consume administrative resources and suffer from the same gremlins that haunt all tape processes.

The real issue is why it should be necessary to make so many copies of data that have already been made so many times before. Considering the incredible advances in computing technology over the years, it seems absurd that more intelligence could not be applied to data protection, and it highlights the fundamental weakness of tape as a portable media for off-site storage.

Restoring from tape
It would almost be comical if it weren’t so vexing, but exceptions are normal where recovering from tape is concerned. Things often go wrong with backup that keeps it from completing as expected. It’s never a problem until it’s time to recover data and then it can suddenly become extremely important in an unpleasant sort of way. Data that was skipped during backup cannot be recovered. Even worse, tape failures during recovery prevents data from being restored.

Unpleasant surprises tend to be just the beginning of a long detour where restores are concerned. Fortunately, there may be copies from earlier backup jobs that are available to recover. Unfortunately, several weeks or months of data could be lost. When this happens, somebody has a lot of reconstruction work to do to recreate the data that couldn’t be restored.

One thing to expect from disaster recovery is that more tapes will need to be used than assumed. Another is that two different administrators are likely to vary the process enough so that the tapes they use are different—as well as the time they spend before deciding the job is done, which implies the job is never completely finished. Most people who have conducted a disaster recovery would say there was unfinished business that they didn’t have time to figure out. Their efforts were good enough—they passed the test—but unknown problems were still lurking.

Source of Information : Rethinking Enterprise Storage

The inefficiencies and risks of backup processes

2017-10-17T10:41:00.000+08:00

If cloud storage had existed decades ago, it’s unlikely that the industry would have developed
the backup processes that are commonly used today. However, the cloud didn’t exist, and IT teams had to come up with ways to protect data from a diverse number of threats, including large storms, power outages, computer viruses, and operator errors. That’s why vendors and IT professionals developed backup technologies and best practices, to make copies of data and store them off site in remote facilities where they could be retrieved after a disaster. A single “backup system” is constructed from many different components that must be implemented and managed correctly for backup to achieve its ultimate goal: the ability to restore the organization’s data after a disaster has destroyed it.

Many companies have multiple, sometimes incompatible, backup systems and technologies protecting different types of computing equipment. Many standards were developed over the years, prescribing various technologies, such as tape formats and communication interfaces, to achieve basic interoperability. Despite these efforts, IT teams have often had a difficult time recognizing the commonality between their backup systems. To many, it is a byzantine mess of arcane processes.

Technology obsolescence is another difficult aspect of data protection. As new backup storage technologies are introduced, IT teams have to manage the transition to those technologies as well as retain access to data across multiple technologies. This tends to be more problematic for long-term data archiving than backup, but it is a consideration that weighs on IT teams nonetheless.

Disaster recovery is the most stressful, complex undertaking in all of IT. Recreating replacement
systems from tape backups involves many intricate details that are very difficult to foresee and plan for. Doing this without the usual set of online resources is the ultimate test of the IT team’s skills—a test with a very high bar and no chance for a retry. Most IT teams do not know what their own recovery capabilities are; for example, how much data they could restore and how long it would take. When you consider how much time, money, and energy has been invested in backup, this is a sad state of affairs for the IT industry. Data growth is only making the situation worse.

Source of Information : Rethinking Enterprise Storage

Hybrid cloud storage architecture

2017-10-16T10:32:00.000+08:00

Hybrid cloud storage overcomes the problems of managing data and storage by integrating on-premises storage with cloud storage services. In this architecture, on-premises storage uses the capacity on internal SSDs and HDDs, as well as on the expanded storage resources that are provided by cloud storage. A key element of the architecture is that the distance over which data is stored is extended far beyond the on-premises data center, thereby providing disaster protection. The transparent access to cloud storage from a storage system on-premises is technology that was developed by StorSimple and it is called Cloud-integrated Storage, or CiS. CiS is made up of both hardware and software. The hardware is an industry-standard iSCSI SAN array that is optimized to perform automated data and storage management tasks that are implemented in software.

The combination of CiS and Windows Azure Storage creates a new hybrid cloud storage architecture with expanded online storage capacity that is located an extended distance from the data center.

Change the architecture and change the function
CiS performs a number of familiar data and storage management functions that are significantly transformed when implemented within the hybrid cloud storage architecture.

Snapshots
CiS takes periodic snapshots to automatically capture changes to data at regular intervals. Snapshots give storage administrators the ability to restore historical versions of files for end users who need to work with an older version of a file. Storage administrators highly value snapshots for their efficiency and ease of use—especially compared to restoring data from tape. The main limitation with snapshots is that they are restricted to on-premises storage and susceptible to the same threats that can destroy data on primary storage. Implementing snapshots in a hybrid cloud storage architecture adds the element of extended distance, which makes them useful for backup and disaster recovery purposes.

Data tiering
CiS transparently performs data tiering, a process which moves data between the SSDs and HDDs in the CiS system according to the data’s activity level with the goal of placing data on the optimal cost/performance devices. Expanding data tiering with a hybrid cloud storage architecture transparently moves dormant data off site to the cloud so it no longer occupies on-premises storage. This transparent, online “cold data” tier is a whole new storage level that is not available with traditional storage architectures, and it provides a way to have archived data available online.

Thin provisioning
SAN storage is a multitenant environment where storage resources are shared among multiple servers. Thin provisioning allocates storage capacity to servers in small increments on a first-come, first-served basis, as opposed to reserving it in advance for each server. The caveat almost always mentioned with thin provisioning is the concern about over-committing resources, running out of capacity, and experiencing the nightmare of system crashes, data corruptions, and prolonged downtime.

However, thin provisioning in the context of hybrid cloud storage operates in an environment where data tiering to the cloud is automated and can respond to capacityfull scenarios on demand. In other words, data tiering from CiS to Windows Azure Storage provides a capacity safety valve for thin provisioning that significantly eases the task of managing storage capacity on-premises.

Source of Information : Rethinking Enterprise Storage

Best practices or obsolete practices

2017-10-15T10:32:00.001+08:00

The IT team does a great deal of work to ensure data is protected from threats such as natural disasters, power outages, bugs, hardware glitches, and security intrusions. Many of the best practices for protecting data that we use today were developed for mainframe environments half a century ago. They are respected by IT professionals who have used them for many years to manage data and storage, but some of these practices have become far less effective in light of data growth realities. Some best practices for protecting data are under pressure for their costs, the time they take to perform, and their inability to adapt to change. One best practice area that many IT teams find impractical is disaster recovery (DR). DR experts all stress the importance of simulating and practicing recovery, but simulating a recovery takes a lot of time to prepare for and tends to be disruptive to production operations. As a result, many IT teams never get around to practicing their DR plans. Another best practice area under scrutiny is backup, due to chronic problems with data growth, media errors, equipment problems, and operator miscues. Dedupe backup systems significantly reduce the amount of backup data stored and help many IT teams successfully
complete daily backups. But dedupe systems tend to be costly, and the benefits are limited
to backup operations and don’t include the recovery side of the equation. Dedupe does not change the necessity to store data off-site on tapes, which is a technology that many IT teams would prefer to do away with. Many IT teams are questioning the effectiveness of their storage best practices and are looking for ways to change or replace those that aren’t working well for them anymore.

Doing things the same old way doesn’t solve new problems
The root cause of most storage problems is the large amount of data being stored. Enterprise storage arrays lack capacity “safety valves” to deal with capacity-full scenarios and slow to a crawl or crash when they run out of space. As a result, capacity planning can take a lot of time that could be used for other things. What many IT leaders dislike most about capacity management is the loss of reputation that comes with having to spend money unexpectedly on storage that was targeted for other projects. In addition, copying large amounts of data during backup takes a long time even when they are using dedupe backup systems. Technologies like InfiniBand and Server Message Block (SMB) 3.0 can significantly reduce the amount of time it takes to transfer data, but they can only do so much.

More intelligence and different ways of managing data and storage are needed to change the dynamics of data center management. IT teams that are already under pressure to work more efficiently are looking for new technologies to reduce the amount of time they spend on it. The Microsoft HCS solution discussed in this book is a solution for existing management technologies and methods that can’t keep up.

Source of Information : Rethinking Enterprise Storage

Solid State Disks under the covers

2017-04-08T17:51:00.000+08:00

SSDs are one of the hottest technologies in storage. Made with nonvolatile flash memory, they are unencumbered by seek time and rotational latencies. From a storage administrator’s perspective, they are simply a lot faster than disk drives. However, they are far from being a “bunch of memory chips” that act like a disk drive. The challenge with flash is that individual memory cells can wear out over time, particularly if they are used for low-latency transaction processing applications. To alleviate this challenge, SSD engineers design a number of safeguards, including metadata tracking for all cells and data, compressing data to use fewer cells, parity striping to protect against cell failures, wear-leveling to use cells uniformly, “garbage collecting“ to remove obsolete data, trimming to remove deleted data, and metering to indicate when the device will stop being usable. SSDs manage everything that needs to be managed internally. Users are advised not to use defrag or other utilities that reorganize data on SSDs. They won’t perform faster, but they will wear out faster.

Source of Information : Rethinking Enterprise Storage

Virtual systems and hybrid cloud storage

2017-04-07T17:49:00.000+08:00

IT teams use virtualization technology to consolidate, relocate, and scale applications to keep pace with the organization’s business demands and to reduce their operating costs. Hypervisors, such as ESX and ESXi from VMware and Hyper-V from Microsoft, create logical system images called virtual machines (VMs) that are independent of system hardware thereby enabling IT teams to work much more efficiently and quickly.

But virtualization creates problems for storage administrators who need more time to plan and implement changes. The storage resources for ESX and ESXi hypervisors are Virtual Machine Disk Format (VMDK) files, and for Hyper-V hypervisors, they are Virtual Hard Disk (VHD) files. While VMs are rapidly moved from one server to another, moving the associated VMDKs and VHDs from one storage system to another is a much slower process. VMs can be relocated from one server to another without relocating the VMDKs and VHDs, but the process of load balancing for performance usually involves shifting both VMs and VMDKS/VHDs. Data growth complicates the situation by consuming storage capacity, which degrades performance for certain VMs, and forces the IT team to move VMDKs/VHDs from one storage system to another, which can set off a chain reaction of VMDK/VHD relocations along the way. Hybrid cloud storage gracefully expands the capacity of storage, including VMDKs and VHDs, eliminating the need to move them for capacity reasons. By alleviating the pressures of data growth, hybrid cloud storage creates a more stable environment for VMs.

Source of Information : Rethinking Enterprise Storage

The constant nemesis: data growth

2017-04-06T17:48:00.000+08:00

IDC’s Digital Universe study estimates that the amount of data stored worldwide is more than doubling every two years, so it is no surprise that managing data growth is often listed as one of the top priorities by IT leaders. IT professionals have ample experience with this problem and are well aware of the difficulties managing data growth in their corporate data centers. Balancing performance and data protection requirements with power and space constraints is a constant challenge.

IT leaders cannot surrender to the problems of data growth, so they need a strategy that will diminish the impact of it on their organizations. The hybrid cloud storage approach leverages cloud storage to offload data growth pressures to the cloud. Storage, which has always had an integral role in computing, will continue to have a fundamental role in the transformation to hybrid cloud computing—for its primary functionality (storing data) as well as its impact on those responsible for managing it.

Source of Information : Rethinking Enterprise Storage

The transformation of enterprise storage with cloud storage services

2017-04-05T17:44:00.000+08:00

Storage has been an integral part of information technology from its inception and will continue
to be throughout the cloud computing transformation that is underway. That’s because all the data we create, use, and share has to be stored somewhere if it is to havemore than fleeting value. A lot of this data is stored in corporate data centers, but a rapidlygrowing percentage is being stored in the cloud.

Enterprise storage architectures will need to adapt to this reality and integrate with cloud storage. Just as cloud services have changed the ways we consume data, they will also change how we store, manage, and protect it. It is short-sighted to think of cloud storage merely as big disk drives in the sky when there is so much compute power in the cloud to do interesting things with it. If it is possible to find information needles in data haystacks using data analytics, it is certainly possible to discover new ways to manage all that data more effectively. For example, the implementation of erasure coding in Windows Azure Storage demonstrates how advanced error-correction technology can also be used to effectively manage cloud storage capacity.

But the advancements in enterprise storage won’t all be cloud-resident. In fact, many of the most important changes will occur in on-premises storage management functions that take advantage of hybrid cloud designs. The section “Change the architecture and change the function,” later in this chapter, examines how extending traditional storage architectures with the addition of cloud storage services makes familiar storage management functions much more powerful.

Source of Information : Rethinking Enterprise Storage

Data archiving with Azure SQL Database

2017-04-04T17:40:00.000+08:00

The data archive mechanism with Azure SQL Database generally will be an export from Azure SQL Database. Export creates a non–transactionally consistent copy of some or all database tables using the BACPAC format (schema and data). This BACPAC file can be stored in Azure blog storage and/or downloaded to an on-premises environment and imported when desired to either Azure SQL Database or SQL Server.

To obtain a transactionally consistent export (without using third-party tools), you must either ensure that no writes occur on the source database during the export or create a database copy (which is guaranteed to be transactionally consistent).

Microsoft provides a number of options for you to use to export your data to a BACPAC file. These are as follows:

 Using the Azure portal

 Using the Export Data Tier Application Wizard in Microsoft SQL Server Management Studio, version 13.0.11000 or higher (Download SQL Server Management Studio)

 Using the SQLPackage.exe command-line utility

 Using the Start-AzureSqlDatabaseExport PowerShell cmdlet, which can be combined with the Start-AzureSqlDatabaseCopy PowerShell cmdlet

Generally, for databases smaller than 200 GB, using the Azure portal is the simplest method. If the operation goes over 20 hours, it may be cancelled. If the export does not have a clustered index, it may fail if it takes longer than 6 to 12 hours.

Source of Information : Migrating SQL Server Databases to Azure

Data archiving with SQL Server in an Azure virtual machine

2017-04-03T17:39:00.000+08:00

The data archive mechanism with SQL Server in an Azure virtual machine generally will be a full SQL Server backup that you store for the length of time required. If you are managing your own backups, you will want to store your full database backups in Azure blob storage standard storage (or archive outside Azure). If you are not already using backup to URL to store your backups in Azure blob storage, you will want to copy from your attached disks in premium storage to Azure blob storage using standard storage.

If you are using managed backup and want to archive a full database backup for longer than 30 days, you will need to copy a full backup to a new storage location or periodically perform a full database backup (to URL) on a scheduled basis and store it in standard storage for the length of time required.

If you are using file-snapshot backup, you have another option to archive your database: you periodically can copy the backup snapshots for a given backup set to long-term storage. At any point in time, you can choose to restore to a new database when needed and create a traditional (streaming) SQL Server backup for long-term storage.

Source of Information : Migrating SQL Server Databases to Azure

Database backups with Azure SQL Database

2017-04-02T17:32:00.000+08:00

Database backups with Azure SQL Database are performed for you automatically, and you have no option to perform manual database backups. To protect your data and enable point-in-time restore services, SQL Database takes full backups every week, multiple differential backups every day, and log backups every five minutes. Backup files are stored in geo-redundant storage with read access (RA-GRS) to ensure backups’ availability for disaster recovery purposes. The first full backup is scheduled immediately after a database is created. After the first full backup, all backups are scheduled automatically and managed silently in the background. The exact timing of full and differential backups is determined by the system to balance overall load.

SQL Database provides a point-in-time restore self-service feature for all databases regardless of service tier, but with different restore points as follows:
 Basic: Any restore point within 7 days
 Standard: Any restore point within 14 days
 Premium: Any restore point within 35 days

Point-in-time restore (geo-restore) is built on top of the SQL Database automated backup system and enables you to restore an existing or deleted database to a new database as of a specified point in time. You can restore using the Azure portal, PowerShell, or the REST API. The walk-through below will demonstrate restoring a database to a point in time using the Azure portal.

When restoring an existing database, you can restore to the same logical server or to a different server. When restoring to the same server, you create a new database. Unlike SQL Server, you cannot restore over an existing database. As a result, you will have additional storage costs while both databases exist—unless you choose to delete the existing database before restoring from the backups.

Deleting a logical server also deletes all of its databases and their database backups, after which they cannot be restored regardless of the retention period discussed previously for the automated backups. To store data for longer than the period of time provided by your service tier, you will need to export data to a BACPAC file.

Source of Information : Migrating SQL Server Databases to Azure

SQL Server Managed Backup to Windows Azure

2017-04-01T17:14:00.000+08:00

You can use SQL Server Managed Backup to Windows Azure and have SQL Server manage and automate SQL Server backups to Azure blob storage. With this option, you specify the blob storage location and the retention period and optionally customize backup at the database level. The maximum backup size is 12.8 TB. Managed backup supports point-in-time restore during the retention period. By default, managed backups are enabled at the instance level, with the same retention period for all databases (including newly added databases). You can change the retention period on an individual database, create a custom backup schedule, or disable managed backup for an individual database using Transact-SQL. You also have the option to encrypt the database backups for additional security.

Managed backup was introduced with SQL Server 2014, and the functionality increased substantially with SQL Server 2016. For the purposes of this ebook, I will focus on SQL Server 2016.

With SQL Server 2016, if you do not specify a custom schedule, the type of backups scheduled and the backup frequency is determined based on the workload of the database.
 A full database for a database is taken in the following instances:

When managed backup is enabled for the first time
Log growth since the last full backup is 1 GB or more
One week has passed since the last full backup
The log chain is broken, such as through a manual backup

 A transaction log backup is taken in the following instances:

No log backup history can be found. This usually is true when SQL Server Managed Backup to Windows Azure is enabled for the first time.
The transaction log space used is 5 MB or larger.
The maximum time interval of two hours since the last log backup is reached.
Anytime the transaction log backup is lagging behind a full database backup. The goal is to keep the log chain ahead of full backup.

To use SQL Server managed backup, you create the following prerequisites:

 Create an Azure storage account.

 Create a container within the storage account.

 Generate a Shared Access policy and apply the Shared Access Signature (SAS) token to the Azure blob container.

 Create a SQL Server credential based on the SAS token.

CREATE CREDENTIAL WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = '' ;

After satisfying these prerequisites, you use Transact-SQL to interact with SQL Server Managed Backup to Windows Azure. There are 13 system stored procedures and functions for enabling, configuring, and monitoring SQL Server Managed Backup to Windows Azure. System functions are used to retrieve existing configuration settings, parameter values, and backup file information. Each of these system stored procedures and functions provides code examples for you.

Extended events are used to surface errors and warnings. Alert mechanisms are enabled through SQL Agent jobs and SQL Server Policy-Based Management.

PowerShell cmdlets are also available to configure SQL Server Managed Backup to Windows Azure. SQL Server Management Studio supports restoring backups created by SQL Server Managed Backup to Windows Azure by using the Restore Database task.

Source of Information : Migrating SQL Server Databases to Azure

SQL Server backup to URL

2017-03-31T17:11:00.000+08:00

With SQL Server backup to URL, you specify an Azure blob storage container as the destination for your full, differential, and transaction log backups, and you manage the backup process and the retention strategy. A primary reason to use SQL Server backup to URL is to avoid the cost of backing up to attached disks that use premium storage (although you always can back up to attached premium storage for performance and then archive to blob storage).

To use this backup mechanism, take the following steps:
 Create an Azure storage account.

 Create a container within the storage account to contain your backup files.

 Generate a Shared Access policy and apply the Shared Access Signature (SAS) token to the Azure blob container.

 Create a SQL Server credential based on the SAS token.
CREATE CREDENTIAL WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = '' ;

 Specify TO URL as an argument with your BACKUP DATABASE and BACKUP LOG commands.

BACKUP DATABASE AdventureWorks2016 TO URL = 'https://.blob.core.windows.net//AdventureWorks2016.bak;

Backup to URL is a strategy you will want to use to save money when your Azure virtual machine is using premium storage because SQL Server backup to URL uses standard storage. The maximum size for a single file using a SAS token is 200 GB, and the maximum backup size using a striped backup set is 12.8 terabytes (TB).

Note You can back up to a single backup with a maximum size of 1 TB using the legacy SQL Server 2014 syntax of WITH CREDENTIAL that utilizes the storage account access key rather than a SAS key. This syntax is being deprecated.

Source of Information : Migrating SQL Server Databases to Azure

User authentication with SQL Server in an Azure virtual machine

2017-03-30T17:09:00.000+08:00

The default authentication mode for SQL Server in an Azure virtual machine (and in a SQL Server on-premises installation) is Windows Authentication mode. With this mode, users must connect and be authenticated using an Active Directory user account. In this mode, SQL Server Authentication is disabled and the SA login account is also disabled. You can change the SQL Server Authentication mode to mixed mode authentication. Mixed mode enables both Windows Authentication and SQL Server Authentication. Windows Authentication always is available and cannot be disabled.

User authentication using mixed mode
If you do not join your Azure virtual machine to an Active Directory domain, you generally will use SQL Server Authentication for user authentication. SQL Server Authentication is less secure than Active Directory authentication for the following reasons:

 The encrypted SQL Server Authentication login password must be passed over the network at the time of the connection. Some applications that connect automatically will store the password at the client. These are additional attack points.

 If a user is an Active Directory domain user who has a login and password, the user must provide SQL Server login and password to connect. Keeping track of multiple names and passwords is difficult for many users. Having to provide SQL Server credentials every time one connects to the database can be annoying.

 SQL Server Authentication cannot use Kerberos security protocol.

However, SQL Server Authentication may be the best option for SQL Server in an Azure virtual machine in the following scenarios:

 If the SQL Server database users do not have Active Directory user accounts and cannot be authenticated by an Active Directory domain

 If the web-based applications accessing SQL Server in the virtual machine allow users to create their own identities

 If the software applications accessing the SQL Server instance use a complex permission hierarchy based on known SQL Server logins

To use SQL Server Authentication, you must enable mixed mode authentication and restart the SQL Server instance. When you do so, take the following steps to increase the security for your SQL Server instance:

 Use an account other than the SA login for SQL Server administration.
 Enforce password expiration.
 Enforce Windows password policies, including password length and complexity.
 Require users to change their password at their next login.

When configuring SQL Server in an Azure virtual machine using the Resource Manager deployment model, you have the option to have Azure configure SQL Server for mixed authentication mode when SQL Server is configured on the Azure virtual machine. When you do so, the default SQL Server login account and password are the same as the local administrator account for the virtual machine, although you can define a different account. Password policies are enforced, and the SA login account is not enabled.

If you do not change the authentication mode during the initial configuration of the SQL Server instance from the SQL Server default of Windows Authentication mode to mixed mode, you can use Windows remote desktop to connect to the virtual machine and then use SQL Server Management Studio to connect to SQL Server locally and change the authentication mode if desired.

User authentication using Windows Authentication mode
If you join your Azure virtual machine to an Active Directory domain, you can leave SQL Server in Windows Authentication mode and require your users to connect and be authenticated using Active Directory authentication. When a user connects through an Active Directory user account, SQL Server validates the account name and password using the Active Directory principal token in the operating system. This means the user identity is confirmed by Active Directory. SQL Server does not ask for the password and does not perform the identity validation. Active Directory authentication uses Kerberos security protocol, provides password policy enforcement with regard to complexity validation for strong passwords, provides support for account lockout, and supports password expiration.

In addition, using Windows Authentication mode enables you to manage authentication using groups rather than individual user accounts and avoid the proliferation of user identities across database servers.

Source of Information : Migrating SQL Server Databases to Azure

Configuring and securing connections to Azure SQL Database

2017-03-29T17:07:00.000+08:00

With Azure SQL Database, Azure configures the network for you and by default restricts all access using Azure SQL Database firewall rules, as discussed in Chapter 3, “Getting started with an Azure SQL Database.” If a firewall rule does not exist, Azure will reject all connection attempts from IP addresses that have not been whitelisted explicitly.

 Use database-level firewall rules in conjunction with contained users (discussed below) whenever possible to make your database more portable.

 Use server-level firewall rules when you have many databases that have the same access requirements and you don’t want to spend time configuring each database individually.

Additionally, Azure SQL Database requires encrypted connections at all times while data is “in transit” to and from the database. In your application’s connection string, you must specify parameters to encrypt the connection and not to trust the server certificate (this is done for you if you copy your connection string out of the Azure portal). If you do not, the connection will not verify the identity of the server and will be susceptible to “man-in-the-middle” attacks. For the ADO.NET driver, for instance, these connection string parameters are Encrypt=True and TrustServerCertificate=False. For more information, see Azure SQL Database Connection Encryption and Certificate Validation. The code block below shows a sample ADO.NET connection string.

Server=tcp:[your_sql_database_server_name_here].database.windows.net,1433;Database=[your_sql_database_name_here]>;User ID=[your_username_here]@[your_sql_database_server_name_here];Password={your_password_here};Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

Unlike SQL Server in an Azure virtual machine, with Azure SQL Database you do not have to secure the operating system itself. However, all Azure subscription administrators have access to the SQL Database instance, and you should limit subscription administrators.

Source of Information : Migrating SQL Server Databases to Azure

Configuring and securing connections to SQL Server in an Azure virtual machine

2017-03-28T17:05:00.000+08:00

With an Azure virtual machine, you have several options to restrict and secure connections to your SQL Server instance. The virtual network for your Azure virtual machine is a logical isolation of the Azure cloud dedicated to your subscription. You can fully control the IP address blocks, DNS settings, security policies, and route tables within this network, similarly to how you use these mechanisms to control your on-premises network. You can also segment your virtual network into subnets to further control access to the virtual machines on your virtual network that host your SQL Server instances.

In addition, you can connect the virtual network to your on-premises network using one of the connectivity options available for Azure virtual machines. In essence, you can expand your on-premises network to your Azure virtual network, delivering the benefit of enterprise scale that Azure provides. Finally, you can set up and configure an Azure virtual machine as a domain controller and join your SQL Server virtual machine to this Azure domain controller. This Azure domain controller can be federated with Azure Active Directory, be federated with your on-premises Active Directory, or be a controller within your existing on-premises Active Directory. A full discussion of your options and best practices for configuring a secure connection to your on-premises network is beyond the scope of this ebook. If you do so, you can join your virtual machine to your on-premises Active Directory environment and use Windows user accounts for authentication.

In addition to configuring and securing the virtual network to which your Azure virtual network is connected, you should take these security steps:
 Use a unique local administrator account for your virtual machine that does not have a name of Administrator.

 Use complex strong passwords for all of your accounts, Windows and SQL Server. For more information about how to create a strong password, see the Create Strong Passwords article in the Safety and Security Center.

 Enable encrypted connections for your SQL Server instance and configure your SQL Server instance with a signed certificate.

 Use Windows firewall rules to control database engine access.

 If your virtual machines should be accessed only from a specific network, use network security groups (NSGs) to control traffic and restrict access to certain IP addresses or network subnets. An NSG contains access control rules that allow or deny traffic based on traffic direction, protocol, source address and port, and destination address and port.

Source of Information : Migrating SQL Server Databases to Azure

Migrating a compatible SQL Server database to SQL Database

2017-03-27T17:00:00.000+08:00

To migrate a compatible SQL Server database to Azure SQL Database, Microsoft provides several migration methods for various scenarios. The method you choose depends upon your tolerance for downtime, the size of your SQL Server database, and the speed and quality of your connection to the Microsoft Azure cloud.

If you can afford some downtime or you are performing a test migration of a production database for later migration, consider one of the following three methods:

 SSMS Migration Wizard: For small to medium databases, migrating a compatible SQL Server 2005 or later database is as simple as running the Deploy Database to Microsoft Azure SQL Database Wizard in SQL Server Management Studio.

 Export to BACPAC file and then Import from BACPAC file: If you have connectivity challenges (no connectivity, low bandwidth, or timeout issues) and for medium to large databases, use this manual method. With this method, you export the SQL Server schema and data to a BACPAC file and then import the BACPAC file into SQL Database using either the Deploy Database to Microsoft Azure SQL Database Wizard in SQL Server Management Studio or the SqlPackage command-prompt utility.

 Use BACPAC and BCP together: Use a BACPAC file and BCP for very large databases to achieve greater parallelization for increased performance, albeit with greater complexity. With this method, migrate the schema and the data separately.

Export the schema only to a BACPAC file.
Import the schema only from the BACPAC file into SQL Database.
Use BCP to extract the data into flat files and then parallel load these files into Azure SQL Database.

Source of Information : Migrating SQL Server Databases to Azure

Determining and resolving Azure SQL Database V12 compatibility issues

2017-03-26T16:58:00.000+08:00

To export to a BACPAC file, the database must be compatible with Azure SQL Database V12. The export will fail if the database is not compatible. With SQL Database V12, there are very few remaining compatibility issues other than server-level and cross-database operations. Databases and applications that rely on partially supported or unsupported functions will need some reengineering to fix these incompatibilities before the SQL Server database can be migrated. Review these partially supported or unsupported functions before migrating the SQL Database.

The tooling to detect and fix these incompatibilities is improving, but it still requires some manual intervention in many cases. The list below describes the tooling as it currently exists, including new tooling coming out in preview mode in May 2016.

 The Microsoft SQL Server 2016 Upgrade Advisor preview (UA): This stand-alone tool that currently is in preview will detect and generate a report of SQL Database V12 incompatibilities. This tool does not yet have all of the most recent compatibility rules (but will soon). If no errors are detected, you can continue and complete the migration to SQL Database. If errors are detected, you must use another tool to fix any detected compatibility issues. SSDT is the recommended tool.
As mentioned previously, UA will be updated in late April or early May 2016 and become Data Migration Assistant (DMA). DMA will contain the most recent compatibility rules and will, over time, enable you to fix incompatibilities directly in the tool. This functionality is still coming online, so stay tuned.

 SQL Server Data Tools for Visual Studio (SSDT): For Azure SQL Database V12, the best tool today is SSDT because it uses the most recent compatibility rules to detect SQL Database V12 incompatibilities. To use this option, download the newest version of SSDT. If incompatibilities are detected, you can fix detected issues directly in SSDT.

 SqlPackage: SqlPackage is a command-prompt utility that tests for and, if found, generates a report containing detected compatibility issues. If you use this tool, make sure you use the most recent version (follow the link provided) to use the most recent compatibility rules. If errors are detected, you must use another tool to fix any detected compatibility issues. SSDT is the recommended tool.

 The Export Data-tier Application Wizard in SQL Server Management Studio: This wizard will detect and report errors to the screen. If no errors are detected, you can continue and complete the migration to SQL Database. If errors are detected, you must use another tool to fix any detected compatibility issues. SSDT is the recommended tool.

 SQL Azure Migration Wizard (SAMW): For Azure SQL Database V11, the best tool was SAMW. This is a community-supported CodePlex tool that has not been updated fully for Azure SQL Database V12 and generally is not the best method to use to migrate to Azure SQL Database V12.

Source of Information : Migrating SQL Server Databases to Azure

Migrating using full database and transaction log backups to minimize downtime

2017-03-25T16:57:00.000+08:00

To minimize downtime involved in a migration using a full database backup, you can use a combination of a full database backup, a differential backup (manual method only), and multiple transaction log backups. By using this method, you will keep downtime to a minimum. This method commonly is used with medium to large databases where minimizing downtime is of paramount importance. To test your database in Azure before migrating a production system, either use the full database backup method or use this method to test and verify the migration steps before the actual production migration.

 Manual method: Back up, copy, and then restore (without recovery) a full database backup, a differential backup (optional), and multiple transaction logs while keeping your production system operational. Continue taking and applying transaction log backups without recovery until you are ready to switch over. Apply the final transaction log backup with recovery when you are ready to switch your clients/applications over to connect to the SQL Server instance in the Azure virtual machine rather than the on-premises database.

 AlwaysOn Availability Group method: If you have an on-premises AlwaysOn Availability Group, you can extend the availability group to SQL Server in your virtual machine. Configure your SQL Server instance in the Azure virtual machine as an AlwaysOn replica, seed the replica with a full database backup, keep the replica current with transaction log backups that are applied automatically, and then fail over to the Azure replica when you are ready to switch over your clients to the SQL Server instance in the Azure virtual machine (and turn off AlwaysOn in the source database).

Source of Information : Migrating SQL Server Databases to Azure