When a file system has become severely corrupt, or has been wiped out by re-partitioning or re-formatting, the ability to examine every block of data within a partition, hard drive or RAID array to identify and extract specific fragments of a specific file could be your only hope of recovery. This may sound extreme but it does happen, especially with business critical systems such as email and database servers.

Back in the days of Exchange 5.5, Exchange 2000 and Exchange 2003 (pre service pack 1) the internal structure of the 4K or 8K data pages that made up the information store files had enough encoded data within them to allow the pages to not only be recognised but also to identify where in the file the page was located. The values that allowed us to do this included the page CRC (cyclic redundancy check) checksum value and the pgnoThis page number. These were two 32bit values that sat side by side at the start of the the page structure.

Whilst the CRC value allowed a page to be checked to see if the data was valid and the pgnoThis could identify if the page was in the correct position within the file, it did not allow any errors within the page to be rectified. Research by Microsoft apparently identified that approximately 40 percent of CRC errors identified were caused by single bit “bit flip” errors so, with the release of Exchange 2003 service pack 1 in May 2004, they replaced the CRC and pgnoThis values in the page header structure with an updated CRC value and an ECC value to allow these single bit errors to be corrected when the page is loaded.

The algorithm to produce the first CRC value is seeded with the page number. This means that Exchange can still determine if the page it read is the page it expected to read however the CRC value can now no longer easily be used to identify a page of Exchange data if you don’t know its location as is the case when scavenging data during a data recovery process. Therefore, new techniques have had to be developed. Microsoft’s changes to the ESE (Extensible Storage Engine) architecture to allow a single bit within a block of Exchange data mean it is no longer easy to recovery a block or page of Exchange data when the allocation information for that file no longer exists.

Data recovery work of this nature require an in depth knowledge of the internal structures of files and a lot of dedication to produce the best possible result and this is how we approach every data recovery we do.

References:
Extensible Storage Engine Architecture: http://technet.microsoft.com/en-us/library/aa998171%28EXCHG.65%29.aspx

A NAS storage device  in need of recovery was delivered to us last week. The customer had been using the device and on Friday evening all data was present and correct, but when the customer went to access the device on the Monday morning it was operational but there was no data present. So where had it gone and could a data recovery be achieved?

Once the data from the disks had been secured by imaging the disks and backing up the data our diagnosis began. The RAID unit was basically a Linux system in a box with four hard drives, and used the XFS file system to store data. Using the file system structures from within the XFS file system we were able to identify that the RAID used a stripe size of 64KB and with a standard data order, so far so good. The problem was that when we then examined that data from the disks using this RAID configuration, and attempted to follow the XFS INODE information to access the files something did not seem quite right. We dropped the data from the RAID onto a single disk and connected it to a Linux system, and were able to mount it without any errors, but only three directories were visible so it appeared that there had been either file deletion of a file system re-creation, but our utilities did not indicate a problem, so what was wrong?

On closer examination of the hard drive images, we were able to find the answer and it makes for quite scary reading. Yes, there had been a file system re-creation, but before this the RAID had been re-configured. A previous RAID configuration on the device had used a stripe length of 16KB and whatever had happened over the weekend had resulted in the RAID being reconfigured, rebuilt and a new file system written to it. Armed with this information the new RAID configuration setting were used with our data recovery software. When we examined the resultant data produced with these new settings, we were surprised to find that the previously existing filesystem was almost entirely intact and approximately 99% of the customers data could be recovered.

How was this possible? First, the reconfiguration of the RAID did not cause the parity blocks to be recalculated for the entire RAID volume, parity was only written in the stripes where new data was written. Second, the change in the RAID stripe size caused the allocation group size of the XFS filesystem to change, shifting the location of internal filesystem data and therefore preserving many of the previous XFS structures that we were then able to use to reconstruct the previous filesystem.

The moral of this story? We see many instances of NAS RAID systems that have apparent failures or where some level of rebuilding has taken place. This is not an attempt to disparage what is a highly important type of storage in terms of ease of use and cost, but it does demonstrate a failure to comprehend the risk of reliance upon a single storage device for critical data almost as if the fact that a RAID is in use gives complete protection against failure. For the cost of a NAS data recovery a tape device, or additional NAS units to backup up to, could have been put in place resulting in no need for data recovery work and no interruption to business and to sleep patterns.

It might seem an odd request, to copy tapes that went out with the ark, but if you have a dedicated system based around a VAX workstation (for example some CAM systems used VAX) you might still be using it or have a request for design information from a couple of decades ago. VAX systems were well built so will often be in good working order, but if the method of booting is from TK50 tape then the problems can begin.

One of the biggest problems we see with TK50 tapes in our tape data recovery lab is “stiction”, the tape sticks to the read/write head. This is a result of media decay, the material bonded within the media to keep it sliding over the heads smoothly breaks down and suddenly the tape that was running nice and quickly comes to an abrupt halt. Worse still, there is some momentum involved and the dragging of the tape over the heads whilst waiting for it to stop moving can leave a lot of the recorded surface as a gooey mess on the heads. The result, a nice bit of see-through tape and data that will never been seen again.

Is all lost when the media decays in this manner? Not always, in our data recovery lab we have developed methods for keeping the tape moving and stopping it sticking. Usually this allows a recovery of the majority of the data from tape, though we often only get hold of a tape after several failed attempts have already been made and there are already some holes in the media.

The thing to consider if you do have data stored on TK50, TK70 or half-inch open reel tape is that it could be suffering from decay, this could be getting worse, and if there is a likelihood that data will be needed at some time in the future it is worth weighing up the cost of a data migration project now against the cost of a data recovery exercise some time hence.

What is almost certain is that these regulatory paper tigers are about to be forced to become real, and with it will come new zeal for enforcing regulatory compliance leading to an increase in eDiscovery and eDisclosure requests.

If, for example, the FSA knock at the door with a demand for information what do you do? There will be a deadline, and there are plenty of cases of well known companies being fined very heavily for non-disclosure within the time specified, so life can get fraught and very expensive.

The reality is that, unless there are sophisticated data management systems in place, by the time a demand for disclosure is made it is often too late. Where large volumes of data are stored in tape archives has to be re-instated somewhere, inspected and the required data found and returned. This can be quite an onerous task even with a properly maintained archive catalog.

Our experience is that vague requests for information regarding the transfer of data from a couple of hundred tapes often undergo a metamorphosis to become an urgent requirement to track down particular files from some or all of the tapes, “and it must be done by next Tuesday”. It usually transpires that the deadline is for regulatory purposes.

There is a lot that can be done in advance of any disclosure demands, including the following:

• Assess the potential task. If you hold large volumes of data on tape do you have all of the hardware and infrastructure in place to recover data when required?
• Is the archive adequately catalogued so to allow the rapid identification of any required data.
• Do you have the operational capacity within your IT department to deal with a demand for data whilst still maintaining daily operations.

If the answer to any of these is “no” then the cost saving in avoiding re-cataloguing and possibly data migration to new media might soon be revealed to be a false economy.

There are many data recovery software tools on the market these days that, in some circumstances, can provide a quick and cheap solution if the problem being experienced is relatively straight forward. Because of this market in data recovery tools anyone who has some IT skill and is capable of pressing the ‘go’ button in the data recovery software can claim to provide a service. What happens however if the problem is too complex for the recovery tool or there is a hardware fault?

We often see cases that have already been to other recovery companies and have been declare a ‘no-fix’, stating the data cannot be recovered, where we have been able to successfully recover the data the customer required.

We are able to do this because of our level of knowledge and the commitment we give to each and every job. We do this because we have a passion for the work and take pride in the service and results we can offer.

I have been reverse engineering backup formats, file structures and file systems for many years and have been writing software to recover data from some unimaginable scenarios and I am not the only member of the team with these credentials. We can therefore provide a high level of service to all of the jobs we see.

The level of service we can provide is not beholden to any third-party utilities, if we encounter a data recovery situation, legacy backup format or file system corruption that our existing tool set does not cater for we will develop the solution required, no-fix is not an answer.

The hard disk drive in a computer system contains the most up-to date information along with other forensically valuable information such as internet history and local temporary files, so why should you bother looking at the backup tapes?

Ease of Access

Access to the data from a tape archive is often achieved with far less disruption as the tapes can be handed over without systems being seized and imaged. In some instances it is vital that there is not widespread knowledge that an investigation or system audit is underway so taking the backups from an off-site store might be preferable to locking down the active systems for investigation.

The disruption caused by an audit often spreads further than is ideal. People not under any suspicion end up feeling suspected, so being able to make an assessment of the situation without widespread loss of staff morale can be a very good move. Of course care has to be taken that no action in browsing through data contravenes other rules and that it does not result in widespread knee-jerk disciplinary responses. With the exception of clearly illegal activities it is often better to use any semi-covert system audit to develop policy and to draw a line after which contravention will result in strict action.

Historic Data

Backups are a snap-shot of a system or systems, and this can be invaluable. Data can come and go from local systems, and in some instances a degree of data wiping might be done to cover illegal or undesirable action. But, if a piece of data was in place, and was backed up, then any attempts made to eradicate the evidence will be in vain because the information will be securely stored within the tape archive.

Working back through month end-backups can provide a great opportunity to spot wrongdoing and system abuses. Unless great care has been taken, at some point some information will have been in the road of the backup infrastructure and will be stored ready for examination.

Look before leaping

An understanding of the backup infrastructure is required before embarking upon a investigation through a tape archive as there could be a lot of data to work with. Finding out if it is likely that the data you are after will be somewhere in amongst the tapes is a good start, then prioritising the tapes is the next essential step. That the tape archive provides the benefit of a step-back through snap-shots of the system is a great benefit, but it can mean there is a vast quantity of data, much of it duplicate data or unnecessary system files,  so planning to reduce the time and costs is essential.

Based upon a recent case where there was potentially the need to examine data from between three and four thousand AIT cartridges containing data written using the NetBackup archiving utility (without access to any NetBackup catalog information), the importance of a graduated approach becomes abundantly clear.

3000 tapes that require 3 hours each to read, using 10 systems and with an 80% operating time, would take almost 50 days. That is just the time for reading tapes, factor in time for dealing with the recovered data and organising it for return and you could easily end up doubling the time.

Developing a pre-scanning system for this type of tape reduced the time per tape to identify the data on each tape down to about 15 minutes, so all tapes could be scanned in about 4 days. This allowed the identification of 500 tapes from which data was needed, and eliminated the remainder. The overall time to read all of the data reduced to fewer than 10 days, the result being a faster service with lower costs. So a bit of preparation can pay dividends.

Should tapes always be examined?

There is no hard and fast rule, understanding the systems and where the data could be is the first step. The tape archive might be a great source of data, but if the data you want was never backed up then you could end up throwing away money and time on tape data recovery that is not needed. But, by ignoring those “scary tape things”, you could be missing data that could form a vital part of any computer forensic investigation or audit.

Well the problem is that the data from any single backup is distributed throughout the data from several other backups, and to identify the data from the required backup you have to read all of the data, including that from the other backups. For example, one backup set might occupy only 10% of the space on an LTO4 tape, but the restoration might require that 100% of the tape be read as each block must be read to see if it belongs to the required backup set.

The problem then with an archive wide tape data migration project is that if there are 1000 tapes containing backups from 100 different sources, the migration process might require that each tape be read 100 times, so the entire reading process will be the equivalent of transfering data from 100,000 tapes.

A bit more than one of those jobs to leave for a wet afternoon.

I’m not denigrating multiplexing as a procedure. It improves backup performance and logistics, and it is backup that occupies the most time. Unless we are very unlucky a lot less time is spent restoring data from tapes than is spent writing to them. It is just that when the time comes to move a large amount of data the option of using the original backup software and systems can be just too daunting and resource hungry.

There are ways around the problem. We write our tape data migration software to cater for this type of issue, so that a faster high level scan of the tapes can then mean that the tapes have only to be read once. This means that the data migration process can be completed in an acceptable time without the need for a massive investment in infrastructure and staff.

So what happened? Back in the 1980’s hard drives were expensive, not much trusted and low capacity. Optical disk offered a far higher capacity, and being a removable media technology, the ability to expand storage by simply using more disks. Tape was then seen as a technology in transition, again not adequate on the capacity front, and there was a perception of reliability issues.

Optical disk storage ticked all of the boxes, appeared to be rugged, WORM variants satisfied those for whom data retention was an issue, and they were shiny. Not being as flippant as it might seem, they had the appearance of robustness, and were close enough in terms of usability to resemble hard drive, and so felt comfortingly familiar and trustworthy.

These days we only really see Magneto Optical when people want to migrate away from it or need an optical disk data recovery. Phase Change optical died with the loss of market share in the early 1990’s, some of the ablative formats soldiered on at the high end in the banking and pharmaceutical sectors, and the mainstream of the format evolved into UDO (Ultra Density Optical). But, with only one company pushing the format, against the massive technological advances in hard drive storage and LTO tape, the inevitable happened and the format died.

There is still a lot of data stored on optical disks around the world, we see a steady stream through our optical data migration and data conversion operation, they come from VMS systems, medical scanners and a variety of UNIX platforms, and all finally abandoning the technology for pastures new.

Other optical technologies do still thrive in the form of CD, DVD and Blue Ray, all of which are excellent for data transfer, and for gaming and software distribution, but no longer for the long term archive of data.

Having faithfully adhered to a regime of nightly backup of a SUN UNIX system, the failure of the hard drive appeared to be an inconvenience that would soon be overcome for this customer. The hard drive having swiftly been replaced, the backup tape was brought back from storage and the restore process initiated. Five minutes in, however, and the DLT drive made a nasty noise, the cleaning lights came on and ufsrestore became ufs-cannot-restore.

We received a call from the customer wanting to know if the data from the backup tape could be recovered. The disk was with another data recovery company who had found that there were a significant number of bad sectors that meant the required data could not be recovered intact. This is not a smug self-congratulatory tale of “then we got everything back from the tape” as at the point where the restore failed there was nothing to be done, that was where the backup had been terminated as the result of a tape problem and no-one had noticed.

The process followed by the customer was to use ufsdump to create a backup file on the hard drive, then to use dd to transfer this to tape. This meant that they had an immediately accessible backup on-line in case of file deletion or corruption, and the fall back of a tape backup to cover against a total failure (or not in this instance).

So, the data could not be recovered from the hard drive nor from the tape, so was all lost? Well, no it was not. An examination of the disk revealed that the ufsdump archive could be partially recovered, but that the critical section at the start of the backup where all of the directory information (which includes slightly important data such as the file names) was lost forever. The tape, however, contained a valid backup up to the point of failure, and an examination of this data showed that it extended into the area of data where files were stored.

In the motor trade a “cut-n-shut” car is viewed as a dangerous beast that should never be allowed on the road, for this customer it was the answer to their UNIX data recovery problem as the data from two sources could be pieced together to create a completed backup and a full restore was initiated.

A happy story, a success, but how often are opportunities such as this missed? The best data recovery service companies, not the biggest but those who understand data, will always pick up on such opportunities to get the right result. But, if the data recovery work is being done by someone who just runs some software and if it says “failed” they move on to the next job then there will be unnecessary data losses.

Some people seem to think so, but have been dangerously mis-informed. The error correction used in a RAID5 array is there to provide a level of protection for what will often be business critical data, but can still only survive the loss of a single hard drive from the array. Even RAID 6 with double error correction can only cope with a failure of two drives.

You might wonder how likely it is that multiple disks will fail in a RAID, the answer is that it is quite likely.

Environmental factors, for example heat, can have a severe impact upon the operation of a hard disk drive. With a RAID array the disks are in the same enclosure and so an environmental factor will influence all hard drives.

Electronics and connectivity must also be taken into account. In a RAID 5 array all disks are connected to a single controller, and may be on a single data bus. Instability on the bus can result in hard drives being determined to have failed even when they have not. We have received RAID for data recovery with multiple hard drive failures, but under closer examination it has transpired that one drive has failed and resulted in three drives disappearing from the bus and being logged as failed.

Failure with a RAID 5 array can result from minor problems. A single bad sector on a disk will often result in that disk being flagged as failed. This makes sense as the failed sector cannot be read nor written so the hard drive should no longer be used and must be replaced. But, this does mean that a failure of two disk sectors on separate drives, that is 1024 bytes becoming unreadable, can result in a RAID no longer being accessible.

So, whilst RAID 5 brings many benefits in terms of capacity, performance and data protection, the security given by the RAID’s fault tolerance should not be used as an excuse to relax other procedures for data protection such as tape backup.