As Hamlet so eloquently put it “There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”, just because something is outside of our normal experience does not mean that it is not possible.

One such case landed at our offices recently, 4 disks in a RAID 5 with apparently corrupted data files. Once the drives had been imaged and the data secured we set about examining the RAID and it all looked pretty good. The RAID scheme was a standard last to first data striping with parity shifting first to last every 256KB, and the file system appeared to be perfect.

Then we looked at the MS-Exchange EDB files. The PRIV.EDB started fine then after a while the EDB page checksums were all wrong. The PUB.EDB file did not even start with the correct header pages, though the data did have the appearance of Exchange Information Store data. Examining the data in free space for Exchange EDB headers we did find one for an EDB from the required date.

So, the file system was perfect but the data was not quite right, maybe not corrupted as such, but shifted.

At this point we had not looked further down the disk, but now we started to examine the data in more depth. A second data partition existed on the volume, but on the current RAID configuration the file system was not correct. We could find our way from the boot sector to the MFT, but then after a few sectors the entry numbering went awry. The RAID configuration was different for the second partition.

It is not usual for a RAID to be in this state, so it appeared to us that perhaps the RAID had somehow been re-configured, and that this explained the data issues on the first partition. Sure enough checking the allocation information for PRIV.EDB it was apparent that the data was correct if it was from a low numbered set of file system clusters, but incorrect if it was from high numbered ones, so perhaps the re-configuration had stopped after the bulk of the file system information for partition 1, but not before the end of the data.

Not finding the notion of playing spot the configuration change somewhere in the midst of a large volume of RAID data we set about a new approach.

We had 2 RAID orgnisations, one that worked for partition 1 up to a point and one that probably worked from somewhere in the middle of partition 1 though partition 2. The customer had a set of EDB files that they rather needed back and straddled the point where the RAID configuration changed.

We created 2 versions of the RAID data using each configuration and then extracted the required files.  EDB files have page checksums. And it is possible to work out the page number, so processing an EDB file you can tell if the data is almost certainly correct.  So the next stage was to process each EDB, check each page, and for any that failed check the equivalent page for the alternate file.

The result was stunning, EDB files that not only could be processed using the checking options of eseutil (the Exchange utility program) with minimal errors, and that could then be accessed via Exchange and all mail items examined.

Looking at this as a standard RAID data recovery the answer would have been “your data has gone forever”. Utilities such as R-Studio and Winhex could be used to access the data from the file system, but not to recover the files as the customer needed them, so anyone just playing point and shoot with a recovery utility would be on to a loser.

The answer for this recovery was to examine the data, to really check what was there and to not let preconceptions prejudice the outcome. By finding the facts, and letting them determine the course of the recovery and not letting the demons of “that cannot be possible” cause a distraction, the data was saved.

So, you are an investigator who has been handed a case containing 25 LTO4 cartridges from a TSM archive, now what?

Unlike relatively straightforward backup applications such as BackupExec or ARCserve you cannot simple buy a TSM installation DVD and install it on your Windows Workstation and start examining the tapes. Though even with other applications the access to data from Exchange and other agent backups is not easy without re-creating the originating infrastructure. Even if you had the experience and patience to set up a Tivoli system there is no method for importing foreign tapes.

Like the proverbial London bus, you don’t see one for ages and then two come along at once. In less then a week, one set of TSM tapes arrived as part of a forensic examination, and another for data recovery as the tapes had expired and the customer needed their Notes data restored.

Within the TSM heterogeneous infrastructure, data from many different sources can exist. In these cases the data was a mixture of Windows files, UNIX files and Notes mail database files. Using TSM was not an option, but we did find a TSM salvage utility that was available on one of the Tivioli forums. Unfortunately the writer had not seen data of the flavours we were attempting to recover so the program could not handle it. We decided the the only viable option was to create our own software for the purpose.

This was by no means a straightforward task, TSM data is encapsulated in multiple layers of data buffers each of which requires that its key structures are identified and then a process written to peel it away. Eventually, however, after much huffing and late night working we arrived at the data and were able to extract files, one customer had their email back and the other had evidence that might otherwise have not been available to them.

Tapes represent a valuable source of historical data for external investigations, and for when historical data is required under the regulatory regimes that rule over business areas such as banking. Yet there is a problem with tapes having expired, or being provided by a third party for examination the situation can arise where all of the data are present and correct on good condition well-recorded tapes, but there is not way to recover it.

Having spent the past 26 years of my life working out data from platforms as diverse as Honeywell 2000, VMS, ICL1900 and UNIX systems tackling TSM tape data recovery should have been a walk in the park but it still took us quite a while, and a lot of stress, but ultimately the results were good for all concerned.

The release of LTO5 by Quantum Corporation brings 1.5TB native/3TB compressed tape to the market, and it is a sure fire bet that IBM and HP will shortly follow with their own offerings, which means that for the past 20 years or so, a technology many said was going the way of the Dodo, has managed to more than keep pace with competing technologies, and seen quite a few off (remember how optical disk was the future of storage  back in the late 1980’s?).

So why is tape still with us? In our opinion the answer is that it is just so much more suitable for long term data retention than any competing data storage medium. Disk based backup systems will often win out in terms of ease of backup and speed of restore, but would you trust one to keep your archive safe for many years? Could you afford the electricity bill as the disk based archive grew to mammoth proportions? If what you need is a durable, time served technology that allows large volumes of data to be stored at a relatively low cost of ownership, with options for Write-Once security and easy to use secure data encryption, tape still provides the answer. Certainly operations such as CERN’s Large Hadron Collider rely on tape archives for data storage, and in the financial services sector tape is widely viewed as the leading medium for long term archiving with LTO leading the way.

In our tape data recovery lab we still do receive LTO tapes, but more often than not the problem is one one human error rather than any mechanical failure, the technology is, we believe, very sound.

What about the future? Well it is worth having a look at this article on IBM’s web site. The technology is with us today that could lead to tape capacities up in 35TB native level.  This is not to say that hard disks won’t see equally impressive increases in capacity and performance, nor that solid state storage will not improve in terms of longevity and cost, but it is clear that tape still has a future mapped out, is still a valuable part of any serious storage infrastructure and has the big guns of the storage industry firmly behind it.

Source: http://searchdatabackup.techtarget.com/news/article/0,289142,sid187_gci1405270,00.html?track=sy941

When a file system has become severely corrupt, or has been wiped out by re-partitioning or re-formatting, the ability to examine every block of data within a partition, hard drive or RAID array to identify and extract specific fragments of a specific file could be your only hope of recovery. This may sound extreme but it does happen, especially with business critical systems such as email and database servers.

Back in the days of Exchange 5.5, Exchange 2000 and Exchange 2003 (pre service pack 1) the internal structure of the 4K or 8K data pages that made up the information store files had enough encoded data within them to allow the pages to not only be recognised but also to identify where in the file the page was located. The values that allowed us to do this included the page CRC (cyclic redundancy check) checksum value and the pgnoThis page number. These were two 32bit values that sat side by side at the start of the the page structure.

Whilst the CRC value allowed a page to be checked to see if the data was valid and the pgnoThis could identify if the page was in the correct position within the file, it did not allow any errors within the page to be rectified. Research by Microsoft apparently identified that approximately 40 percent of CRC errors identified were caused by single bit “bit flip” errors so, with the release of Exchange 2003 service pack 1 in May 2004, they replaced the CRC and pgnoThis values in the page header structure with an updated CRC value and an ECC value to allow these single bit errors to be corrected when the page is loaded.

The algorithm to produce the first CRC value is seeded with the page number. This means that Exchange can still determine if the page it read is the page it expected to read however the CRC value can now no longer easily be used to identify a page of Exchange data if you don’t know its location as is the case when scavenging data during a data recovery process. Therefore, new techniques have had to be developed. Microsoft’s changes to the ESE (Extensible Storage Engine) architecture to allow a single bit within a block of Exchange data mean it is no longer easy to recovery a block or page of Exchange data when the allocation information for that file no longer exists.

Data recovery work of this nature require an in depth knowledge of the internal structures of files and a lot of dedication to produce the best possible result and this is how we approach every data recovery we do.

References:
Extensible Storage Engine Architecture: http://technet.microsoft.com/en-us/library/aa998171%28EXCHG.65%29.aspx

A NAS storage device  in need of recovery was delivered to us last week. The customer had been using the device and on Friday evening all data was present and correct, but when the customer went to access the device on the Monday morning it was operational but there was no data present. So where had it gone and could a data recovery be achieved?

Once the data from the disks had been secured by imaging the disks and backing up the data our diagnosis began. The RAID unit was basically a Linux system in a box with four hard drives, and used the XFS file system to store data. Using the file system structures from within the XFS file system we were able to identify that the RAID used a stripe size of 64KB and with a standard data order, so far so good. The problem was that when we then examined that data from the disks using this RAID configuration, and attempted to follow the XFS INODE information to access the files something did not seem quite right. We dropped the data from the RAID onto a single disk and connected it to a Linux system, and were able to mount it without any errors, but only three directories were visible so it appeared that there had been either file deletion of a file system re-creation, but our utilities did not indicate a problem, so what was wrong?

On closer examination of the hard drive images, we were able to find the answer and it makes for quite scary reading. Yes, there had been a file system re-creation, but before this the RAID had been re-configured. A previous RAID configuration on the device had used a stripe length of 16KB and whatever had happened over the weekend had resulted in the RAID being reconfigured, rebuilt and a new file system written to it. Armed with this information the new RAID configuration setting were used with our data recovery software. When we examined the resultant data produced with these new settings, we were surprised to find that the previously existing filesystem was almost entirely intact and approximately 99% of the customers data could be recovered.

How was this possible? First, the reconfiguration of the RAID did not cause the parity blocks to be recalculated for the entire RAID volume, parity was only written in the stripes where new data was written. Second, the change in the RAID stripe size caused the allocation group size of the XFS filesystem to change, shifting the location of internal filesystem data and therefore preserving many of the previous XFS structures that we were then able to use to reconstruct the previous filesystem.

The moral of this story? We see many instances of NAS RAID systems that have apparent failures or where some level of rebuilding has taken place. This is not an attempt to disparage what is a highly important type of storage in terms of ease of use and cost, but it does demonstrate a failure to comprehend the risk of reliance upon a single storage device for critical data almost as if the fact that a RAID is in use gives complete protection against failure. For the cost of a NAS data recovery a tape device, or additional NAS units to backup up to, could have been put in place resulting in no need for data recovery work and no interruption to business and to sleep patterns.

It might seem an odd request, to copy tapes that went out with the ark, but if you have a dedicated system based around a VAX workstation (for example some CAM systems used VAX) you might still be using it or have a request for design information from a couple of decades ago. VAX systems were well built so will often be in good working order, but if the method of booting is from TK50 tape then the problems can begin.

One of the biggest problems we see with TK50 tapes in our tape data recovery lab is “stiction”, the tape sticks to the read/write head. This is a result of media decay, the material bonded within the media to keep it sliding over the heads smoothly breaks down and suddenly the tape that was running nice and quickly comes to an abrupt halt. Worse still, there is some momentum involved and the dragging of the tape over the heads whilst waiting for it to stop moving can leave a lot of the recorded surface as a gooey mess on the heads. The result, a nice bit of see-through tape and data that will never been seen again.

Is all lost when the media decays in this manner? Not always, in our data recovery lab we have developed methods for keeping the tape moving and stopping it sticking. Usually this allows a recovery of the majority of the data from tape, though we often only get hold of a tape after several failed attempts have already been made and there are already some holes in the media.

The thing to consider if you do have data stored on TK50, TK70 or half-inch open reel tape is that it could be suffering from decay, this could be getting worse, and if there is a likelihood that data will be needed at some time in the future it is worth weighing up the cost of a data migration project now against the cost of a data recovery exercise some time hence.

What is almost certain is that these regulatory paper tigers are about to be forced to become real, and with it will come new zeal for enforcing regulatory compliance leading to an increase in eDiscovery and eDisclosure requests.

If, for example, the FSA knock at the door with a demand for information what do you do? There will be a deadline, and there are plenty of cases of well known companies being fined very heavily for non-disclosure within the time specified, so life can get fraught and very expensive.

The reality is that, unless there are sophisticated data management systems in place, by the time a demand for disclosure is made it is often too late. Where large volumes of data are stored in tape archives has to be re-instated somewhere, inspected and the required data found and returned. This can be quite an onerous task even with a properly maintained archive catalog.

Our experience is that vague requests for information regarding the transfer of data from a couple of hundred tapes often undergo a metamorphosis to become an urgent requirement to track down particular files from some or all of the tapes, “and it must be done by next Tuesday”. It usually transpires that the deadline is for regulatory purposes.

There is a lot that can be done in advance of any disclosure demands, including the following:

• Assess the potential task. If you hold large volumes of data on tape do you have all of the hardware and infrastructure in place to recover data when required?
• Is the archive adequately catalogued so to allow the rapid identification of any required data.
• Do you have the operational capacity within your IT department to deal with a demand for data whilst still maintaining daily operations.

If the answer to any of these is “no” then the cost saving in avoiding re-cataloguing and possibly data migration to new media might soon be revealed to be a false economy.

There are many data recovery software tools on the market these days that, in some circumstances, can provide a quick and cheap solution if the problem being experienced is relatively straight forward. Because of this market in data recovery tools anyone who has some IT skill and is capable of pressing the ‘go’ button in the data recovery software can claim to provide a service. What happens however if the problem is too complex for the recovery tool or there is a hardware fault?

We often see cases that have already been to other recovery companies and have been declare a ‘no-fix’, stating the data cannot be recovered, where we have been able to successfully recover the data the customer required.

We are able to do this because of our level of knowledge and the commitment we give to each and every job. We do this because we have a passion for the work and take pride in the service and results we can offer.

I have been reverse engineering backup formats, file structures and file systems for many years and have been writing software to recover data from some unimaginable scenarios and I am not the only member of the team with these credentials. We can therefore provide a high level of service to all of the jobs we see.

The level of service we can provide is not beholden to any third-party utilities, if we encounter a data recovery situation, legacy backup format or file system corruption that our existing tool set does not cater for we will develop the solution required, no-fix is not an answer.

The hard disk drive in a computer system contains the most up-to date information along with other forensically valuable information such as internet history and local temporary files, so why should you bother looking at the backup tapes?

Ease of Access

Access to the data from a tape archive is often achieved with far less disruption as the tapes can be handed over without systems being seized and imaged. In some instances it is vital that there is not widespread knowledge that an investigation or system audit is underway so taking the backups from an off-site store might be preferable to locking down the active systems for investigation.

The disruption caused by an audit often spreads further than is ideal. People not under any suspicion end up feeling suspected, so being able to make an assessment of the situation without widespread loss of staff morale can be a very good move. Of course care has to be taken that no action in browsing through data contravenes other rules and that it does not result in widespread knee-jerk disciplinary responses. With the exception of clearly illegal activities it is often better to use any semi-covert system audit to develop policy and to draw a line after which contravention will result in strict action.

Historic Data

Backups are a snap-shot of a system or systems, and this can be invaluable. Data can come and go from local systems, and in some instances a degree of data wiping might be done to cover illegal or undesirable action. But, if a piece of data was in place, and was backed up, then any attempts made to eradicate the evidence will be in vain because the information will be securely stored within the tape archive.

Working back through month end-backups can provide a great opportunity to spot wrongdoing and system abuses. Unless great care has been taken, at some point some information will have been in the road of the backup infrastructure and will be stored ready for examination.

Look before leaping

An understanding of the backup infrastructure is required before embarking upon a investigation through a tape archive as there could be a lot of data to work with. Finding out if it is likely that the data you are after will be somewhere in amongst the tapes is a good start, then prioritising the tapes is the next essential step. That the tape archive provides the benefit of a step-back through snap-shots of the system is a great benefit, but it can mean there is a vast quantity of data, much of it duplicate data or unnecessary system files,  so planning to reduce the time and costs is essential.

Based upon a recent case where there was potentially the need to examine data from between three and four thousand AIT cartridges containing data written using the NetBackup archiving utility (without access to any NetBackup catalog information), the importance of a graduated approach becomes abundantly clear.

3000 tapes that require 3 hours each to read, using 10 systems and with an 80% operating time, would take almost 50 days. That is just the time for reading tapes, factor in time for dealing with the recovered data and organising it for return and you could easily end up doubling the time.

Developing a pre-scanning system for this type of tape reduced the time per tape to identify the data on each tape down to about 15 minutes, so all tapes could be scanned in about 4 days. This allowed the identification of 500 tapes from which data was needed, and eliminated the remainder. The overall time to read all of the data reduced to fewer than 10 days, the result being a faster service with lower costs. So a bit of preparation can pay dividends.

Should tapes always be examined?

There is no hard and fast rule, understanding the systems and where the data could be is the first step. The tape archive might be a great source of data, but if the data you want was never backed up then you could end up throwing away money and time on tape data recovery that is not needed. But, by ignoring those “scary tape things”, you could be missing data that could form a vital part of any computer forensic investigation or audit.

Well the problem is that the data from any single backup is distributed throughout the data from several other backups, and to identify the data from the required backup you have to read all of the data, including that from the other backups. For example, one backup set might occupy only 10% of the space on an LTO4 tape, but the restoration might require that 100% of the tape be read as each block must be read to see if it belongs to the required backup set.

The problem then with an archive wide tape data migration project is that if there are 1000 tapes containing backups from 100 different sources, the migration process might require that each tape be read 100 times, so the entire reading process will be the equivalent of transfering data from 100,000 tapes.

A bit more than one of those jobs to leave for a wet afternoon.

I’m not denigrating multiplexing as a procedure. It improves backup performance and logistics, and it is backup that occupies the most time. Unless we are very unlucky a lot less time is spent restoring data from tapes than is spent writing to them. It is just that when the time comes to move a large amount of data the option of using the original backup software and systems can be just too daunting and resource hungry.

There are ways around the problem. We write our tape data migration software to cater for this type of issue, so that a faster high level scan of the tapes can then mean that the tapes have only to be read once. This means that the data migration process can be completed in an acceptable time without the need for a massive investment in infrastructure and staff.