Not too long before this incident occurred, we had gone through the effort of formalizing our backup procedures. New scripts were prepared and tested independently by both our system administrators and Oracle Database administrators. The SAs produced a backup script which dump(1)ed each file-system, one after the other, to a tape. Likewise, the DBAs prepared a backup script to preserve the Oracle database contents as a cpio(1) archive to a tape. The two scripts were subsequently integrated, so that the entire system backup would be performed nightly to a single high-density 4mm DDS tape, which was the state-of-the-art at the time - mid 1996.
Unfortunately, after the integration of the entire backup procedure, no one had properly tested the resulting combined script by attempting to restore data from the resulting backup tapes. While the file-system and database portion performed flawlessly as independent backups, it was soon discovered that the database backup, which executed after the file-system backups, either began by issuing an mt rew command which caused the tape to be rewound (this was the method we sometimes used simply to determine if a tape was loaded), or it did not access the tape drive by its "no rewind" device file.
Conventionally, a Unix tape device file whose name contain the letter "n", as in "/dev/rmt/0hbn", leaves the tape read/write head at the current position after a tape operation completes. This feature enables one to issue a sequence of backup commands which write multiple backups, one after another, to a single tape. Without this feature, every backup operation would be followed by the tape being rewound automatically, and mean that a subsequent backup would overwrite the previous. In other words, without the no-rewind device, a tape would hold the data from only one backup operation, such as a single file-system dump.
As you can imagine, the consequences of this faux paus were disasterous. When the SA attempted to "fast forward" the tape, using mt fsf n, to the position of the lost file-system's backup, the tape drive encountered the "End of Data" marker and stopped there instead. In short order it was concluded that the relatively small database backup had "overwritten" one or more of the relatively large file-system backups, and then the tape drive dutifully wrote a new "End of Data" marker just after the database backup, but before the position of file-system backup that we needed to access. Essentially, the tape drive itself was keeping us from reaching our data, which unfortunately was located past the "End of Data" marker.
With some understanding of tape markers, and past experienced with tapes of many sorts, the SAs laid out their options, each of which were pursued to some degree. Simultaneously, a couple nosy system programmers - myself included - who had in the past been responsibly for some SysAdmin tasks, editorialized amongst themselves as to the merits of each of these pursuits. We independently followed our own tack, in part, because of a division of responsibilities within the growing company and some areas were off limits to us unless deemed absolutely necessary. I'll describe each of the candidate approaches.
An intial thought was that we could send the broken disk to a firm specializing in data recovery. If memory serves me, a phone call was made and it was determined that it would cost $5,000 minumum, regardless of whether or not anything could be retrieved. Also, the round-trip time for this process would be at least 2 to 4 days. Since we were game to follow our other leads, and hopefully make some progress in short order, this was not an interesting option to pursue immediately.
A number of us thought we could investigate how one might be able to ask the tape drive to skip past the premature "End of Data" (EOD) marker. Also, the brute-force option of simply physically winding the tape past the marker, and then loading it into the tape drive, was considered.
Alas, these were short paths to nowhere. Apparently the DDS specification dictates the characteristics of the EOD marker and the behavior of the drive with regard to it. Neither the mt(1) command nor the driver itself seem to have a feature to skip past EOD. We surmised that it is either impossible for the tape drive hardware to read through or past EOD, or at that it is only within the control of the drive firmware.
A brute force method was also tried. We wound the tape ahead manually with the assistance of either the eraser-end of a pencil or ball-point pen inserted into one of the tape reel access holes. Unfortunately upon loading the tape into the drive, though, the tape drive would automatically rewind the tape before any operations could be performed.
Another thought was that we could unwind the tape and attempt to splice out the premature "End of Data" marker - certainly nothing elegant, but still a possibility.
Conveniently, the backup operation had been performed in this erroneous way for some time. As a side effect we had accumulated boxes of similarly botched tapes. The older tapes provided lots of raw material with which to attempt to find a workable recovery procedure by trial-and-error.
For splicing, the tapes unfortunately weren't good 'ol reel-to-reel 9 track tapes, but rather they were tiny 4mm Digtal Data Storage "DDS" tapes, just like the Digital Audio Tapes "DAT" media. Physical manipulation of the media was further impeded by the tape enclosure which is similar to that of a VHS video tape, only much smaller, with a spring-loaded "door" that snaps shut to protect the magnetic medium. The tape and tape drive specifications were examined to determine the length of the tape and data density, and some rough values in meters were decided upon as to where the splice should occur. Before too long, the computer room floor was littered with snippets of tape, every one a reminder of another failed attempt. In the end, none of the spliced tapes was able to be read. My coworker hypothesized that this may be due to the "helical scanning" head that is used to write the data to the tape itself - it is perhaps impossibly to splice such a tape amidst the data and preserve its integrity.
While fruitless experimentation continued in the computer room, I turned to my web browser and visited "http://www.deja.com", which was "www.dejanews.com" at the time, hoping to find the advices of others who had been in this predicament. Deja.com provides a wonderful archive of USENET news, a sort of "USENET Library of Congress", but with commercials. Using the site's "Power Search" feature I found a thread from the "comp.sys.hp.hpux" newsgroup with the subject "[Q] reading past End Of Tape with HP-UX 9.05". To my surprise the poster described a method by which one can potentially recover data from a tape trashed in the way I've described by:
Load a clobbered tape into the drive.
Postion the tape to EOD, using the "'n'o rewind" device file:
# mt -f /dev/rmt/0hbn eod
Create a compressed sample data file, e.g. "/tmp/17secs_of_tape". This file, when written to the tape positioned at EOD, will hopefully result in enough data being written to clobber an EOD marker. Here, we call it 17secs_of_tape because that is the estimated length of time which should be sufficient to clobber the ten-second long EOD marker. This marker is relatively large since it must be able to be reliably observed and identified by the tape drive when performing a "fast-forward" operation. This is similar to how some modern cassette tape players locate the quite spaces between songs on a tape of music.
The following is a quote from Kevin Jones <kev@bri.hp.com>, an HP employee who, in an "unofficial" capacity, authored and posted a step-by-step procedure from which I worked. Regarding this portion of the procedure he wrote:
On 354x0 drives, with a native transfer rate of 184 Kbytes/sec, the 17secs_of_tape file should be just over 3 Mbytes of compressed data. You can use whatever data you want, not just the hp-ux kernel. On HP C15x3 drives with a native transfer rate of 510 Kbytes/sec, you need about 8 Mbytes of data.
Kevin chose to use the HP-UX kernel itself as a source of data to write, since it was a known quantity to him, and contains sufficiently random content to produce a compressed data file of a given size. If you are not running HP-UX, or are running a version with a kernel very different in size from the 9.05 kernel, it would perhaps be best to choose some other data file. If it exists on your system, I suggest using "/dev/random" as a source of unpredictable content, which is unlikely to be able to be compressed much by the tape drive. In this example I'll presume you want a file of 8MB in size:
# ls -l /tmp/17secs_of_tape # determine whether or not it exists # ... # dd bs=1024 count=8192 if=/dev/random of=/tmp/17secs_of_tape # ls -l /tmp/17secs_of_tape # should be 8MB in size
Remember that a file containing more data than necessary could result in you accidentally overwriting even more of your precious data just beyond the premature EOD marker. Be conservative with the file size unless you are certain that there is much tape between the EOD and the data you wish to recover.
Write the sample data to the tape:
# dd if=/tmp/17secs_of_tape of=/dev/rmt/0hbn
Immediately (without performing any other tape operation), power off the tape drive. If the tape drive is internal, consider changing to single-user mode, and be sure to sync(1) the file-system(s) and wait for the drive activity to complete before powering off the system.
Wait a bit, as you probably should whenever powering equipment off and back on.
Fast-forward to just past the first EOF marker (if you have multiple files, or dump(1)s on the tape):
# mt -f /dev/rmt/0hbn fsf 1or, if you want to have your backup/archive utility attempt to "re-sync" and potentially even recover data from within the over-written backup, you might try:
# mt -f /dev/rmt/0hbn fsr 100 # adjust number of records accordingly
Attempt to recover data that was once beyond the premature EOD marker:
# restore if /dev/rmt/0hbn
One complication in our situation was that we were using a compression feature of the tape hardware. On many systems this feature is used when one accesses the tape drive using a device file with the letter "h" in the name, as in "/dev/rmt/0hbn". This compression means that it is hard to judge just how much sample data (in bytes) one must write to to clobber the EOD marker, because the degree of compression achieved depends on the content of the data itself. To deal with this, it is best to use "unrepetitive" or already compressed data when making your "nsecs_of_tape" file because it is repetition in a data stream that enables effective compression.
Another complication was that we did not have access to an external DDS tape drive - they were all internally mounted within the cabinet of HP-UX machines. We decided to enlist the services of another similarly configured, but currently unused, HP-UX machine and, when the time came, throw its "big switch" - there-by powering down the whole machine and the tape drive as a side-effect. While this may seem extremely risky, it was a machine which we could afford to "lose" and it has been noted that powering off an external drive, if we had one, may have elicited similarly undefined behavior perhaps resulting in a kernel panic. If you consider doing this as well, it would be prudent to attempt this recovery operation from single-user mode if possible.
In the end we had great success with the method described above. The software engineers' work was restored and all returned to normal. This experience reminded us that successful execution of a backup script doesn't necessarily mean that one can recover data from the backup. As a result, backup recovery tests were added to the list of SAs backup responsibilities. The creative method which was ultimately employed to do the "impossible" serves as an example of what can be done when SAs or engineers take the time to share their experiences with their co-worker and their peers elsewhere.
I would like to thank Kevin Jones <kev@bri.hp.com> and Frank Slootweg <franks@neth.hp.com> for their participation in USENET news and, specifically, in assisting with this problem.
Dave is a benevolent hacker and UNIX aficionado. He is a
Senior Systems Programmer in the Network Engineering Technology group
within the Division of Information Technology (DoIT),
at the University of Wisconsin, Madison.
He can be reached at plonka@doit.wisc.edu or via http://net.doit.wisc.edu/~plonka/.