Monitoring files in preservation archives

Question

What are efficient and accurate techniques for monitoring the recoverability and integrity of files in very large preservation archives?

In very large archives, the time taken to recompute checksums periodically (scrubbing) is substantial, perhaps taking more than all the available time depending on the read bandwidth available! Also, each access to a preserved file increases the risk of damage due to hardware or software failure. Tapes are most stable in a cold, dark place far from exposure to the hazards of data centers. Disks are most at risk when the read/write head is flying close to the medium. All approaches are probabilistic, so which are most efficient and accurate?

To give the problem specificity, let's assume a fixed probability of local single-bit errors for each medium (one probability for tape, another for disk, SSD, etc) during a standard time period, and ignore all other types of errors (loss of an entire volume, for instance). We can also assume a fixed read bandwidth for each medium.

Answer 1

This is not an answer to the question per se, but an extended comment.

There was an interesting set of slides regarding CERN's Advanced Storage (CASTOR) at CHEP'2010. This is a large mixed media archive with some 25 PB of tape archives (ca. 2010). One of the presentations: "Tape archive challenges" highlights some of the issues that seem to question the assumptions set in the OP.

Tapes are most stable in a cold, dark place far from exposure to the hazards of data centers.

Not quite. Even if you have a controlled environment in the tape storage you should expect a very small percentage of the tapes to fail. Therefore it makes sense to keep repacking the data constantly. Also older tape drives may fail making the data on old tapes inaccessible.

The latter reason makes archivists keep copying tapes at a much higher rate than their physical life span would suggest. Marty Perlmutter, "The Lost Picture Show: Hollywood Archivists Can't Outpace Obsolescence".

Disks are most at risk when the read/write head is flying close to the medium.

When the disk is spinning it constantly monitors itself for errors due to wear and errors due to defects in the manufacturing process. While rotation increases the probability of an error it also through predictive failure analysis reduces the chance of an error to occur undetected. On the contrary, as a mechanical unit a disk that was stored on a shelf might just fail to spin up.

A probabilistic approach to consistency checking has to be aligned with the statistical thresholds set for different actions. Suppose that a probabilistic model says that with probability 50% there is one error for a 1000 cartridges in a tape library. Does it give any hint on the possible action but to check consistency for the complete archive in order to find the possibly damaged part so that it can be rebuilt.

Many papers with statistical models of reliability of computer systems based on queuing theory have this problem. While they provide formulas to quickly estimate parameters of computer systems they often rely on some generic assumptions that may or may not apply in a particular setting.