Verifying randomness and uniqueness of TRNG output

Question

We need to verify two claims about a set of numbers:

• They are random
• They are unique in the set (the set is about 200 billion (2TB bin storage))

Supposedly, the numbers were generated by a TRNG (based on a radiation detector). The TRNG itself is a PCI Express card (which we do not have yet) and is probably a TRNG feeding a CSPRNG.

What ways are there to verify the claims? For randomness there are the NIST and other specs and tools.

But how can one tie a verification of the first claim to verifying the second (uniqueness in the set)? Keep in mind that the output of the TRNG has almost certainly been filtered from duplicates. That is most probably going to break the randomness tests.

Answer 1

There is no such thing as a random number and as such it is a category error to attempt to verify whether a particular set of numbers is ‘random’. There are random generators of numbers; the relevant question is whether you—or, more importantly, a well-funded adversary—can predict the output when you don't know it a priori. To answer that, you must:

1. Study the physics of mechanism inside the device. Hire the world's best theoretical physicists to find the best model with humanity's current understanding of physics for how the mechanism behaves.
   • What verification can you do here? You, or the experimental physicists you hired, can devise experiments to rule out alternative hypotheses for how physics might work. But there's nothing at this stage that you can meaningfully do to the output of your RNG device.
2. Study the engineering of the device. Hire competent engineers to determine whether there are ways for the device to fail, and if so, model how it will behave when it does fail.
   • What verification can you do here? You, or the statisticians you hired too, can devise a hypothesis test to distinguish a failed component from a working component based on the output of your RNG device.
   • Make sure that computing the hypothesis test doesn't leak secrets through side channels.
3. If there is a CSPRNG involved, study the cryptography of the CSPRNG. Hire the smartest cryptanalysts in the world to find distinguishers for the CSPRNG.
   • What verification can you do here? You, or the cryptography engineers you hired too, can write automatic tests using known-answer test vectors to confirm that the CSPRNG implements the same design that has been subjected to decades of scrutiny and vetted by cryptanalysts publishing in serious peer-reviewed academic conferences around the world.
   • Note that the CSPRNG is essentially guaranteed to prevent hardware health tests, which is why you should apply those first to the hardware RNG output before conditioning it with a CSPRNG.

Alternatively, if you don't like these answers, I hear you can buy a book of certified random numbers from RAND Corporation, so named presumably because they're experts on the matter.

---

As for detecting duplicates, it depends on what units the data are in. Duplicate bits? Well, there's only two possible bits, so you probably have duplicates unless you've found new kinds of bits hitherto unknown to humankind! Duplicate terabit-length bit strings? You probably have only one of those even if you have several terabytes of data. But maybe you meant something like 32-bit or 256-bit strings. One way to detect duplicates in a large list is to sort it—costing $O(n \log n)$ time in the naive view of things, or $O(n \sqrt n)$ in the more realistic area-time metric which may become relevant in large volumes of data—and then scan through it for duplicates. But this seems to be a general non-cryptographic algorithm question.

Answer 2

@SqueamishOssifrage has given an accepted answer, so I assume that it addresses your concerns. However the question as stated was about randomness tests, and I think there's something to add concerning that aspect of the question.

I am more familiar with the TestU01 than NIST tests, but based on what I've read in Johnston's Random Number Generators--Principles and Practices, I believe that the following is true for at least most of the NIST SP800-22 tests as well.

These are tests such that, if your numbers were generated by a series of independent uniformly distributed random variables--which is what I take a uniformly distributed TRNG to be supposed to produce--then it would be very improbable that the set of numbers would exhibit the patterns that these tests look for. That is why the tests are based on p-values: A small p-value for a test means that the test is detecting a pattern that would be very improbable if the numbers were generated by a uniformly distributed TRNG, providing evidence that the numbers are not like those that would be expected from such a TRNG.

Your problem is that you believe that the numbers are unlike those produced by a TRNG in that duplicates have been removed--or at least you hope that they have been.

(Ruggero's question in comments and Squeamish Ossifrage's comments about the size of the numbers is relevant here. If they were 16-bit numbers, for example, then given the size of your set, you would be guaranteed to have duplicates.)

So, in theory, you could take each test in the NIST suite or TestU01, and work out what patterns would be improbable on the assumption that duplicates have been removed, using this to construct new tests. Then apply your alternative tests to the numbers. This would require some statistical reasoning and hopefully not a lot of extra coding. TAOCP volume 2, chapter 3 would be a good starting point. It's old, and there are additional tests in the recent test suites (with smaller p-values required for failure, I believe). However, Knuth walks you through the reasoning processes for constructing these tests and provides a semi-cookbook/semi-theoretical introduction to the relevant statistical methodology.

Alternatively, given the size of your numbers and the size of the set, maybe it is possible to prove that the tests in the test suites would not be very sensitive to the presence or absence of a few duplicates. In that case, you could use the test suites as is, and use a separate duplicate-finding algorithm to test for uniqueness, as Squeamish Ossifrage suggested.

EDIT: Another option would be to add back duplicates, and then run the NIST or TestU01 suites. This is not as crazy as it might sound. On the assumption that the generation process is uniform and independent for each number produced, there is a probability distribution over numbers of duplicates (given the size of the set and the number of bits in the numbers). You might want to test for duplicates first, but if there are none, then you can use a sampling method to add them back with probabilities equal to the probabilities of their appearance in the first place. I know of at least one scientific paper that had to do something roughly analogous to this in order to apply statistical tests that the authors had designed. (I'll add the citation if anyone's interested, but it's far from cryptography, and finding the relevant part of the paper and understanding what they're doing requires some explanation.)

ANOTHER EDIT: @SqueamishOssifrage's comments made me realize that I should emphasize that PRNG test suites like TestU01, NIST SP800-22, or Dieharder were originally designed for people developing PRNGs. They may be designed to be easy to use, but it would be worth at least skimming the background documentation for TestU01. Implicit in my statements above about what is probable or improbable is that the improbable can happen. That is, one can get a test "failure" even if the set of numbers being tested should be considered OK, because it is only probable that a good set of numbers will pass each test. If you can test lots of numbers, this is less likely, but it's still possible. (However, I believe that by default TestU01 uses very small p-values such as $10^{-10}$ as standards for failure. That is, failure means that your data has patterns in it that would only have a 1/10000000000 chance of occurring if it had been generated by a truly uniformly distributed generator. On the other hand, there are multiple tests in TestU01, so your data gets multiple chances to exceed this limit. But it is an extreme limit nonetheless.)