How to benchmark a quantum computer?

Question

Using a simple puzzle game to benchmark quantum computers is the most clever approach I have seen so far.

The author of the aforementioned article, James, makes a nice analogy to buying a laptop ("more than just a single number when comparing") in an answer to How should different quantum computing devices be compared?.

Another answer by whurley to the same question mentions the IEEE Standard for Quantum Computing Performance Metrics & Performance Benchmarking.

In When will we know that quantum supremacy has been reached? there is an answer by Niel which includes this snippet:

But the bottom line is that, yes, "quantum supremacy" is precisely about "not being able to simulate quantum computers of certain sizes", or at least not being able to simulate certain specific processes that you can have them perform, and this benchmark depends not only on quantum technology but on the best available classical technology and the best available classical techniques. It is a blurry boundary which, if we are being serious about things, we will only be confident that we have passed a year or two after the fact. But it is an important boundary to cross.

My question is bipartite:

• What will be the primary metrics for quantum computers? (Clasical examples include processor speed, RAM & hard drive space)

• Best tools / strategies for benchmarking above metrics? (Existing or proposed)

Answer 1

This may not exactly answer your question (which I suspect is still very much an open question, and what you're likely to get as answers are opinions), but have you looked at blind quantum computation? See here for another perspective.

One way that we can describe that premise is to imagine some company claims to have developed a fabulous universal quantum computer. But it's so expensive, difficult to control etc. that only they can be trusted to run stuff on it, and it's not directly open for use by other people. How do you know it's really running as a quantum computer, and not just that, for example, they have some new classical algorithm that can simulate quantum computations better than we were expecting?

Blind quantum computation achieves this by making the quantum computer perform a computation without knowing what that computation actually is! The company who owns the computer is left blind to what it's supposed to be doing, so their cheating strategies are severely curtailed.

However, I assume that this method would not be applicable to the quantum supremacy scenario.

Answer 2

One approach to benchmarking that I find appealing is to run a random circuit sampling experiment just long enough for a collision to occur, as explained nicely in this paper by A. Mari.

Briefly, one repeatedly samples from a random $n$-qubit quantum circuit and counts the number of shots needed until the first collision (i.e., the number of shots needed until a repeat is found). If the circuit were always spitting out junk then the samples are drawn uniformly at random and by the birthday paradox one would need $\sqrt {2^n}$ samples until a repeat is found. But if quantum computer faithfully executed the circuit, then under reasonable assumptions about the depth of the circuit one would only need $\sqrt {2^{n-1}}$ shots until the first collision - i.e. quadratically fewer samples.

The expected number of shots until collision and the expected number of collisions scales nicely with the linear cross-entropy benchmark - which is hard to calculate for larger $n$ and deeper circuits. So, Mari proposes to trade off classical computational resources (to classically determine the cross-entropy fidelity) for quantum circuit shot executions (to find a collision).

What I really like about Mari's paper is that we can test two different quantum computers against each other. Say Alice is a hero device having known and very high fidelity while Bob is the young start-up quantum computer of unknown fidelity. We can give Alice and Bob the same circuit and have them go at it, waiting until the time where Bob outputs a string that Alice had already output (or vice-versa), and from there assess Bob's fidelity.