How to Read a Quantum Benchmark

A hardware vendor puts a slide in front of you. It reports a quantum volume of 4,194,304. The next company you meet reports 35 algorithmic qubits. A third quotes a two-qubit gate fidelity of 99.9% and nothing else. Each number was produced by competent engineers running a published protocol, each is defensible, and together they tell you almost nothing about how the three machines rank against each other.

This is not vendor bad faith, and treating it that way will make you a worse reader of the field. Quantum benchmarks measure different properties, on different assumptions, with different amounts of software help permitted. Two of them are related by a logarithm. One of them describes a machine nobody has built yet. Reading them well is a specific skill, and it is learnable without a laboratory.

This guide walks through the five numbers you will actually be shown, what each one is sensitive to, and where accurate figures still drift away from the thing you wanted to know.

What the number is actually claiming

Every benchmark result is three claims compressed into one figure. What was measured. Under what conditions. Against what baseline.

The first claim is the easiest to check and the one people skip. A gate fidelity is a statement about single operations. Fidelity here means the fraction of the time an operation does what it was supposed to do, averaged over a set of test operations. A quantum volume is a statement about whole circuits. A speed metric is a statement about the control electronics and the software stack as much as about the qubits. These are not competing estimates of one underlying quantity, so a machine can lead on one and trail badly on another without anybody having lied.

The second claim is where diligence earns its fee. Was error mitigation used, and which kind? Were results combined across repeated runs? Was the figure measured on the full device or on a hand-picked subset of its best qubits? Vendors usually disclose this, in the technical appendix rather than the press release.

The third claim is the one that ages fastest. Any statement of the form “this would take a classical computer N years” is a claim about the best classical algorithm known to the people making the comparison. That baseline has moved, repeatedly, and it has always moved in the same direction.

The five numbers you will be shown

Random circuit sampling

Random circuit sampling is the benchmark behind the quantum supremacy headlines. The protocol was formalised by Sergio Boixo and colleagues in 2018. A processor applies a long sequence of randomly chosen gates across all its qubits, including entangling gates, which are two-qubit operations that link the state of one qubit to another and are the source of most of the error in current machines. The processor then measures the result, many times over, producing a large pile of bit strings.

The output distribution is scored with cross-entropy benchmarking, usually written XEB. XEB compares how often the machine produced each bit string against how often ideal quantum mechanics says it should have. A perfect machine scores 1. A machine producing pure noise scores 0. Real devices land somewhere between, and in supremacy experiments they land very close to the bottom of that range.

Google’s 2019 Sycamore result, published by Frank Arute and colleagues in Nature, is the reference point. Fifty-three qubits, circuits twenty layers deep, one million samples in about 200 seconds, and a measured XEB fidelity of roughly 0.2%. That fidelity looks like a failing grade and isn’t one. On a circuit that complex, a fidelity distinguishable from zero is itself the finding, because random guessing produces exactly zero and the classical verification of anything better is expensive. A team at the University of Science and Technology of China pushed the same protocol onto 60 qubits and 24 layers in 2021, reporting an XEB fidelity around 0.037%.

What the number tells you: this device can hold a large entangled state together for long enough to produce statistically detectable quantum structure. That is a real and demanding property.

What it does not tell you: whether the device can run anything you would pay for. Random circuits are random by construction. They are chosen to be hard to simulate, not to be useful, and a machine can be strong here while being poorly suited to a structured algorithm that needs a particular connectivity pattern or a much deeper circuit. Random circuit sampling is a frontier test, and in diligence it belongs in the category of scientific credibility rather than product capability.

Quantum volume

Quantum volume was introduced in its current form by Andrew Cross and colleagues at IBM in 2019, and it was built for the NISQ period, the noisy intermediate-scale quantum era in which machines have tens to hundreds of imperfect qubits and no error correction. The question it answers is narrow and useful: what is the largest square circuit this machine can run and still get a meaningful answer?

Square means width equals depth. Width is the number of qubits, depth is the number of successive rounds of gates. The protocol generates many random circuits of size n by n, runs them, and checks whether the machine produced heavy outputs often enough. A heavy output is a bit string whose ideal probability sits above the median, and the pass threshold is a heavy output rate above two thirds with statistical confidence. Push n up until the device fails. Quantum volume is 2 raised to the largest n that passed.

So a machine that clears 7 by 7 has a quantum volume of 128. A machine that clears 10 by 10 has 1,024. Honeywell, now Quantinuum, crossed four digits with its trapped-ion H1 system in 2021 and reported 4,096 on H1-2 the following year, which is a 12 by 12 circuit. IBM moved from 16 on early five-qubit devices up through 32, 64 and 128 over roughly the same period.

The metric is genuinely architecture-neutral, which is why it caught on. The same protocol runs on superconducting circuits, trapped ions and photonic systems, and it punishes every weakness at once: gate errors, readout errors, crosstalk, limited connectivity, and a compiler that produces inefficient gate sequences. A vendor cannot buy quantum volume by adding qubits alone, which is what lets it rank architectures that a raw qubit count cannot.

Two cautions. First, the exponential scale is a presentation problem. Each additional qubit doubles the reported figure, so a number that doubles on one extra qubit will eventually impress people who have not noticed they are reading a logarithm in disguise. Twenty-two clean qubits reads as 4,194,304. Second, quantum volume says nothing about speed. A machine that takes an hour per circuit and a machine that takes a second can report the same value.

Algorithmic qubits

IonQ proposed algorithmic qubits, written #AQ, as a more legible alternative. The stated relationship is direct: #AQ is the base-two logarithm of quantum volume, so a quantum volume of 4,194,304 becomes 22 algorithmic qubits. The difference is not the arithmetic. It is the test set.

Where quantum volume uses random circuits, #AQ runs a suite of actual algorithms at increasing sizes and asks where the results stop being correct. The suite draws on the application-oriented benchmarks published by Thomas Lubinski and collaborators for the Quantum Economic Development Consortium, which cover chemistry simulation, optimisation routines, quantum Fourier transforms and search. The largest problem size at which the machine still returns usable answers across the suite is its algorithmic qubit count.

IonQ’s Jungsang Kim framed the meaning of the number this way: an #AQ of 20 means the machine can run a reference circuit across 20 qubits containing more than 400 entangling gate operations and return a correct result with meaningful confidence. IonQ publishes #AQ values for its commercial systems itself, and those values revise upward with each system generation, so a figure quoted in a pitch deck is a vendor measurement rather than an independently replicated one.

The appeal for a non-specialist reader is obvious. An algorithmic qubit count is a claim about problem size, and problem size is the unit your portfolio company’s use case is written in.

The dispute is equally real, and you should know it before you quote the figure. Quantinuum’s benchmarking team published a 2024 post arguing that the #AQ protocol as defined permits techniques that raise the score without improving the hardware. Their two main objections concern combining results across multiple runs by taking the most common answer, and compiling gate sequences in ways tailored to the specific benchmark circuits. IonQ’s position is that a customer who wants a correct answer will use exactly those techniques, so a benchmark that forbids them measures the wrong thing.

Both positions are coherent, and the disagreement is not really about arithmetic. Quantum volume is a hardware measurement that tries to exclude software help. Algorithmic qubits is an outcome measurement that tries to include it. When a vendor quotes #AQ, the question to ask is which mitigations were applied and what the score looks like without them.

Gate error rates

Randomized benchmarking is the oldest of these methods and the least glamorous. The technique, developed by Emanuel Knill and colleagues in 2008 and extended by Easwar Magesan, Jay Gambetta and Joseph Emerson in 2011, applies a random sequence of gates and then applies the inverse of that sequence. In a perfect machine the qubits return to where they started. In a real one they don’t, and the rate at which the return probability decays as sequences get longer gives you the average error per gate.

This produces the numbers you see on hardware dashboards: a two-qubit gate error of 0.5%, a single-qubit error of 0.02%, a readout error of 1.2%. It is deliberately insensitive to preparation and measurement errors, which is why hardware teams trust it for tracking their own progress.

Read gate errors as an input rather than a result. They tell you roughly how deep a circuit can go before noise swamps the signal, and they are the thing that has to improve before any of the higher-level scores move. They don’t tell you how the device behaves when all qubits are driven at once, because randomized benchmarking averages over gate types and can miss worst-case behaviour on a specific operation. A vendor quoting a best-pair two-qubit fidelity and a median two-qubit fidelity is being more honest than one quoting only the first, and the gap between the two is informative on its own.

Speed

Two metrics address throughput, and only one of them describes machines that exist.

CLOPS, circuit layer operations per second, was proposed by IBM in 2021 as a rough analogue of FLOPS. It measures how many layers of a quantum volume circuit a system can execute per second, counted end to end, so qubit reset time, readout time and the classical processing between iterations all count against it. IBM reported figures around 1,400 CLOPS on its faster systems at the time. Speed becomes the binding constraint for variational algorithms, which call the quantum processor thousands of times inside a classical optimisation loop. A machine with excellent fidelity and poor throughput will run those workloads badly, and no quality metric will show it.

rQOPS, reliable quantum operations per second, was proposed by Microsoft for machines that don’t yet exist. It counts error-corrected logical operations per second, and Microsoft’s stated threshold for commercially interesting work is one million rQOPS. On today’s hardware, rQOPS for any long computation is effectively zero. Treat it as a target that tells you how a serious error-correction roadmap is being scored internally, not as a measurement.

How honest numbers drift

Teaching to the test

Every published, fixed benchmark eventually gets optimised against. This is not a quantum problem, and the clearest case sits in the automotive industry.

European fuel-economy and emissions figures were measured for years on the New European Driving Cycle, a fixed test schedule published in advance. Because the schedule was known, manufacturers tuned to it: gearshift points matched to the exact speed profile, tyres inflated above normal pressure, alternators disconnected, door seams taped, test-weight vehicles stripped of options. Almost all of it was legal, and each figure was measured. The International Council on Clean Transportation tracked the divergence between test values and real-world consumption over a decade, and its 2016 From Laboratory to Road update put the gap at roughly 40% by 2015. Regulators eventually replaced the cycle with a harder one. The lesson generalises: a fixed test with published parameters measures performance on the test, and the gap between the test and the workload grows quietly, without anyone committing fraud.

Quantum benchmarking is at an earlier point on the same curve, and the community is arguing about it openly, which is a good sign rather than a bad one. The quantum volume and algorithmic qubit dispute is a version of this argument. So is the broader question of error mitigation, which covers any post-processing that improves the reported answer without improving the hardware: running a circuit at several noise levels and extrapolating to zero noise, discarding runs that fail a consistency check, taking a majority vote across repeated executions.

Mitigation is legitimate and useful. It is also expensive, usually in the form of many more circuit repetitions, and its cost tends to grow sharply with circuit size. A score achieved with heavy mitigation and a score achieved without it describe different machines, and the difference will show up as soon as the workload gets bigger than the benchmark.

The classical baseline moves

The other systematic drift runs in the opposite direction, against the vendor.

Google’s 2019 estimate that its 200-second sampling run would take 10,000 years on the best available supercomputer was contested within weeks. IBM published an argument that a different use of disk storage on Summit would bring the classical time down to about two and a half days. Later work using tensor network methods on GPU clusters cut the estimate further still. None of this retracted the 2019 experiment; it reset the classical baseline the experiment was measured against.

The pattern repeated in 2023, when IBM published a Nature paper reporting accurate results from a 127-qubit processor on a spin-dynamics problem beyond the reach of brute-force classical simulation. Within weeks, several groups reproduced the same results classically using tensor network and perturbative techniques, some of them on modest hardware. The 127-qubit measurement held up, but the classical-difficulty claim that had carried the headline did not. Google’s Willow announcement in December 2024, with a stated classical cost of 10^25 years for its 105-qubit sampling task, sits under the same caveat, and Google said as much.

For a diligence memo this has one practical implication. A quantum advantage claim is a joint claim about a quantum device and a classical algorithm, and only one half of it is under the vendor’s control. Underwrite the hardware measurement. Discount the comparison.

Physical qubits, logical qubits, and the metrics that replace these

Everything above measures machines without error correction, where the qubits on the chip are the qubits in the algorithm. Those are physical qubits. Error correction spreads one unit of protected information across many physical qubits so that errors can be detected and reversed as the computation runs, and the protected unit is a logical qubit.

The exchange rate is unforgiving. Current estimates for useful fault-tolerant machines run from hundreds to thousands of physical qubits per logical qubit, depending on the code, the error rate and the algorithm. This is why a headline count of a thousand physical qubits and a headline count of a thousand logical qubits describe machines separated by roughly a decade of engineering.

When a vendor’s roadmap crosses into error correction, most of the benchmarks in this guide stop being the right instrument. Quantum volume and algorithmic qubits are both defined against noisy, unprotected circuits. The metrics that replace them will look more like rQOPS: counts of reliable logical operations, logical error rates per cycle, and the physical-to-logical overhead ratio. We tell learners to watch that last figure most closely, because it converts every hardware improvement into a number of usable qubits and it is the number a roadmap is least able to fake.

Two questions do the most work at this stage. Is the logical qubit count demonstrated or projected? And is the logical error rate below the physical error rate, sustained, as the code size increases? A logical qubit that performs worse than the physical qubits it consumes is a research result rather than a component.

A reading procedure

When a benchmark number arrives, work through these in order. The whole pass takes about twenty minutes if the technical appendix exists, and the absence of the appendix is itself an answer.

1. Name the metric precisely. Quantum volume, #AQ, XEB fidelity, average two-qubit gate error and CLOPS are not interchangeable, and press coverage frequently blends them. If the source doesn’t say which protocol produced the number, you don’t yet have a number.
2. Ask which qubits. Full device, or a selected subset? Best pair, median, or worst? A median two-qubit fidelity across all pairs is a far stronger statement than a best-pair figure, and both are commonly published.
3. Ask what software was allowed. Error mitigation, majority voting across runs, post-selection, benchmark-specific compilation. Then ask for the same figure with those switched off.
4. Separate demonstrated from announced. A roadmap target, a projected value and a measured protocol result are three different things, and vendors are generally careful to distinguish them in filings even when marketing does not.
5. Discount every classical comparison. Treat any “N years on a supercomputer” figure as the best classical estimate at the time of writing and assume it will fall.
6. Check whether the metric answers your question. If the use case is variational and requires thousands of circuit calls, throughput dominates and quality metrics will mislead you. If the use case needs deep circuits on many qubits, gate error dominates.
7. Look for a second number from an independent source. Independent replication of quantum benchmarks remains thin, which is precisely why the cases where it exists carry disproportionate weight.

Applied consistently, this turns an unfalsifiable pitch into a short list of specific questions. Most vendors will answer them, and a vendor who cannot has told you something useful.

Building the habit

Benchmark literacy isn’t a matter of memorising which company leads on which metric, because the leaderboard turns over every few quarters and the metrics themselves are being renegotiated in public. The durable part is the reading procedure: knowing what each protocol is sensitive to, where the software help enters, and which half of an advantage claim the vendor actually controls.

That is the part we teach. Quantum Academy’s Certified Quantum Technology Professional (CQTP) program covers hardware evaluation and benchmark interpretation as a working skill, with the protocols worked through in enough detail that you can read a vendor appendix rather than a vendor summary. Program details and enrollment are at quantumacademy.com/.

For a deeper technical treatment of the individual protocols, including the mathematics behind heavy output generation and cross-entropy scoring, see the benchmarking reference on PostQuantum.com.