✨ Add typed structured quantum benchmarks by denialhaag · Pull Request #2135 · munich-quantum-toolkit/core

denialhaag · 2026-08-17T09:52:58Z

Description

🤖 AI text below 🤖

Add a typed foundation for structured quantum benchmarks.

The new MLIR-free MQT::CoreBench library owns each benchmark's parameters, validation, logical output, analytic reference, and evaluation. This vertical slice contains:

Bernstein--Vazirani with an explicit hidden bitstring and static or dynamic method;
GHZ with qubit count, topology, and measurement basis;
Grover search with an explicit marked bitstring and automatic or explicit iteration count;
QFT with qubit count, period exponent, and standard or semiclassical method;
QPE with precision, an exact rational phase in turns, and standard or iterative method.

Strict, versioned JSON instances materialize all defaults. Self-checking manifests record the resolved parameters, output convention, reference model, and a deterministic case ID. The C++, Python, and mqt-core-bench APIs expose the same contract. The command-line tool lists and describes benchmarks, generates structured QC or jeff, and evaluates counts against a manifest.

The MLIR layer contains compact, structured QC emitters for the five benchmark families. It consumes typed instances and keeps its private dispatch table aligned with the semantic registry instead of duplicating defaults. The Python mqt.core.bench module delegates program generation to the existing MLIR extension, which keeps one MLIR runtime per process.

Distribution-level execution tests against the DD simulator remain a follow-up on the stack that starts at #2077.

Fixes #1115

AI notice

This PR and its contents were created with the assistance of Opus 5 via Claude Code and GPT-5.6 Sol via Codex.

Checklist

The pull request only contains commits that are focused and relevant to this change.
I have added appropriate tests that cover the new/changed functionality.
I have updated the documentation to reflect these changes.
I have added entries to the changelog for any noteworthy additions, changes, fixes, or removals.
I have added migration instructions to the upgrade guide (if needed).
The changes follow the project's style guidelines and introduce no new warnings.
The changes are fully tested and pass the CI checks.
I have reviewed my own code changes.

If PR contains AI-assisted content:

Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope, as required by our AI Usage Guidelines.
Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖 (titles are exempt).
I have disclosed AI assistance in the PR description.
I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

codecov · 2026-08-17T10:34:27Z

Codecov Report

❌ Patch coverage is 90.74367% with 117 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/bench/JSON.cpp	87.5%	58 Missing ⚠️
mlir/bench/MQTCoreBench.cpp	75.0%	41 Missing ⚠️
mlir/bench/Generate.cpp	68.0%	15 Missing ⚠️
src/bench/QPE.cpp	97.9%	2 Missing ⚠️
src/bench/SHA256.cpp	98.5%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Assisted-by: Claude Opus 5 via Claude Code

Assisted-by: Opus 5 via Claude Code

denialhaag · 2026-08-18T10:23:41Z

Modulo some more testing, we should be able to mark this PR as fixing #1115.

Assisted-by: Opus 5 via Claude Code

Assisted-by: OpenAI Codex

Rename the unreleased public component to bench, add typed Bernstein--Vazirani and QFT families, centralize private dispatch, simplify generation, and make the notebook executable. Preserve Daniel Haag’s original structured-program commits in the branch history while keeping new contracts and architecture in this commit. Assisted-by: Codex

Process the no-swap Fourier circuit from the most significant physical qubit and apply semiclassical feed-forward in the same order. Assisted-by: Codex (cherry picked from commit abd9ac38fcf3b7068588895e1bf16eb6725a6943)

Apply the repository formatter to the new BV and Fourier sources and their semantic tests. Assisted-by: Codex

Record combined-stack, installation, wheel, documentation, test, and lint results, including the unrelated QDMI Python blocker. Assisted-by: Codex

DRovara

This is really cool! I didn't expect to find an entire benchmark suite with actual evaluations here when I opened the PR but this is pretty nice.

In fact, looking ahead, I can really imagine adding further custom benchmarks to this to e.g. create a reproduction environment for paper results, but that's beyond the scope for now.

I guess the one disadvantage I see here is that it seems like adding another benchmark program requires quite some effort, but I guess there could be worse things.

The programs themselves look correct, I didn't notice any issues there.

DRovara · 2026-08-24T13:44:58Z

+MQT Core defines a benchmark as validated, benchmark-specific parameters plus an
+analytic reference. One instance produces a structured QC program, a resolved
+manifest, and a stable case ID. Each benchmark returns one classical register
+named `result`. Outcome strings are big-endian: the highest-index result bit is
+the leftmost character.


I feel like this is a bit difficult to understand, especially considering the loaded term "benchmark".

If I understand it correctly, how about a rewording like:

Benchmarks in MQT Core are defines as a tuple of: - benchmark-specific parameters - an analytic reference They allow the construction of: - A structured QC program - a resolved manifest - a stable case ID Resulting programs return one classical register named [...]

For instance, from the current description, I still wouldn't understand that we are talking about specific programs here. And since your first sentence defines a benchmark as parameters + analytic reference, the sentence "each benchmark returns [...]" feels a bit weird - how can parameters + analytic reference even return anything?

Yeah. I like the direction of your suggestion! I'll take that into the next iteration!

DRovara · 2026-08-24T13:46:32Z

+```{code-cell} ipython3
+import json
+import subprocess
+
+
+def run_bench(*arguments: str) -> dict[str, object]:
+    """Run mqt-core-bench and parse its JSON output."""
+    completed = subprocess.run(
+        ["mqt-core-bench", *arguments],
+        check=True,
+        capture_output=True,
+        text=True,
+    )
+    return json.loads(completed.stdout)
+
+
+catalog = run_bench("list")
+[entry["id"] for entry in catalog["benchmarks"]]
+```
+
+```{code-cell} ipython3
+qft_schema = run_bench("describe", "qft")
+qft_schema["properties"]["parameters"]
+```


Isn't it a bit weird to illustrate example usages of a CLI tool with python code and subprocess? This makes it seem like subprocess is the exected way to run it...

Yeah. It certainly is; especially since there is Python code for doing this.
I suppose this was mostly about showing how the CLI can be used, but there are better ways to do that in Jupyter Notebooks. I'll revisit this.

DRovara · 2026-08-24T13:55:22Z

+namespace mqt::bench {
+
+/// Circuit method used for the quantum Fourier transform.
+enum class QFTMethod : uint8_t { Standard, Semiclassical };


I just realised that (I think) there is no documentation anywhere on what these options mean. Here, I didn't quite get what "Semiclassical" means. Is it the same iterative approach as the iQPE?

No, these should probably be documented inline with Doxygen comments.

Same idea as iQPE. semiclassical is just the established name from the literature.

DRovara · 2026-08-24T13:57:48Z

+    return {result};
+  }
+
+  auto query = builder.allocQubitRegisterStorage(width, "query");


This is just a quick random thought but would it make sense/how much effort would it be to also provide an option that can be used for all benchmark programs that replaces any and all qubit registers with multiple individually allocated single qubits?

Hm. I see the idea. Feels like this would be a bit complicated though. Especially for the algorithms using loops because they currently index into registers in the loops, which is not directly possible if the program just has loose qubits.
At the top of my head, I don't see a very easy solution. But maybe I am also overlooking things.

…hmark-foundation

Clarify the instance/program boundary and document the dynamic Fourier methods in the executable notebook. Align the feature with current repository lint and guidance after the main merge. Assisted-by: Codex

MatthiasReumann · 2026-08-28T05:52:45Z

+      }
+    }
+
+    b.scfFor(0, search, 1, [&](Value iv) { b.h(b.loadQubit(q.value, iv)); });


Seeing this, I think it could make sense to have an abstraction that "applies" single qubit gates to registers.

One could even think about allowing registers as arguments of the gates and defining broadcast rules similar to OpenQASM. But I am not quite sure we want this, because it adds a whole bunch of complexity.

What should be easy is adding convenience methods in the builders for easing the construction of the construct here.

I think a modifier along the lines

%out:9 = qco.broadcast %in:9 (%target) { qco.h(%target) } : (!qco.qubit, ...., !qco.qubit) %tensor_out = qco.broadcast %tensor_in (%target) { qco.x(%target) } : (!tensor.tensor<!qco.qubit>)

could be kind of nice as this doesn't add overhead to the individual gates. But definitely not a must have right now.

There may be a precedent in MLIR for how to handle this. This is kind of like vectorization; and there is a vector dialect in MLIR. Might be worth to brainstorm these ideas with an LLM and collect the thoughts in an issue.
This si nothing to be addressed in this PR.

denialhaag

Thanks a lot for all the improvements, @burgholzer! I really like this a lot! 🙂

I went through the PR pretty carefully now and left a comment wherever I stumbled. I'll feed these comments into Codex in a bit. I'm not sure about some of my comments, which is why I sometimes explicitly tagged you.

burgholzer · 2026-08-29T14:29:44Z

Thanks a lot for all the improvements, @burgholzer! I really like this a lot! 🙂

I went through the PR pretty carefully now and left a comment wherever I stumbled. I'll feed these comments in a bit. I'm not sure about some of my comments, which is why I explicitly tagged you sometimes.

Nice! I think I commented on every tag. Let me know if I missed something!

Assisted-by: GPT-5.6 Sol via Codex

Call the resolved configuration an instance and define its generated manifest as the semantic sidecar used for evaluation. Assisted-by: GPT-5.6 Sol via Codex

Keep each emitter self-contained, use lowercase MLIR paths, and add recurring constant operations to the program builder. Apply the remaining C++ review cleanups in the same structural batch. Assisted-by: GPT-5.6 Sol via Codex

Make notebook output easier to read, hide setup details, and link the generated program API. Cover the cross-platform benchmark launcher directly so Python patch coverage includes it. Assisted-by: GPT-5.6 Sol via Codex

Assisted-by: GPT-5.6 Sol via Codex

denialhaag

Another set of comments:

Name the registry iterator type explicitly instead of assuming that std::array iterators are pointers. MSVC uses a wrapper iterator type. Assisted-by: GPT-5.6 Sol via Codex

Group benchmark CLI tests after the core CLI cases. Use QFT and QPE consistently, update constant helper comments to current Doxygen style, and keep the benchmark guide independent of its current families. Assisted-by: GPT-5.6 Sol via Codex

denialhaag · 2026-08-29T20:08:54Z

I think I commented on every tag. Let me know if I missed something!

Thanks a lot for the quick input, @burgholzer! 🙂

I might have gotten a bit lost in the details here, but I think I'm happy with this now. I'm kind of indifferent about adding more benchmarks now or in one or more follow-ups. The latter would certainly make reviewing easier, but only if we leave the infrastructure (largely) alone. Either way, we should add most (if not all) structured benchmarks mentioned in the jeff repository sooner rather than later. 🤔

Assisted-by: GPT-5.6 Sol via Codex

burgholzer · 2026-08-29T20:51:09Z

I'll try to get a review in for this tomorrow, so that we can get this merged and not hold it up for too long. Follow-ups should come quite naturally.

denialhaag self-assigned this Aug 17, 2026

denialhaag added feature New feature or request MLIR Anything related to MLIR labels Aug 17, 2026

MatthiasReumann reviewed Aug 17, 2026

View reviewed changes

Comment thread mlir/benchmark/programs/QFT.cpp Outdated

denialhaag force-pushed the benchmarks-dot-jeff branch from 33b6b18 to a8f62c2 Compare August 17, 2026 12:48

denialhaag added 17 commits August 18, 2026 09:53

Add generator for structured jeff benchmark programs

16aba47

Assisted-by: Claude Opus 5 via Claude Code

Drop the Cap'n Proto link workaround

259ff7c

Assisted-by: Claude Opus 5 via Claude Code

Report an empty modifier body separately in the conversion to jeff

e8619c3

Assisted-by: Claude Opus 5 via Claude Code

Update jeff-mlir to merged version

4e466e3

Clean up a bit

b9b85ab

Fix linter errors

a618842

Promote the benchmark programs to a reusable library

a710b73

Assisted-by: Claude Opus 5 via Claude Code

Add benchmark programs for statically structured algorithms

38a6566

Assisted-by: Claude Opus 5 via Claude Code

Rename the benchmark namespace to mqt::benchmark

fa904d6

Assisted-by: Claude Opus 5 via Claude Code

Collect the benchmark program sources with a glob

9a25438

Assisted-by: Claude Opus 5 via Claude Code

Fold qco.reset without casting its operand to a qubit type

2ce1ff1

Assisted-by: Opus 5 via Claude Code

Add benchmark programs for dynamically structured algorithms

ccc09f0

Assisted-by: Opus 5 via Claude Code

Add benchmark program for Shor's algorithm

30b64d9

Assisted-by: Opus 5 via Claude Code

Drop needless const for core IR values

836b58b

Assisted-by: Opus 5 via Claude Code

Add index and float constant helpers to the QC program builder

6f946b6

Assisted-by: Opus 5 via Claude Code

Use the builder constant helpers in the benchmark programs

4c8cd49

Assisted-by: Opus 5 via Claude Code

Extract the shared benchmark helpers into BenchmarkUtils.h

2d99355

Assisted-by: Opus 5 via Claude Code

denialhaag force-pushed the benchmarks-dot-jeff branch from c0882d7 to 2d99355 Compare August 18, 2026 10:06

denialhaag added 2 commits August 18, 2026 12:29

Fix linter error

2c88b42

Assisted-by: Opus 5 via Claude Code

Clean up comments

9910d2e

Assisted-by: Opus 5 via Claude Code

denialhaag changed the title ~~✨ Add generator for structured jeff benchmark programs~~ ✨ Add structured benchmark programs and a jeff generator Aug 18, 2026

denialhaag added 2 commits August 18, 2026 14:54

Drop the Cap'n Proto BUILD_TESTING workaround

b0c4ef2

Assisted-by: Opus 5 via Claude Code

Move the jeff generator header to mlir/include

18e363b

Assisted-by: Opus 5 via Claude Code

Merge origin/main into the structured benchmark foundation

6d0fed8

Assisted-by: OpenAI Codex

burgholzer changed the title ~~✨ Add structured benchmark programs and a jeff generator~~ ✨ Add typed structured quantum benchmarks Aug 23, 2026

burgholzer self-assigned this Aug 23, 2026

burgholzer removed the conflict label Aug 23, 2026

burgholzer added 4 commits August 23, 2026 23:11

🐛 Correct QFT target order

f2a38fa

Process the no-swap Fourier circuit from the most significant physical qubit and apply semiclassical feed-forward in the same order. Assisted-by: Codex (cherry picked from commit abd9ac38fcf3b7068588895e1bf16eb6725a6943)

🎨 Format structured benchmark sources

4fd6cec

Apply the repository formatter to the new BV and Fourier sources and their semantic tests. Assisted-by: Codex

📝 Complete the structured benchmark ExecPlan

71225e3

Record combined-stack, installation, wheel, documentation, test, and lint results, including the unrelated QDMI Python blocker. Assisted-by: Codex

DRovara reviewed Aug 24, 2026

View reviewed changes

burgholzer mentioned this pull request Aug 25, 2026

✨ Add expected noise-free outputs for circuits munich-quantum-toolkit/bench#924

Open

mergify Bot added the conflict label Aug 27, 2026

burgholzer added 2 commits August 27, 2026 22:47

Merge remote-tracking branch 'origin/main' into codex/structured-benc…

373b206

…hmark-foundation

📝 Address structured benchmark feedback

ae933a4

Clarify the instance/program boundary and document the dynamic Fourier methods in the executable notebook. Align the feature with current repository lint and guidance after the main merge. Assisted-by: Codex

mergify Bot removed the conflict label Aug 27, 2026

MatthiasReumann reviewed Aug 28, 2026

View reviewed changes

denialhaag commented Aug 29, 2026

View reviewed changes

denialhaag added 6 commits August 29, 2026 17:14

Fix benchmark CI portability

0414376

Assisted-by: GPT-5.6 Sol via Codex

Merge remote-tracking branch 'origin/main' into benchmarks-dot-jeff

7bbff15

Assisted-by: GPT-5.6 Sol via Codex

Rename benchmark requests to instances

cab9140

Call the resolved configuration an instance and define its generated manifest as the semantic sidecar used for evaluation. Assisted-by: GPT-5.6 Sol via Codex

Refine benchmark program structure

a35fe2f

Keep each emitter self-contained, use lowercase MLIR paths, and add recurring constant operations to the program builder. Apply the remaining C++ review cleanups in the same structural batch. Assisted-by: GPT-5.6 Sol via Codex

Polish benchmark documentation and coverage

f1507b2

Make notebook output easier to read, hide setup details, and link the generated program API. Cover the cross-platform benchmark launcher directly so Python patch coverage includes it. Assisted-by: GPT-5.6 Sol via Codex

Fix linter errors

b479653

Assisted-by: GPT-5.6 Sol via Codex

denialhaag commented Aug 29, 2026

View reviewed changes

denialhaag added 2 commits August 29, 2026 21:47

Use portable benchmark iterators

c1d8941

Name the registry iterator type explicitly instead of assuming that std::array iterators are pointers. MSVC uses a wrapper iterator type. Assisted-by: GPT-5.6 Sol via Codex

Align benchmark docs and CLI tests

a641c1b

Group benchmark CLI tests after the core CLI cases. Use QFT and QPE consistently, update constant helper comments to current Doxygen style, and keep the benchmark guide independent of its current families. Assisted-by: GPT-5.6 Sol via Codex

denialhaag mentioned this pull request Aug 29, 2026

feat: Add jeff programs for structured benchmarks unitaryfoundation/jeff#96

Draft

Avoid platform-dependent iterator types

bb4838f

Assisted-by: GPT-5.6 Sol via Codex

Uh oh!

Uh oh!

Conversation

denialhaag commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

AI notice

Checklist

Uh oh!

codecov Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

denialhaag commented Aug 18, 2026

Uh oh!

DRovara left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

denialhaag left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

burgholzer commented Aug 29, 2026

Uh oh!

denialhaag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denialhaag commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

burgholzer commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

denialhaag commented Aug 17, 2026 •

edited

Loading

codecov Bot commented Aug 17, 2026 •

edited

Loading

denialhaag left a comment •

edited

Loading

denialhaag commented Aug 29, 2026 •

edited

Loading