Support gungraun json format - #962
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
| Project | Bencher |
| Branch | support-gungraun-json-format |
| Testbed | intel-v1 |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result microseconds (µs) (Result Δ%) | Upper Boundary microseconds (µs) (Limit %) |
|---|---|---|---|
| Adapter::Json | 📈 view plot 🚷 view threshold | 4.66 µs(-0.53%)Baseline: 4.68 µs | 5.00 µs (93.17%) |
| Adapter::Magic (JSON) | 📈 view plot 🚷 view threshold | 4.59 µs(+1.10%)Baseline: 4.54 µs | 4.81 µs (95.52%) |
| Adapter::Magic (Rust) | 📈 view plot 🚷 view threshold | 25.74 µs(+0.01%)Baseline: 25.74 µs | 26.98 µs (95.40%) |
| Adapter::Rust | 📈 view plot 🚷 view threshold | 3.52 µs(-0.28%)Baseline: 3.53 µs | 3.93 µs (89.59%) |
| Adapter::RustBench | 📈 view plot 🚷 view threshold | 3.52 µs(-0.35%)Baseline: 3.53 µs | 3.93 µs (89.44%) |
epompeii
left a comment
There was a problem hiding this comment.
@gamma0987 thank you for the contribution!
I'm thinking we should extend the existing tree structure of the adapters:
rust_gungraun_jsonis your new JSON adapterrust_gungraun_stdoutis the existingnomparserrust_gungraunis now a parent node that attempts to parse JSON then stdout
This will be another level in the tree below: magic -> rust -> rust_gungraun.
|
Sure. No problem |
|
What changed (CLI):
UI:
|
epompeii
left a comment
There was a problem hiding this comment.
Thank you for the updates @gamma0987.
Main question is on the --output-format support.
| const RUST_GUNGRAUN_JSON_INT: i32 = 25; | ||
| const RUST_GUNGRAUN_STDOUT_INT: i32 = 26; |
There was a problem hiding this comment.
Pedantic but lets have stdout be first and json be second here. stdout is the default so IMO it make sense to list if first here and everywhere (other than the actual rust_gungraun parsing, that is).
| test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s | ||
|
|
||
|
|
||
| {"baselines":[null,null],"benchmark_exe":"/bencher/examples/rust/gungraun/target/release/deps/play_game-4a31ce0553c3b8ad","benchmark_file":"/bencher/examples/rust/gungraun/benches/play_game.rs","details":null,"function_name":"bench_play_game_100","id":"first_id","kind":"LibraryBenchmark","module_path":"play_game::bench_play_game_group::bench_play_game_100","package_dir":"/bencher/examples/rust/gungraun","profiles":[{"flamegraphs":[],"log_paths":["/callgrind.bench_play_game_100.log"],"out_paths":["/callgrind.bench_play_game_100.out"],"summaries":{"parts":[{"details":{"Left":{"command":"/bencher/examples/rust/gungraun/target/release/deps/play_game-4a31ce0553c3b8ad --gungraun-run 00000 00000 00000","details":null,"parent_pid":null,"part":1,"path":"/callgrind.bench_play_game_100.out","pid":12345,"thread":1}},"metrics_summary":{"Callgrind":{"D1MissRate":{"diffs":{"diff_pct":"0","factor":"1"},"metrics":{"Left":{"Float":0.1}}},"D1mr":{"diffs":{"diff_pct":"0","factor":"1"},"metrics":{"Left":{"Int":6}}}}}}],"total":{"regressions":[],"summary":{"Callgrind":{"D1MissRate":{"diffs":{"diff_pct":"0","factor":"1"},"metrics":{"Left":{"Float":0.1}}},"D1mr":{"diffs":{"diff_pct":"0","factor":"1"},"metrics":{"Left":{"Int":6}}}}}}},"tool":"Callgrind"},{"flamegraphs":[],"log_paths":["/dhat.bench_play_game_100.log"],"out_paths":["/dhat.bench_play_game_100.out"],"summaries":{"parts":[{"details":{"Right":{"command":"/bencher/examples/rust/gungraun/target/release/deps/play_game-4a31ce0553c3b8ad --gungraun-run 00000 00000 00000","details":null,"parent_pid":null,"part":1,"path":"/dhat.bench_play_game_100.out","pid":12345,"thread":1}},"metrics_summary":{"Dhat":{"AtTEndBlocks":{"diffs":null,"metrics":{"Left":{"Int":1}}}}}}],"total":{"regressions":[],"summary":{"Dhat":{"AtTEndBlocks":{"diffs":null,"metrics":{"Left":{"Int":1}}}}}}},"tool":"DHAT"}],"project_root":"/bencher/examples/rust/gungraun","summary_output":null,"version":"6"} |
There was a problem hiding this comment.
It seems like we are supporting the --output-format option set to json but not pretty-json?
Would it be worth going ahead and covering both of them with the rust_gungraun_json adapter? I can see this being a footgun for Bencher users where they also want a somewhat human readable JSON output in a file that they then pass to bencher.
This PR adds a gungraun JSON v6 adapter to complement the existing text-based gungraun parser, enabling users to consume machine-readable output directly.
What changed:
rust/gungraun.rs.AdapterRustGungraunJsoninlib/bencher_adapter/src/adapters/rust/gungraun_json.rsgungraun_summary::util::parse_slice()for version-aware parsing (only v6 currently)rust_gungraun_jsonadapter in addition to the existingrust_gungraunadapterDocumentation updates are intentionally deferred to a follow-up PR.
Note on benchmark naming and ids:
Already discussed in #778. This parser implementation uses (
file::group::function+::id) and the optional id as benchmark name to identify gungraun benchmarks uniquely. In addition to #778, I want to add that the main problem is that text parsed benchmark names include the "most likely" truncatedBenchmarkSummary::details. Exactly replicating the text parsed benchmark name is impossible because the user can disable or change the truncation length with configuration values. The JSON summary doesn't reflect that decision and doesn't store the user provided truncation configuration value. Required follow-up PR: A database migration to convert the old benchmark names to the new ones. Changing the old names to the new names is straightforward but irreversible although I don't think there's any need to go back.Closes #638