Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/hello/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ base-scheduler-iouring = "=0.5.0"
[profile.release]
debug = 1

# Scenarios for `base-benchmark` 0.4 and `cargo bake benchmark:* --manifest-path benchmarks/hello/Cargo.toml`. Each is a multi-process scenario: the server is the subject, a service whose ready line is the address it bound, and its backends are the variants; the client is base-benchmark-load, an external program installed at a pinned version and given with `--program load=PATH`, the same for every subject. CPU placement is given per run with `--role-cpus`, so no CPU numbers appear here; each measuring host's document in ../hosts gives its placement.
# Scenarios for `base-benchmark` 0.4 and `cargo bake benchmark:* --manifest-path benchmarks/hello/Cargo.toml`. Each is a multi-process scenario: the server is the subject, a service whose ready line is the address it bound, and its backends are the variants; the client is base-benchmark-load, an external program installed at a pinned version and given with `--program load=PATH`, the same for every subject. CPU placement is given per run with `--role-cpus`, so no CPU numbers appear here; readme.md gives each measuring host's placement, as under Placement on hana.
# Every client validates every response. A smoke run warns about invalid responses rather than failing on them, so the smoke arguments add `--fail-on invalid`, failing the sample instead: a correct server, raw controls included at one request outstanding per connection, never sends one. Measured runs mark the window's start 500 ms before it (`--window-lead`), rather than the client's default 100 ms, since an acknowledgement after the window's start fails the run: it tolerates five times the default's delay, though not the roughly 650 ms of one load spike, whose acknowledgement came 551 ms after the window began and which only an idle host avoids. The services' window then also covers up to 500 ms of the warmup's load, about 6% of the 8-second window, at the window's load for every subject. See readme.md.
[package.metadata.base-benchmark]
binary = "base-http-hello-benchmark"
Expand Down
20 changes: 19 additions & 1 deletion benchmarks/hello/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ cargo bake benchmark:run http-closed --manifest-path benchmarks/hello/Cargo.toml

### On a measuring host

Each measuring host has a document in [benchmarks/hosts](../hosts/readme.md) that gives its hardware and the `--role-cpus` to use on it for each scenario, such as [hana's](../hosts/hana.md). Measure there with `--host`. Install the client there once; a relative `--program` path is relative to the remote home directory:
Each measuring host is described, for every Base repository that measures on it, in base-benchmark's [host documents](https://github.com/base-rs/base-benchmark/blob/main/hosts/readme.md), such as [hana's](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md). This readme records the `--role-cpus` to use on each host for each scenario, as [on hana](#placement-on-hana). Measure there with `--host`. Install the client there once; a relative `--program` path is relative to the remote home directory:

```sh
ssh HOST cargo install --locked --root .cache/base-benchmark-load base-benchmark-load --version 0.1.0
Expand All @@ -77,6 +77,24 @@ cargo bake benchmark:run http-closed --host HOST --manifest-path benchmarks/hell

The trees, and the libraries they depend on outside `benchmarks/hello`, are mirrored on the host with rsync, the host runs the crates.io release of the same `base-benchmark`, and the campaign comes back to `benchmarks/hello/target/base-benchmark` locally. Run prepared artifacts on the host that built them. [Measuring on another host](https://github.com/base-rs/base-benchmark/blob/main/context/usage.md#measuring-on-another-host) lists what both machines need. A recorded campaign also needs at least 10 rounds (`--rounds 10`), `--mode recorded`, a `--decision-rule`, `--idle-host true`, and committed trees.

### Placement on hana

[hana](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md) has 4 cores and 8 threads, whose SMT siblings are CPUs `N` and `N + 4`. From the [client layout screen](results/client-layout-20261109/readme.md):

| CPUs | Use |
| --- | --- |
| 0 and 4 | The OS, the runner (`base-benchmark`) and the client's controller when it is pinned |
| 1 and 2 | The closed-loop client's two workers, one core each |
| 3 | The server; the hello backends are single-threaded |
| 5, 6 and 7 | Idle: the client's and the server's SMT siblings |

- `raw-closed` and `http-closed`: `--role-cpus client=1,2 --role-cpus server=3`. base-benchmark-load runs `--workers 2` within CPUs 1 and 2, its controller with them; it wakes only to mark the window. Run directly, without base-benchmark, pin them instead with `--cpus 1,2 --controller-cpu 0`, which measured the same.
- Not two workers on one core's siblings (`client=1,5`): both saturated to drive what two cores drove at two-thirds busy, and that layout changed the server's CPU per request. Not one worker: it limited `raw-closed` by 5–6%.
- The server's sibling (3,7) did not help a single-threaded backend.
- `http-rate` and `raw-rate` run one worker, and the screen did not measure them. They have used `--role-cpus client=1,5 --role-cpus server=2,3`, the client's one core with its controller on the sibling and the server on two cores, until a screen of them says otherwise.

Change this placement only with measurements behind it. Earlier investigations used other placements, recorded with their results, such as the server on CPU 0 and the client on CPUs 2, 3, 6 and 7 in the [Linux epoll and io_uring investigation](hana-linux.md) (Linux 7.2.11 then, [results](results/hana-linux-20261027/readme.md)), the [equal offered-rate follow-up](fixed-rate.md) and the [echo benchmark](../echo/readme.md).

### A late window start

The client's measured window and the server's CPU time are tied by a handshake: during the warmup, `--window-lead` before the window begins, the client marks the window's start, and the runner snapshots the server's CPU time and acknowledges it. The client's schedule cannot wait, so an acknowledgement after the window has begun fails the sample, and the run with it, once its load is done. With the client's default 100 ms lead, a load spike on the host lost a whole campaign this way: `acknowledged the window's start 550.967 ms after the window began`.
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/hello/results/client-layout-20261109/readme.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Client layout for the closed-loop scenarios on hana

Screening evidence, gathered 9 November 2026 on [hana](../../../hosts/hana.md): which CPUs base-benchmark-load's workers should run on in `raw-closed` and `http-closed`, with the server on one CPU. It is one host, one day and six rounds, on a host shared with light services, run by the script here rather than by base-benchmark, so it chooses a placement rather than recording a performance result.
Screening evidence, gathered 9 November 2026 on [hana](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md): which CPUs base-benchmark-load's workers should run on in `raw-closed` and `http-closed`, with the server on one CPU. It is one host, one day and six rounds, on a host shared with light services, run by the script here rather than by base-benchmark, so it chooses a placement rather than recording a performance result.

## Decision

Expand Down Expand Up @@ -70,7 +70,7 @@ Run afterwards, to check whether turbo frequency explains B's lead in `http-clos

## Conditions

- hana as its [host document](../../../hosts/hana.md) describes it on this date: Xeon E3-1240 v6, Linux 7.2.14-arch1-1, the `powersave` governor with `balance_performance` energy preference, turbo enabled. No host setting was changed. It was idle apart from its usual light services; no other benchmark ran, and every sample held the host lock, `/tmp/base-benchmark-host.lock`.
- hana as its [host document](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md) describes it on this date: Xeon E3-1240 v6, Linux 7.2.14-arch1-1, the `powersave` governor with `balance_performance` energy preference, turbo enabled. No host setting was changed. It was idle apart from its usual light services; no other benchmark ran, and every sample held the host lock, `/tmp/base-benchmark-host.lock`.
- base-http `eaa13fd`, the hello server built on hana with `cargo build --release --locked` and Rust 1.99.1 (SHA-256 `27c64823d6ff9d64f5343e4ff708e7825393440ee4432ca706fa85e098467615`). base-benchmark-load 0.1.0, `cargo install --locked` on hana (SHA-256 `3e0129da3fa1776d7d8ec714a8aed399fc71706438767e32fb128c7afb9f6adb`).
- The scenarios' measured arguments: closed loop, 64 connections, `--depth 1` for the raw controls, a 2-second warmup, an 8-second window and a 500 ms `--window-lead`, validating the body, `Content-Type` and `Content-Length` of every response. Only the worker and controller placement differed.
- The server was started under `taskset -c SERVER_CPUS`, fresh for every sample, and stopped with `SIGTERM` after it. The client ran under `taskset -c` its CPUs, with `--cpus` and `--controller-cpu 0` in the pinned layouts. The script itself, acting as the runner, ran on CPU 0 with the OS. The controller went on CPU 0 rather than 5, so that it was in the same place in every pinned layout and on neither a worker's CPU nor its sibling; it wakes only to mark the window.
Expand Down
75 changes: 0 additions & 75 deletions benchmarks/hosts/hana.md

This file was deleted.

13 changes: 0 additions & 13 deletions benchmarks/hosts/readme.md

This file was deleted.

Loading