diff --git a/benchmarks/hello/Cargo.toml b/benchmarks/hello/Cargo.toml index 15a8d55..37307eb 100644 --- a/benchmarks/hello/Cargo.toml +++ b/benchmarks/hello/Cargo.toml @@ -41,7 +41,7 @@ base-scheduler-iouring = "=0.5.0" [profile.release] debug = 1 -# Scenarios for `base-benchmark` 0.4 and `cargo bake benchmark:* --manifest-path benchmarks/hello/Cargo.toml`. Each is a multi-process scenario: the server is the subject, a service whose ready line is the address it bound, and its backends are the variants; the client is base-benchmark-load, an external program installed at a pinned version and given with `--program load=PATH`, the same for every subject. CPU placement is given per run with `--role-cpus`, so no CPU numbers appear here; each measuring host's document in ../hosts gives its placement. +# Scenarios for `base-benchmark` 0.4 and `cargo bake benchmark:* --manifest-path benchmarks/hello/Cargo.toml`. Each is a multi-process scenario: the server is the subject, a service whose ready line is the address it bound, and its backends are the variants; the client is base-benchmark-load, an external program installed at a pinned version and given with `--program load=PATH`, the same for every subject. CPU placement is given per run with `--role-cpus`, so no CPU numbers appear here; readme.md gives each measuring host's placement, as under Placement on hana. # Every client validates every response. A smoke run warns about invalid responses rather than failing on them, so the smoke arguments add `--fail-on invalid`, failing the sample instead: a correct server, raw controls included at one request outstanding per connection, never sends one. Measured runs mark the window's start 500 ms before it (`--window-lead`), rather than the client's default 100 ms, since an acknowledgement after the window's start fails the run: it tolerates five times the default's delay, though not the roughly 650 ms of one load spike, whose acknowledgement came 551 ms after the window began and which only an idle host avoids. The services' window then also covers up to 500 ms of the warmup's load, about 6% of the 8-second window, at the window's load for every subject. See readme.md. [package.metadata.base-benchmark] binary = "base-http-hello-benchmark" diff --git a/benchmarks/hello/readme.md b/benchmarks/hello/readme.md index e510024..81ccfa6 100644 --- a/benchmarks/hello/readme.md +++ b/benchmarks/hello/readme.md @@ -63,7 +63,7 @@ cargo bake benchmark:run http-closed --manifest-path benchmarks/hello/Cargo.toml ### On a measuring host -Each measuring host has a document in [benchmarks/hosts](../hosts/readme.md) that gives its hardware and the `--role-cpus` to use on it for each scenario, such as [hana's](../hosts/hana.md). Measure there with `--host`. Install the client there once; a relative `--program` path is relative to the remote home directory: +Each measuring host is described, for every Base repository that measures on it, in base-benchmark's [host documents](https://github.com/base-rs/base-benchmark/blob/main/hosts/readme.md), such as [hana's](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md). This readme records the `--role-cpus` to use on each host for each scenario, as [on hana](#placement-on-hana). Measure there with `--host`. Install the client there once; a relative `--program` path is relative to the remote home directory: ```sh ssh HOST cargo install --locked --root .cache/base-benchmark-load base-benchmark-load --version 0.1.0 @@ -77,6 +77,24 @@ cargo bake benchmark:run http-closed --host HOST --manifest-path benchmarks/hell The trees, and the libraries they depend on outside `benchmarks/hello`, are mirrored on the host with rsync, the host runs the crates.io release of the same `base-benchmark`, and the campaign comes back to `benchmarks/hello/target/base-benchmark` locally. Run prepared artifacts on the host that built them. [Measuring on another host](https://github.com/base-rs/base-benchmark/blob/main/context/usage.md#measuring-on-another-host) lists what both machines need. A recorded campaign also needs at least 10 rounds (`--rounds 10`), `--mode recorded`, a `--decision-rule`, `--idle-host true`, and committed trees. +### Placement on hana + +[hana](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md) has 4 cores and 8 threads, whose SMT siblings are CPUs `N` and `N + 4`. From the [client layout screen](results/client-layout-20261109/readme.md): + +| CPUs | Use | +| --- | --- | +| 0 and 4 | The OS, the runner (`base-benchmark`) and the client's controller when it is pinned | +| 1 and 2 | The closed-loop client's two workers, one core each | +| 3 | The server; the hello backends are single-threaded | +| 5, 6 and 7 | Idle: the client's and the server's SMT siblings | + +- `raw-closed` and `http-closed`: `--role-cpus client=1,2 --role-cpus server=3`. base-benchmark-load runs `--workers 2` within CPUs 1 and 2, its controller with them; it wakes only to mark the window. Run directly, without base-benchmark, pin them instead with `--cpus 1,2 --controller-cpu 0`, which measured the same. +- Not two workers on one core's siblings (`client=1,5`): both saturated to drive what two cores drove at two-thirds busy, and that layout changed the server's CPU per request. Not one worker: it limited `raw-closed` by 5–6%. +- The server's sibling (3,7) did not help a single-threaded backend. +- `http-rate` and `raw-rate` run one worker, and the screen did not measure them. They have used `--role-cpus client=1,5 --role-cpus server=2,3`, the client's one core with its controller on the sibling and the server on two cores, until a screen of them says otherwise. + +Change this placement only with measurements behind it. Earlier investigations used other placements, recorded with their results, such as the server on CPU 0 and the client on CPUs 2, 3, 6 and 7 in the [Linux epoll and io_uring investigation](hana-linux.md) (Linux 7.2.11 then, [results](results/hana-linux-20261027/readme.md)), the [equal offered-rate follow-up](fixed-rate.md) and the [echo benchmark](../echo/readme.md). + ### A late window start The client's measured window and the server's CPU time are tied by a handshake: during the warmup, `--window-lead` before the window begins, the client marks the window's start, and the runner snapshots the server's CPU time and acknowledges it. The client's schedule cannot wait, so an acknowledgement after the window has begun fails the sample, and the run with it, once its load is done. With the client's default 100 ms lead, a load spike on the host lost a whole campaign this way: `acknowledged the window's start 550.967 ms after the window began`. diff --git a/benchmarks/hello/results/client-layout-20261109/readme.md b/benchmarks/hello/results/client-layout-20261109/readme.md index a951a0f..28cb339 100644 --- a/benchmarks/hello/results/client-layout-20261109/readme.md +++ b/benchmarks/hello/results/client-layout-20261109/readme.md @@ -1,6 +1,6 @@ # Client layout for the closed-loop scenarios on hana -Screening evidence, gathered 9 November 2026 on [hana](../../../hosts/hana.md): which CPUs base-benchmark-load's workers should run on in `raw-closed` and `http-closed`, with the server on one CPU. It is one host, one day and six rounds, on a host shared with light services, run by the script here rather than by base-benchmark, so it chooses a placement rather than recording a performance result. +Screening evidence, gathered 9 November 2026 on [hana](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md): which CPUs base-benchmark-load's workers should run on in `raw-closed` and `http-closed`, with the server on one CPU. It is one host, one day and six rounds, on a host shared with light services, run by the script here rather than by base-benchmark, so it chooses a placement rather than recording a performance result. ## Decision @@ -70,7 +70,7 @@ Run afterwards, to check whether turbo frequency explains B's lead in `http-clos ## Conditions -- hana as its [host document](../../../hosts/hana.md) describes it on this date: Xeon E3-1240 v6, Linux 7.2.14-arch1-1, the `powersave` governor with `balance_performance` energy preference, turbo enabled. No host setting was changed. It was idle apart from its usual light services; no other benchmark ran, and every sample held the host lock, `/tmp/base-benchmark-host.lock`. +- hana as its [host document](https://github.com/base-rs/base-benchmark/blob/main/hosts/hana.md) describes it on this date: Xeon E3-1240 v6, Linux 7.2.14-arch1-1, the `powersave` governor with `balance_performance` energy preference, turbo enabled. No host setting was changed. It was idle apart from its usual light services; no other benchmark ran, and every sample held the host lock, `/tmp/base-benchmark-host.lock`. - base-http `eaa13fd`, the hello server built on hana with `cargo build --release --locked` and Rust 1.99.1 (SHA-256 `27c64823d6ff9d64f5343e4ff708e7825393440ee4432ca706fa85e098467615`). base-benchmark-load 0.1.0, `cargo install --locked` on hana (SHA-256 `3e0129da3fa1776d7d8ec714a8aed399fc71706438767e32fb128c7afb9f6adb`). - The scenarios' measured arguments: closed loop, 64 connections, `--depth 1` for the raw controls, a 2-second warmup, an 8-second window and a 500 ms `--window-lead`, validating the body, `Content-Type` and `Content-Length` of every response. Only the worker and controller placement differed. - The server was started under `taskset -c SERVER_CPUS`, fresh for every sample, and stopped with `SIGTERM` after it. The client ran under `taskset -c` its CPUs, with `--cpus` and `--controller-cpu 0` in the pinned layouts. The script itself, acting as the runner, ran on CPU 0 with the OS. The controller went on CPU 0 rather than 5, so that it was in the same place in every pinned layout and on neither a worker's CPU nor its sibling; it wakes only to mark the window. diff --git a/benchmarks/hosts/hana.md b/benchmarks/hosts/hana.md deleted file mode 100644 index 6fac5f6..0000000 --- a/benchmarks/hosts/hana.md +++ /dev/null @@ -1,75 +0,0 @@ -# hana - -Gathered 9 November 2026. hana, reached as `hana.local`, is a physical Linux server used as a measuring host. It is shared with light services, so it is quiet rather than isolated: a media server with occasional transcoding, SMB file sharing, a key-value store, a mail transfer agent, monitoring and endpoint security agents, ZFS's event daemon, SSH, and other users' sessions. Don't change its configuration, and don't use root. - -## Hardware - -| | | -| --- | --- | -| CPU | Intel Xeon E3-1240 v6 (Kaby Lake, family 6, model 158, stepping 9), microcode `0xf8`, one socket | -| Cores and threads | 4 cores, 8 threads; SMT siblings are CPUs `N` and `N + 4`: 0 and 4, 1 and 5, 2 and 6, 3 and 7 | -| Caches | 32 KiB L1 data and 32 KiB L1 instruction per core, 256 KiB L2 per core, 8 MiB L3 shared | -| Frequency | 3.7 GHz base, 4.1 GHz maximum turbo, 0.8 GHz minimum; busy CPUs ran at about 4.0 GHz with two to three cores loaded | -| Memory | 62 GiB (`MemTotal` 65,653,900 kB), 93 GiB swap; type and speed need root (`dmidecode`) and are not recorded | -| NUMA | One node, CPUs 0–7 | - -## Software and settings - -| | | -| --- | --- | -| Distribution and kernel | Arch Linux (rolling), Linux 7.2.14-arch1-1, `PREEMPT_DYNAMIC`, root on ZFS | -| Kernel command line | No CPU isolation, `nohz_full` or mitigation options: `isolated` and `nohz_full` are empty | -| Frequency scaling | `intel_pstate`, active, with hardware P-states; governor `powersave` and energy preference `balance_performance` on every CPU; turbo enabled (`no_turbo` 0), `hwp_dynamic_boost` 1 | -| SMT | On (`smt/control`) | -| Mitigations | Kernel defaults: PTI for Meltdown, IBRS for Spectre v2 and Retbleed, buffer clearing for MDS and MMIO stale data, which report "SMT vulnerable", TSX disabled; see `/sys/devices/system/cpu/vulnerabilities` | -| Network | Benchmarks use loopback: MTU 65,536, qdisc `noqueue`. Two four-queue Ethernet interfaces are bonded; their eight queue interrupts are spread one per CPU over CPUs 1–7 (CPU 4 has two), so every measuring CPU also takes the host's network interrupts. The SATA controller's interrupt is on CPU 0. Default IRQ affinity is all CPUs. | -| Network settings | `net.core.default_qdisc` `fq_codel`, `tcp_congestion_control` `cubic`, `somaxconn` 4096, `busy_poll` and `busy_read` 0 | -| Rust | 1.99.1, the distribution's package (`/usr/bin/cargo`) | - -## CPU placement - -For base-http's hello benchmark, from the [client layout screen](../hello/results/client-layout-20261109/readme.md): - -| CPUs | Use | -| --- | --- | -| 0 and 4 | The OS, the runner (`base-benchmark`) and the client's controller when it is pinned | -| 1 and 2 | The closed-loop client's two workers, one core each | -| 3 | The server; the hello backends are single-threaded | -| 5, 6 and 7 | Idle: the client's and the server's SMT siblings | - -- `raw-closed` and `http-closed`: `--role-cpus client=1,2 --role-cpus server=3`. base-benchmark-load runs `--workers 2` within CPUs 1 and 2, its controller with them; it wakes only to mark the window. Run directly, without base-benchmark, pin them instead with `--cpus 1,2 --controller-cpu 0`, which measured the same. -- Not two workers on one core's siblings (`client=1,5`): both saturated to drive what two cores drove at two-thirds busy, and that layout changed the server's CPU per request. Not one worker: it limited `raw-closed` by 5–6%. -- The server's sibling (3,7) did not help a single-threaded backend. -- `http-rate` and `raw-rate` run one worker, and the screen did not measure them. They have used `--role-cpus client=1,5 --role-cpus server=2,3`, the client's one core with its controller on the sibling and the server on two cores, until a screen of them says otherwise. - -Earlier investigations used other placements, recorded with their results, such as the server on CPU 0 and the client on CPUs 2, 3, 6 and 7 in the [Linux epoll and io_uring investigation](../hello/hana-linux.md) (Linux 7.2.11 then), the [equal offered-rate follow-up](../hello/fixed-rate.md) and the [echo benchmark](../echo/readme.md). - -## Measuring - -- Take the host lock for every measurement, `flock /tmp/base-benchmark-host.lock`, as base-benchmark does, and check `pgrep -af 'base-benchmark|wrk|hello-benchmark'` for another session's run, waiting for it rather than interfering. -- Build and prepare before a campaign, not during one, and run nothing else while measuring. -- Single rounds in the screen still ran up to 9% below their cohort's median, so interleave rounds and read ranges as well as medians. - -## Gathered with - -All without root, on 9 November 2026: - -```sh -lscpu; lscpu -e; lscpu -C -grep -m1 'model name' /proc/cpuinfo; grep -m1 microcode /proc/cpuinfo -cat /sys/devices/system/cpu/cpu[0-3]/topology/thread_siblings_list -cat /sys/devices/system/cpu/cpu0/cpufreq/{scaling_driver,scaling_governor,energy_performance_preference,cpuinfo_min_freq,cpuinfo_max_freq,base_frequency} -cat /sys/devices/system/cpu/cpu*/cpufreq/{scaling_governor,energy_performance_preference} | sort | uniq -c -cat /sys/devices/system/cpu/intel_pstate/{status,no_turbo,hwp_dynamic_boost} -cat /sys/devices/system/cpu/{smt/control,isolated,nohz_full} -grep . /sys/devices/system/cpu/vulnerabilities/* -cat /proc/cmdline; uname -a; head -4 /etc/os-release -free -h; grep MemTotal /proc/meminfo; ls /sys/devices/system/node/ -dmidecode -t memory # needs root: not readable -ip -br link; ip link show lo -cat /proc/sys/net/core/{default_qdisc,somaxconn,busy_poll,busy_read} /proc/sys/net/ipv4/tcp_congestion_control -awk 'NR == 1 || /eno|enp|ahci/' /proc/interrupts; cat /proc/irq/default_smp_affinity -systemctl list-units --type=service --state=running -rustc --version; cargo --version -grep 'cpu MHz' /proc/cpuinfo # sampled during measurement, by layout.py --frequencies -``` diff --git a/benchmarks/hosts/readme.md b/benchmarks/hosts/readme.md deleted file mode 100644 index 45cea79..0000000 --- a/benchmarks/hosts/readme.md +++ /dev/null @@ -1,13 +0,0 @@ -# Benchmark hosts - -Each host that base-http's benchmarks are measured on has one document here, named after the host, such as [hana.md](hana.md). Benchmark readmes link to it rather than repeat its hardware or CPU placement. - -A host document records: - -- what the host is and how to reach it, and what else runs on it, by kind of service, without naming people; -- its hardware: CPU model, cores, threads and the SMT sibling layout, caches, base and turbo frequency, memory size and, where readable, type and speed, and NUMA nodes; -- its software and settings: frequency scaling driver, governor and energy preference, kernel, distribution and command line, CPU vulnerability mitigations, network interfaces and their interrupt placement, and loopback settings, since the benchmarks run over loopback, and the Rust toolchain used; -- the CPU placement for client and server benchmarks: which CPUs the client's workers, its controller, the server, and the runner and OS use, which SMT siblings stay idle, and the measurements that support it; -- the date it was gathered and the commands that gathered it, all readable without root, so that it can be gathered again. - -Gather it again when the host's kernel, firmware, settings or toolchain change, and keep the previous values where results depend on them. A placement changes only with measurements behind it.