Skip to content

perf(metrics): per-core histogram sample storage (sequenced after the allocator spike) #924

Description

@membphis

Context

Split out of AISIX-Cloud#1259 item 3b. The worker-handle-cache PR removes the per-emit registration path; this issue tracks the OTHER metrics cost that cache cannot touch: histogram sample storage.

Every histogram record() pushes (f64, Instant) into the exporter's shared lock-free bucket. On the Graviton benchmark rig at c=128 saturation, that subtree costs ~1.98us/req across the four per-request histograms:

  • block allocation (calloc, 64-slot blocks, allocator arena contention across 4 TPC workers): ~1.19us
  • cross-core CAS on the shared bucket: ~0.43us
  • epoch machinery: ~0.15us, Instant::now per sample: ~0.13us

A spike leg that diverted samples to a thread-local ring (same (f64, Instant) tuple, registration untouched) measured the upper bound: +1.1% throughput / 1.6-2.3us/req (rps vs drift-immune flamegraph basis).

Why not folded into the handle-cache PR

  • Different mechanism and risk profile: a real per-core shard changes WHEN samples reach the exporter's storage. The unsolved design question is idle-worker flush — thread-local buffers are unreachable from the upkeep task, so either samples may sit buffered until the next request on that worker, or a cross-thread buffer registry is needed (shared state again, upkeep-only).
  • Measurement sequencing: the largest share is allocator arena contention. The planned allocator spike (item 4 scope, todo 7) will shrink exactly that share; landing this first would double-claim the same microseconds. Measure this on a post-allocator baseline; if the marginal gain drops under the noise band, closing this as not-worth-it is a legitimate outcome.

Definition of done

  • Design note answering the idle-flush question (bounded sample lateness vs buffer registry)
  • Spike on the program rig with same-session anchors, c=128
  • Go: implementation keeping series names/labels/values and summary semantics unchanged; No-go: numbers recorded here and closed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Long-tail integrations — backlogenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions