Skip to content

storage: add coordinated covering-index maintenance for ingestion - #2

Open
peter-olom wants to merge 2 commits into
mainfrom
fix/storage-dedup-cpu
Open

peter-olom wants to merge 2 commits into
mainfrom
fix/storage-dedup-cpu

Conversation

@peter-olom

Copy link
Copy Markdown
Owner

Trace deduplication can repeatedly scan thousands of rows sharing a trace ID because the existing index omits span ID. Add an explicit index-only maintenance operation that pauses the writer through the existing gate and builds a nonunique covering index on drop, trace and span identity. Ordinary startup does not build it; deduplication SQL, NULL semantics, durable queue receipts and FULL durability remain unchanged.

The operation checks database/temp disk headroom, skips pruning/vacuum, and preserves the existing retention deadline on success and failure. Existing scheduled retention behavior is unchanged.

Validation:

  • Existing build and 17 tests passed; no persistent tests added.
  • Disposable actual HTTP ingestion fixtures covered duplicate and NULL spans, queue/restart recovery, owner exclusion, disk rejection and scheduler success/failure.
  • Three HTTP batches into a 30,000-span trace group improved from 15.226 seconds to 73 ms mean (fixture only).
  • The staging index was built with the receiver stopped, a complete private volume backup, an exclusive owner and an eight-minute total bound. Build/checkpoint took 104.8 seconds; the original server recovered and drained its queue before the reviewed image was deployed and publicly verified healthy.

The deployed immutable tag remains cpu-20261015-225437c4. This PR retains the same two reviewed commits in normal source history. Merging main also runs the existing docs-production workflow; merge remains subject to the required Build check and independent approval.

Why:
- Deduplication scans large trace groups using a two-column index and fetches
  table rows for every span comparison, saturating the storage worker.

What:
- Add an explicitly requested index-only operation to coordinated maintenance.
- Build a nonunique covering index on drop, trace and span behind the paused
  writer and request gate, with database/temp disk and queue headroom checks.
- Preserve deduplication SQL, NULL handling, existing duplicates, receipts,
  source timestamps, FULL ingestion durability and retention scheduling.
- Document queue backpressure, safe cancellation, recovery and rollback.

Notes:
- Startup and scheduled retention do not create the index implicitly.
- Verified HTTP ingestion, durable queue replay, deduplication, full queue,
  process interruption and disk-headroom failure with disposable fixtures.
- No persistent tests were added or changed. No deployment is included.
Why:
- A failed index-only operation must not unexpectedly schedule ordinary
  pruning or vacuuming within the normal maintenance retry interval.

What:
- Restore the preexisting retention deadline after index-only success or
  failure, while keeping ordinary maintenance retry behavior unchanged.

Notes:
- Verified disk-headroom failure through the HTTP maintenance flow: zero
  deadline drift and a resumed writer. Ordinary failure still retries in 60s.
- No persistent tests were added or changed. Not published or deployed.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T09:16:45.244860Z 225437c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 225437c45f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/server/retention.ts
running = false; phase = lastError ? 'failed' : 'idle'; lastFinishedAt = Date.now();
lastDurationMs = lastStartedAt === null ? null : lastFinishedAt - lastStartedAt;
schedule(lastError || lastOperation?.complete === false ? Math.min(60_000, maintenanceConfig.interval_ms) : maintenanceConfig.interval_ms, !lastError && lastOperation?.complete !== false);
if (options.index_only) schedule(Math.max(0, scheduledAt - Date.now()), false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Persist the preserved retention deadline across restarts

When an index-only operation fails while scheduled retention is enabled, this restores the old deadline only in the current process. The subsequent persistence still records lastError and an incomplete lastOperation, while startup ignores the saved next_run_at; therefore, restarting before that deadline makes startRetentionScheduler() schedule ordinary pruning/vacuuming within at most one minute instead of preserving the prior deadline. Persist and restore enough index-operation state to distinguish this failure from a failed retention run.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant