Repository navigation
storage: add coordinated covering-index maintenance for ingestion - #2
peter-olom wants to merge 2 commits into
Conversation
Why: - Deduplication scans large trace groups using a two-column index and fetches table rows for every span comparison, saturating the storage worker. What: - Add an explicitly requested index-only operation to coordinated maintenance. - Build a nonunique covering index on drop, trace and span behind the paused writer and request gate, with database/temp disk and queue headroom checks. - Preserve deduplication SQL, NULL handling, existing duplicates, receipts, source timestamps, FULL ingestion durability and retention scheduling. - Document queue backpressure, safe cancellation, recovery and rollback. Notes: - Startup and scheduled retention do not create the index implicitly. - Verified HTTP ingestion, durable queue replay, deduplication, full queue, process interruption and disk-headroom failure with disposable fixtures. - No persistent tests were added or changed. No deployment is included.
Why: - A failed index-only operation must not unexpectedly schedule ordinary pruning or vacuuming within the normal maintenance retry interval. What: - Restore the preexisting retention deadline after index-only success or failure, while keeping ordinary maintenance retry behavior unchanged. Notes: - Verified disk-headroom failure through the HTTP maintenance flow: zero deadline drift and a resumed writer. Ordinary failure still retries in 60s. - No persistent tests were added or changed. Not published or deployed.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 225437c45f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| running = false; phase = lastError ? 'failed' : 'idle'; lastFinishedAt = Date.now(); | ||
| lastDurationMs = lastStartedAt === null ? null : lastFinishedAt - lastStartedAt; | ||
| schedule(lastError || lastOperation?.complete === false ? Math.min(60_000, maintenanceConfig.interval_ms) : maintenanceConfig.interval_ms, !lastError && lastOperation?.complete !== false); | ||
| if (options.index_only) schedule(Math.max(0, scheduledAt - Date.now()), false); |
There was a problem hiding this comment.
Persist the preserved retention deadline across restarts
When an index-only operation fails while scheduled retention is enabled, this restores the old deadline only in the current process. The subsequent persistence still records lastError and an incomplete lastOperation, while startup ignores the saved next_run_at; therefore, restarting before that deadline makes startRetentionScheduler() schedule ordinary pruning/vacuuming within at most one minute instead of preserving the prior deadline. Persist and restore enough index-operation state to distinguish this failure from a failed retention run.
Useful? React with 👍 / 👎.
Trace deduplication can repeatedly scan thousands of rows sharing a trace ID because the existing index omits span ID. Add an explicit index-only maintenance operation that pauses the writer through the existing gate and builds a nonunique covering index on drop, trace and span identity. Ordinary startup does not build it; deduplication SQL, NULL semantics, durable queue receipts and FULL durability remain unchanged.
The operation checks database/temp disk headroom, skips pruning/vacuum, and preserves the existing retention deadline on success and failure. Existing scheduled retention behavior is unchanged.
Validation:
The deployed immutable tag remains
cpu-20261015-225437c4. This PR retains the same two reviewed commits in normal source history. Merging main also runs the existing docs-production workflow; merge remains subject to the required Build check and independent approval.