feat(ingestion): report parsing and chunking as stages - #146
Conversation
Source ingestion held no event sink at all. The revision status already moved through PARSING and CHUNKING, but a status says where a job is, not how long it stayed there — so an upload that spent four minutes parsing a scanned PDF and one that spent four milliseconds on a text file left exactly the same trace. Both stages are emitted under the same jobId the graph indexing stages use, so one upload reads as one operation across two processors rather than as two unrelated halves. The engine measures each window and carries both durations out on ProcessedSourceDocument. Its caller makes one process call and cannot see where parsing ended, and reporting the pair as a single duration would hide which half a slow document spent its time in — the two have unrelated causes. A semantic chunker failing over to the recursive one stays inside the chunk window, being time the document really did spend chunking. The sink is composed the same way the indexing processor composes it, through failureTolerant(composite(...)), so ingestion telemetry fails the same way and never becomes an availability dependency. A timing assertion was written for the engine and removed. Starting the chunk clock before parsing still passed every bound it could state on a fixture that small, so it proved nothing it claimed. The stages are proven end to end in the pipeline integration test instead, against a real database, through a sink contributed as an ordinary bean so the wiring is exercised and not just the emit call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 12 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
1 participant
Closes the rest of the unblocked stage gap.
Stagedeclares fourteen values; production now emits thirteen.What was invisible
Source ingestion held no event sink at all. The revision status already moved through
PARSINGandCHUNKING, but a status says where a job is, not how long it stayed there — so an upload that spent four minutes parsing a scanned PDF and one that spent four milliseconds on a text file left exactly the same trace.Shape
PARSE— one source document in, canonical blocks outCHUNK— those blocks in, chunks outBoth under the same
jobIdthe graph indexing stages already use, so one upload reads as one operation across two processors rather than two unrelated halves.The engine measures each window itself and carries both out on
ProcessedSourceDocument. Its caller makes oneprocesscall and cannot see where parsing ended; reporting the pair as one duration would hide which half a slow document spent its time in, and the two have unrelated causes — a parser handed a large scanned file, against a chunker falling back to a different algorithm. That failover stays inside the chunk window, because it is time the document really did spend chunking.The sink is composed exactly as the indexing processor composes it,
failureTolerant(composite(...)), so ingestion telemetry fails the same way and never becomes an availability dependency.A test I wrote and then deleted
I first asserted that
parseDuration + chunkDuration <= wholeCall, reasoning that two disjoint windows cannot exceed the call containing them, and that this would catch a chunk clock started before parsing.I mutated the engine to start the chunk clock at
parseStartedAt. The test still passed. On a fixture that small, parsing is fast enough that double-counting it stays under the call's own overhead — the assertion could not fail for the reason it existed.So it is gone, and the comment in its place says why rather than leaving a future reader to re-derive it. The stages are proven end to end in
SourceIngestionPipelineIntegrationTestsinstead: real Postgres, real upload, through a recorder contributed as an ordinary sink bean so the composition is exercised and not merely the emit call. It asserts chunking consumes exactly the blocks parsing produced, and that both carry the one job identifier.Confirmed the integration test actually executes rather than silently skipping:
tests="3" skipped="0".Remaining
GENERATEis the only unproduced stage left, and it is blocked on the boundary decision inchallenge-generation-telemetry.md(added in #145). Deletion/rebuild is absent from the enum entirely and is recorded as its own decision.Verification
:apps:worker:testin full, including Testcontainers. Green.🤖 Generated with Claude Code