Reconcile document uploads on a schedule

[jamiewmeldrum]

Completes the document upload lifecycle: a document registered for upload is either promoted once its file lands in S3, or removed when the file never arrives.

What's here

The reconcile itself. DocumentUploadReconciler finds AWAITING_UPLOAD documents older than the presign expiry plus a five-minute grace period, checks S3 for each key concurrently, promotes the ones that arrived to UNAPPROVED, and hard-deletes the ones that did not. Promotion is written by the transaction's dirty check rather than an explicit save.

The transaction spans the S3 calls, deliberately. That holds a pooled connection across the round trips. The job is single-threaded, so the cost is one connection, and it buys atomicity: a run either commits both the promotions and the expiries or neither. That in turn makes a run's success binary, which the 0.4 monitoring work depends on.

An S3 attempt timeout. The SDK sets none by default, so an unresponsive S3 would burn the 30s socket timeout on each of three attempts while the reconcile held that connection. S3ClientConfigurer bounds a single attempt at 10s and deliberately leaves the whole-call budget open, since a document download's duration scales with the file and an attempt's does not. It merges into the builder's existing override configuration so Micronaut's user-agent suffix survives.

Work is taken in batches, oldest first. A run takes at most practiq.document-upload-reconcile.batch-size (100) documents, ordered by created_at with an id tiebreak so the order is total and nothing at the back of the queue can starve. The remainder waits for the next run, which is safe because the two sets the run produces — promoted and expired — partition the batch: no row can survive a run it was included in, so a successful run always reduces the backlog by exactly what it examined.

The reconciler returns a summary of what it examined, promoted and expired, plus what it could not reach. remaining is derived from the page total rather than counted again, and describes the state after the run rather than before it — that is the number that says whether the batch size is undersized. The scheduler logs it, so the result is consumed rather than discarded.

A scheduler. Interval, initial delay and an on/off switch are all configurable. The initial delay matters: Micronaut passes an absent one to the scheduler as null, which becomes zero, firing a reconcile while the context is still starting.

Divergence from the plan

Slice 3 was sequenced as a run-resource on an internal HTTP surface, triggered by EventBridge. This ships an in-process @Scheduled instead — no deployed environment or Terraform exists yet to trigger it. Known cost: @Scheduled fires on every instance, so two instances would run two concurrent reconciles and collide on deleteAll against @Version-ed rows. Latent while there is one instance; it needs revisiting before autoscaling. CLAUDE.md still documents the endpoint form and wants updating.

Testing

• Four ITs drive the real entry point and prove the writes reach the database — the promotion is invisible to any unit test, since mocking the repository leaves the dirty-check flush unobservable.
• One CT proves the S3 timeout behaviourally, against a socket that accepts and never responds, rather than asserting the configured value back.
• One CT proves the schedule fires, and that the test environment's switch actually suppresses it.
• A backlog IT proves convergence: six documents against a batch of five, the five oldest taken on the first run and the remainder cleared by the second. Rows are inserted newest-first so an unordered query cannot pass by matching Postgres's physical scan order — an earlier version of this test did exactly that.
• Each of these was checked by mutation: the production change it targets was reverted and the expected test confirmed to fail.