Add upload_heartbeat for batch uploads that reserve jobs up front - #8
Open
NicPfitzer wants to merge 14 commits into
Open
NicPfitzer wants to merge 14 commits into
NicPfitzer wants to merge 14 commits into
Conversation
A job created ahead of its upload is held for 30 minutes; if no video data arrives it is treated as abandoned and deleted. A script that reserves jobs for a batch and uploads them one at a time outlives that hold, and the jobs still waiting are deleted mid-run — every later call for them fails with a 404 that says nothing about why. The SDK's own upload paths are not exposed: create_job is immediately followed by the transfer, and once a multipart upload opens the job is no longer a candidate for expiry. This is for callers who reserve ids ahead of time, which the idempotent upload_request_id flow encourages. Also sync the contract fixture with the backend's published surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A job created ahead of its upload is held for 30 minutes; if no video data
arrives it is treated as abandoned and deleted. A script that reserves jobs for
a batch and uploads them one at a time outlives that hold, and the jobs still
waiting are deleted mid-run — every later call for them fails with a 404 that
says nothing about why.
This is what took out a 165-video web upload past the ~50th item; the same
expiry applies to API callers.
Changes
Kanopy.upload_heartbeat(job_ids)— renews the hold on jobs that are reservedbut not yet uploading. Returns
refreshedandmissing;missingids cannotbe revived, so re-create the job to retry that video.
SDK_OPERATIONSand synctests/fixtures/openapi.public.jsonwith the backend's published contract.
Scope note
The SDK's own upload paths are not exposed and are unchanged:
create_jobis immediately followed by the transfer, and once a multipart upload opens the
job is no longer a candidate for expiry. This is for callers who reserve ids
ahead of time — which the idempotent
upload_request_idflow encourages.Requires the backend to publish the operation (kanopy-backend #162).
Tests
Full suite green (34 passed), including the contract tests that bind
SDK_OPERATIONSto the reviewed public schema.🤖 Generated with Claude Code