Skip to content

github_repos: preserve cursor after partial fetch failures - #245

Open
fallintoplace wants to merge 1 commit into
ClickHouse:mainfrom
fallintoplace:fix/github-repos-partial-failure-cursor
Open

github_repos: preserve cursor after partial fetch failures#245
fallintoplace wants to merge 1 commit into
ClickHouse:mainfrom
fallintoplace:fix/github-repos-partial-failure-cursor

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Problem

github_repos uses one timestamp cursor across every configured repository. A repository fetch failure was converted to an empty list, making it indistinguishable from a successful repository with no issues.

If another repository returned a newer item, the shared cursor advanced past unseen items in the failed repository. Those items were then permanently excluded after the repository recovered.

Fix

  • distinguish a failed repository fetch from a successful empty response
  • continue returning records from healthy repositories
  • preserve the shared cursor whenever any repository fails
  • wait for a complete first fetch before establishing the initial baseline

Successful records may be fetched again while the cursor is held; inbox persistence deduplicates unchanged retries by source and ID.

Verification

Regression coverage exercises raised and returned failures, a two-run recovery where the previously missed item is recovered, and incomplete first-run baseline creation.

  • .venv/bin/python -m pytest -q tests/test_github_repos_source.py tests/test_source_filters.py: 34 passed
  • broader suite: 2940 passed

@fallintoplace
fallintoplace force-pushed the fix/github-repos-partial-failure-cursor branch from 893f5ed to b0c9b45 Compare August 1, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant