Skip to content

fix(indexing): per-input token limit & 400 bisection, pipeline stage cancellation, and persisted sync progress (#178, #179, #180) - #181

Merged
spelech merged 2 commits into
mainfrom
fix/deployment-indexing-issues-178-179-180
Nov 20, 2026
Merged

spelech merged 2 commits into
mainfrom
fix/deployment-indexing-issues-178-179-180

Conversation

@spelech

@spelech spelech commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Summary

Resolves three indexing and embedding pipeline issues found during large-repository deployment:

  1. Per-Input Token Cap & HTTP 400 Batch Bisection (#178):

    • Adds EMBEDDING_MAX_TOKENS_PER_INPUT / max_tokens_per_input setting (default 8000 tokens) across config_store.py, embeddings.py, EmbeddingSettingsRequest, .env.example, and configuration docs.
    • Pre-truncates oversized chunks exceeding max_tokens_per_input before batch slicing (truncate_text_to_token_limit) and logs the source file path.
    • On an HTTP 400 error during batch embedding (execute_api_embeddings_batch), recursively bisects the batch slice to isolate the single bad input, logs its file path, skips only that chunk (None), filters unembedded points before vector store upsert, and continues repository synchronization without aborting the sync.
  2. Pipeline Producer Cancellation & Clean Stage Thread Shutdown (#179):

    • Checks channel.is_closed (_cancelled or _error is not None) and the return value of channel.write(...) in _produce_parsed_files, _produce_local_files, and _stage2_coalescer so upstream stages stop immediately when any downstream stage fails or the sync is cancelled.
    • Joins Stage 1 (t1) and Stage 2 (t2) threads in ChannelIngestionPipeline.run()'s finally block before shutting down embed_executor (cancel_futures=True) and before sync_single_git_repo cleans up the temporary clone directory, eliminating spurious [Errno 2] No such file or directory and cannot schedule new futures after shutdown errors.
  3. Persist-Stage Progress & Error Status Retention (#180):

    • Moves processed_files updates and [n/total] Ingested <path> log lines from Stage 1 (_produce_parsed_files) to Stage 3/4 (_persist_parsed_batch) after vector store upsert and relational DB persistence complete.
    • Adds parsed_files tracking (Parsed n/total) for Stage 1 parsing progress without advancing processed_files.
    • Prevents GitProgressTracker.update_step from overwriting a job's status back to "syncing" once the job has failed (status == "error") or been cancelled.

Closes #178
Closes #179
Closes #180

Verification

  • Added unit and integration tests in tests/test_embedding_parallel_batch.py, tests/backend/test_git_syncer_progress.py, and tests/backend/test_git_progress.py covering pre-batch token truncation, HTTP 400 batch bisection and bad-chunk skipping, producer/coalescer shutdown on persist stage failure, persist-stage processed_files/Ingested counting, and error status preservation.

@spelech
spelech merged commit a0f8100 into main Nov 20, 2026
15 checks passed
@spelech
spelech deleted the fix/deployment-indexing-issues-178-179-180 branch November 20, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant