Skip to content

fix(rag): allow indexing into an empty Chroma collection - #28

Open
johnnyhuy wants to merge 1 commit into
mainfrom
agent/issue-26-empty-chroma
Open

johnnyhuy wants to merge 1 commit into
mainfrom
agent/issue-26-empty-chroma

Conversation

@johnnyhuy

@johnnyhuy johnnyhuy commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Allow the first document to be indexed into an initialised, empty Chroma collection, and let searches on that collection return no matches. Check explicitly for a missing adapter instead of using Chroma's collection length as an initialisation check.

Part of #3
Closes #26

The existing child issue #26 defines this slice. The parent remains open. Five new regression cases use temporary local Chroma storage and fixed embeddings, covering empty searches with and without a channel, the first insert and retrieval, and missing adapters. The existing retrieval fixture now inserts through Wingman's public wrapper too.

Verification

All commands ran with a cleared environment, MISE_NO_ENV=1 and UV_PYTHON=3.13. After installation, UV_FROZEN=1 kept the committed lockfile fixed. No real Slack or model tokens or .env files were read or changed.

  • mise run install passed after trusting this new worktree's mise config. Installation refreshed unrelated lockfile wheel metadata, which was reverted before verification and commit.
  • mise run test-backend passed with 77 tests. Before the fix, the same suite produced 3 failures, 69 passes and 5 fixture errors, all failures and errors from the empty collection guard.
  • mise run test-frontend passed its existing placeholder command, No frontend tests yet. This is not frontend test coverage.
  • mise run test-runtime passed all 7 tests with fixtures and local listeners.
  • mise run dev-backend started on 127.0.0.1:18026, served the requests below with HTTP 200 and shut down cleanly after capture.
  • git diff --check passed. No migrations or schema changes.
tests/test_retrieval.py::test_search_empty_collection_returns_no_matches[None] PASSED
tests/test_retrieval.py::test_search_empty_collection_returns_no_matches[CTARGET] PASSED
tests/test_retrieval.py::test_first_document_can_be_indexed_and_retrieved PASSED
tests/test_retrieval.py::test_missing_adapter_still_raises[add] PASSED
tests/test_retrieval.py::test_missing_adapter_still_raises[search] PASSED
======================= 77 passed, 112 warnings in 2.65s =======================
runtime tests 7, pass 7, fail 0

Offline proof

Backend and retrieval capture succeeded. The local backend used temporary SQLite and Chroma storage with fixed embeddings. A temporary startup harness disabled settings file loading, mocked Slack authentication, blocked Slack API calls and Socket Mode startup, blocked external network connections, and replaced runtime generation with a fixture response. It used the existing module-level vector_store and rag_engine singletons. The answer text below is a fixture response, not model output.

The before-and-after transcript ran the adapter from origin/main and this branch against an empty local collection with query and document embeddings fixed at [0.0, 1.0]. Scores are squared L2 distances, where lower is nearer; they prove the first document reached the index, not retrieval quality on real support questions. A missing score means no chunk was retrieved.

{"query":"How do I connect?","fixture_embeddings":[0.0,1.0]}
{"step":"empty_search_before_fix","error":"Vector store not initialized","chunks":[],"score":null}
{"step":"first_insert_before_fix","error":"Vector store not initialized","chunks":[],"score":null}
{"step":"empty_search_after_fix","chunks":[],"score":null}
{"step":"first_insert_after_fix","chunks":[{"content":"Use the IT portal to connect.","metadata":{"source":"api"},"squared_l2_distance":0.0}]}

The curl capture then exercised a separate fresh collection through the backend, using the default chunk settings.

curl --fail --silent --show-error -w '\nHTTP %{http_code}\n' http://127.0.0.1:18026/health
# {"status":"healthy"}
# HTTP 200

curl --fail --silent --show-error -w '\nHTTP %{http_code}\n' http://127.0.0.1:18026/api/ask \
  -H 'Content-Type: application/json' \
  -d '{"question":"How do I connect?","request_id":"fixture-empty"}'
# {"answer":"Fixture response only; no model call.","sources":[],"confidence":"low"}
# HTTP 200

curl --fail --silent --show-error -w '\nHTTP %{http_code}\n' http://127.0.0.1:18026/api/documents \
  -H 'Content-Type: application/json' \
  -d '{"title":"VPN guide","content":"Use the IT portal to connect.","source":"api"}'
# {"id":1,"message":"Document added successfully"}
# HTTP 200

curl --fail --silent --show-error -w '\nHTTP %{http_code}\n' http://127.0.0.1:18026/api/ask \
  -H 'Content-Type: application/json' \
  -d '{"question":"How do I connect?","request_id":"fixture-first-document"}'
# {"answer":"Fixture response only; no model call.","sources":[{"chunk":0,"source":"api","title":"VPN guide"}],"confidence":"low"}
# HTTP 200

The runtime fixture captured the following retrieved context after the insert. Before the insert, both context and chunks were empty.

{
  "query": "How do I connect?",
  "context": "From api:\nUse the IT portal to connect.",
  "chunks": [{
    "content": "Use the IT portal to connect.",
    "metadata": {"chunk": 0, "source": "api", "title": "VPN guide"},
    "squared_l2_distance": 0.0
  }]
}

No capture path was blocked. No dashboard UI changed, so a screenshot was not applicable. Live Slack delivery and external embedding/model services remain unverified by design. No Slack messages, DMs or live Socket Mode connections were made. Local proof files are not included in the code commit; this PR body records the transcripts.

GitHub CI passed lint-and-test and all three image builds at e0006b5. CodeQL and both Socket Security checks also passed.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow the first document to be added to an empty Chroma collection

1 participant