Make paper search bounded and observable

[odfalik]

Summary

• move first-use conversion/indexing/embedding to a single background worker and persist queued/processing/completed/failed state beside the paper
• return actionable status, paper_dir, polling, and retry guidance within the initial MCP call
• constrain direct-PDF discovery to that PDF and its exact output directory, never sibling temp content
• remove the unconditional whole-library R2 pull from every local search
• make status/import paths lightweight by avoiding HuggingFace/Chroma initialization and reading local chunk counts directly
• update MCP instructions/docs and regression coverage

Root causes

1. server.search synchronously ran an R2 rclone copy with a 300-second timeout before every query, including already-indexed local sources.
2. get_paper_info constructed RAGClient just to count chunks, initializing the HuggingFace embedding stack; package initializers also eagerly imported that stack.
3. Direct PDF duplicate detection iterated every directory beside the PDF, so /tmp/paper.pdf inspected unrelated temp metadata.
4. First-use processing was synchronous despite normally exceeding common MCP client deadlines.

Validation

• PYTHONPATH=$PWD /Users/oded/Repos/paper-intelligence/.venv/bin/python -m pytest -q — 39 passed
• python -m compileall -q paper_intelligence tests
• git diff --check
• cold lightweight import: ~19–33 ms, embedding stack not loaded
• ready grep/status smoke: ~0.2 ms / ~0.1 ms

Residual risk

• Semantic RAG still necessarily initializes the local embedding model on first semantic query; grep and status paths do not.
• Background work is durable at the status-file level and is resumed by a later search after a server restart, but an in-flight worker is not an external job queue.
• Remote R2 sync is no longer an implicit search-side effect; local explicit sources are now authoritative for bounded search behavior.