Keep first-use PDF processing within MCP deadlines

[odfalik]

Summary

• move first-use PDF conversion/indexing/embedding into a detached subprocess so the MCP request loop can return processing state immediately and remain responsive to get_paper_info
• remove the synchronous Torch import from direct-PDF path resolution
• bound collection-wide hybrid search: grep remains collection-wide, while semantic fanout over more than 20 papers returns an explicit partial status; scoped RAG reuses one embedding model
• keep direct PDFs isolated from siblings and tolerate unreadable children of an explicitly selected library

Root cause

Pi has no requestTimeoutMs override for paper-intelligence, so pi-mcp-adapter uses its MCP client SDK default: 60,000 ms (DEFAULT_REQUEST_TIMEOUT_MSEC). The canonical 2026-09-22 call failed at exactly 60 seconds.

The v0.5.0 background-thread fix still imported paper_intelligence.tools.convert synchronously merely to derive the output directory; that module imported Torch at import time. It then ran Marker/Torch/embedding work in a thread inside the MCP server process, allowing heavy initialization to keep both the original response and subsequent status requests from being serviced before the client deadline.

The local library currently has 107 indexed paper directories. The old RAG loop constructed a new RAGClient (and therefore a new HuggingFace embedding model) for every directory, so whole-library hybrid search was also unbounded.

Verification

• python -m pytest tests/ -q --ignore=tests/test_integration.py — 19 passed
• python -m pytest -q before the final import-path-only refinement — 43 passed
• focused real-PDF conversion + detached first-use integration after that refinement — 2 passed; the integration asserts the initial response arrives in under 5 seconds and waits for successful background completion
• direct PDF discovery benchmark — 0.0022 s, Torch not imported
• read-only grep scan of all 107 local papers with no match — 0.07 s
• clean wheel build includes paper_intelligence/processing_worker.py