Make paper search bounded and observable

CHANGELOG.md

@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Changed
+- First-use PDF conversion and embedding now run as an observable background job; `search` returns promptly with polling instructions instead of exceeding MCP client deadlines.
+- `get_paper_info` reports queued, processing, ready, incomplete, or failed state without initializing the embedding model.
+- Search no longer blocks on an unconditional whole-library R2 pull.
+
+### Fixed
+- Direct PDF searches inspect only the requested PDF and its output directory, never unrelated sibling temp content.
+
 ## [0.3.0] - 2026-01-02
 
 ### Added

README.md

@@ -104,119 +104,50 @@ Any MCP-compatible client can use paper-intelligence:
 
 ## 📖 MCP Tools
 
-### `process_paper`
-
-<details>
-<summary>Full pipeline: Convert PDF → Index headers → Create embeddings</summary>
-
-**Parameters:**
-- `pdf_path` (string): Path to PDF file
-- `use_llm` (boolean, optional): Enhanced accuracy mode (default: false)
-- `chunk_size` (integer, optional): Text chunk size for embedding (default: 512)
-- `chunk_overlap` (integer, optional): Overlap between chunks (default: 50)
-
-**Example:**
-```
-Process the paper at ~/Downloads/attention-is-all-you-need.pdf
-```
-
-**Output Structure:**
-```
-attention-is-all-you-need/
-├── attention-is-all-you-need.md   # Converted markdown
-├── metadata.json                   # Processing version info
-├── index.json                      # Header hierarchy
-├── chroma/                         # Embeddings database
-└── images/                         # Extracted figures
-```
-
-</details>
-
 ### `search`
 
-<details>
-<summary>Unified search with grep and/or semantic RAG</summary>
+Search one or more explicitly requested PDFs, processed paper directories, or library
+directories with `grep`, `rag`, or `hybrid` mode. Direct PDF searches never inspect
+sibling files or directories.
 
 **Parameters:**
-- `query` (string): Search query (text, regex, or semantic)
-- `paper_dirs` (array): List of paper directories to search
+- `query` (string): Text, regex, or semantic query
+- `sources` (array): PDF paths, paper directories, or an explicitly selected library directory
 - `mode` (string, optional): `"grep"`, `"rag"`, or `"hybrid"` (default: hybrid)
 - `top_k` (integer, optional): Number of results (default: 5)
-- `regex` (boolean, optional): Treat query as regex (default: false)
-
-**Example Queries:**
-```
-# Semantic search across papers
-Search for "attention mechanism implementation" in my processed papers
-
-# Exact text search
-Search for "transformer" using grep mode
-
-# Regex search
-Search for "BERT|GPT|T5" with regex enabled
-```
-
-**Returns:** Results with line numbers, surrounding context, and header location.
-
-</details>
+- `regex` (boolean, optional): Treat the grep query as a regex (default: false)
 
-### `convert_pdf`
+A new or incomplete PDF is converted, indexed, and embedded in the background because
+this normally takes **1–3 minutes**, longer than the roughly 30-second deadline used by
+many MCP clients. The first call returns promptly:
 
-<details>
-<summary>Convert PDF to Markdown (without embeddings)</summary>
-
-**Parameters:**
-- `pdf_path` (string): Path to PDF file
-- `output_dir` (string, optional): Custom output directory
-- `use_llm` (boolean, optional): Enhanced accuracy mode
-
-**Returns:** `markdown_path`, `images_dir`, `image_count`
-
-</details>
-
-### `get_paper_info`
-
-<details>
-<summary>Check processing status of a paper</summary>
-
-**Parameters:**
-- `paper_dir` (string): Path to paper directory
-
-**Example Response:**
 ```json
 {
-  "has_markdown": true,
-  "has_index": true,
-  "has_embeddings": true,
-  "has_images": true,
-  "image_count": 12,
-  "version": "0.2.0",
-  "processed_at": "2025-01-15T10:30:00Z"
+  "success": true,
+  "status": "processing",
+  "message": "First-use processing ... is continuing in the background.",
+  "processing": [{
+    "paper_dir": "/path/to/paper",
+    "retry_after_seconds": 30,
+    "next_step": "Call get_paper_info ..."
+  }]
 }
 ```
 
-</details>
-
-### `index_markdown` / `embed_document`
-
-<details>
-<summary>Individual pipeline steps (for advanced use)</summary>
+Processing continues after that response. Poll `get_paper_info` using `paper_dir`; when
+it reports `status: "ready"`, retry the original search. Already-processed grep searches
+do not initialize the semantic model. RAG and hybrid searches initialize it when needed.
+Search no longer performs an unconditional remote-library sync, which previously allowed
+a local query to block for up to five minutes.
 
-**`index_markdown`** — Extract header hierarchy into searchable JSON
-```python
-index_markdown(markdown_path="~/Downloads/paper/paper.md")
-```
+### `get_paper_info`
 
-**`embed_document`** — Create embeddings for semantic search
-```python
-embed_document(
-    markdown_path="~/Downloads/paper/paper.md",
-    chunk_size=512,
-    chunk_overlap=50
-)
-```
+Check a paper's processing state without loading the embedding model. Pass either the
+paper directory returned by `search` or the original PDF path.
 
-</details>
+Statuses are `queued`, `processing`, `ready`, `incomplete`, or `failed`. Failed responses
+include the background job's error message. A ready response includes artifact presence,
+metadata, and a lightweight local chunk count when available.
 
 ## 📊 Example Output
 

paper_intelligence/server.py

@@ -1,10 +1,8 @@
 """Paper Intelligence MCP Server.
 
 Provides AI agents with efficient, searchable access to PDF documents.
-PDFs are automatically processed on first search (1-3 minutes), then
-all subsequent searches are instant.
-
-Papers are synced to/from Cloudflare R2 via rclone for shared access.
+First-use PDF processing runs in the background (normally 1-3 minutes), so
+search returns promptly with observable status instead of exceeding MCP deadlines.
 """
 
 from typing import Literal
@@ -16,9 +14,10 @@
     "paper-intelligence",
     instructions=(
         "Search PDF documents efficiently. "
-        "Pass PDF paths directly to search - they're auto-processed on first use "
-        "(takes 1-3 minutes, then instant). "
-        "Use get_paper_info to check processing status."
+        "Pass PDF paths directly to search. First use queues 1-3 minute background "
+        "processing and returns status='processing'; call get_paper_info on the returned "
+        "paper_dir until status='ready', then retry search. Searches inspect only the "
+        "sources explicitly supplied."
     ),
 )
 
@@ -36,8 +35,9 @@ def search(
 ) -> dict:
     """Search PDF documents and paper directories.
 
-    PDFs are automatically processed on first use (1-3 minutes per PDF).
-    Subsequent searches are instant.
+    First use queues background processing (normally 1-3 minutes) and returns
+    status="processing" promptly. Processing continues after the tool call returns.
+    Call get_paper_info with the returned paper_dir and retry when status="ready".
 
     Args:
         query: Search query (text, regex pattern, or semantic query)
@@ -52,13 +52,12 @@ def search(
     Returns:
         Search results with content, location, and relevance scores
     """
-    from .sync import pull
     from .tools.search import search as _search
 
-    # Pull latest from R2 before searching
-    pull()
-
-    result = _search(
+    # Search is deliberately local and bounded. The previous unconditional whole-
+    # library R2 pull could block every query for five minutes, even when all requested
+    # sources were already indexed.
+    return _search(
         query=query,
         sources=sources,
         mode=mode,
@@ -69,20 +68,13 @@ def search(
         use_llm=use_llm,
     )
 
-    # If any papers were auto-processed, push to R2
-    if result.get("processing_notes"):
-        from .sync import push
-        push()
-
-    return result
-
 
 @mcp.tool()
 def get_paper_info(paper_dir: str) -> dict:
-    """Check processing status of a paper directory.
+    """Check processing status without loading the embedding model.
 
     Args:
-        paper_dir: Path to paper directory
+        paper_dir: Paper directory, or the original PDF path
 
     Returns:
         Processing status and metadata

paper_intelligence/tools/__init__.py

@@ -1,17 +1,31 @@
-from .convert import convert_pdf
-from .embed import embed_document, query_paper
-from .index import get_header_context, index_markdown, search_headers
+"""Paper tool exports with ML-heavy modules loaded only when requested."""
+
+from importlib import import_module
+
 from .search import get_paper_info, grep_search, rag_search, search
 
+_LAZY_EXPORTS = {
+    "convert_pdf": ".convert",
+    "index_markdown": ".index",
+    "get_header_context": ".index",
+    "search_headers": ".index",
+    "embed_document": ".embed",
+    "query_paper": ".embed",
+}
+
 __all__ = [
-    "convert_pdf",
-    "index_markdown",
-    "get_header_context",
-    "search_headers",
-    "embed_document",
-    "query_paper",
+    *_LAZY_EXPORTS,
     "search",
     "grep_search",
     "rag_search",
     "get_paper_info",
 ]
+
+
+def __getattr__(name: str):
+    module_name = _LAZY_EXPORTS.get(name)
+    if module_name is None:
+        raise AttributeError(name)
+    value = getattr(import_module(module_name, __name__), name)
+    globals()[name] = value
+    return value