GraphMind is an auditable RAG system for complex company questions. It combines knowledge-graph traversal in Neo4j with semantic retrieval in PostgreSQL/pgvector, routes each question to a suitable retrieval path, and validates that every returned citation resolves to evidence retrieved for that request.
Quickstart · Evaluation snapshot · Architecture · Design trade-offs · Evaluation guide
Vector-only RAG can retrieve text that looks relevant without reliably connecting facts spread across tables, filings, and corporate relationships. Questions such as “Which subsidiaries of a parent company operate in a given jurisdiction?” may require traversing several entities and combining graph relationships with passages from source documents.
GraphMind indexes SEC 10-K filings in two complementary forms:
- Knowledge graph: entities and relationships in Neo4j, queried through controlled Cypher templates.
- Vector index: text chunks in PostgreSQL/pgvector, retrieved by semantic similarity.
A router selects vector, graph, or hybrid; hybrid retrieval combines ranked results with Reciprocal Rank Fusion (RRF). The final answer is grounded in a request-scoped evidence registry rather than unrestricted model context.
| Capability | Implementation |
|---|---|
| Hybrid retrieval | Query routing across graph traversal and dense vector retrieval |
| Rank fusion | Reciprocal Rank Fusion (RRF) |
| Evidence and citations | Request-scoped [G#] graph facts and [P#] passage registry with mechanical resolution checks |
| Claim support | Separate model-based supported/unsupported check |
| Resilience | Deterministic router fallback for low-confidence or rate-limited decision calls |
| Input safety | Retrieved filing text treated as untrusted data |
| Baseline | Vector-only comparison endpoint at POST /query/baseline |
| Delivery | Terminal CLI (Click + Rich), FastAPI, React, Docker Compose, GitHub Actions |
| Automated checks | Repository currently documents 151 automated tests |
GraphMind has two distinct evaluation tracks. Keep their results separate:
- Current standardized benchmark: GraphMind Eval v1.0, 100 questions, run by
python -m eval.run_benchmark. Each execution creates its own manifest and report; see Canonical benchmark artifacts and provenance. - Historical benchmark: the earlier M8 exploratory comparison and the legacy five-system ablation runner, documented in
docs/benchmark_methodology.md. These are not the same dataset or run as Eval v1.0.
The dataset contains 100 questions across 10 technology filers (NVDA, AMD, INTC, AAPL, AVGO, MSFT, GOOGL, AMZN, MU, QCOM), with 20 questions in each of five categories. The checked-in dataset is pinned by SHA-256 (a36172b661d1f0691e8f21aff5aca2d43264aa30f14265147b171ea1e1378d2e) at eval/datasets/graphmind_eval_v1.jsonl. Schema/checksum validation does not by itself independently prove every source passage exists in the ingested corpus; see the benchmark methodology and limitations.
- Single-Hop (20 questions): Direct factual retrieval from single passages (e.g. officers, founding years, HQ).
- Two-Hop (20 questions): Relational traversals (acquisitions, shared foundries, joint customer deployments).
- Three-Hop (20 questions): Multi-hop reasoning (supply chains, career paths, competitive product ecosystems).
- Aggregation (20 questions): Multi-company synthesis (cross-filer headcounts, fabless vs foundry comparisons).
- Out-of-Scope (20 questions): Unanswerable queries (unindexed filers like Tesla/SpaceX, confidential blueprints) testing strict refusal.
| Metric / Dimension | System A (Vector Baseline) | System B (GraphMind Hybrid) | Delta / Gap | Evaluation Status |
|---|---|---|---|---|
| Overall Accuracy | 72.0% (72/100) | 76.0% (76/100) | +4.0% | N=100 paired evaluation |
| Wilson 95% Confidence Interval | [62.5%, 79.9%] | [66.8%, 83.3%] | Overlapping bounds | Single-run confidence |
| Single-Hop Accuracy (N=20) | 90.0% (18/20) | 85.0% (17/20) | -5.0% | Vector baseline superior on direct lexical lookups |
| Two-Hop Accuracy (N=20) | 70.0% (14/20) | 85.0% (17/20) | +15.0% | Hybrid graph traversal bridges relational hops |
| Three-Hop Accuracy (N=20) | 55.0% (11/20) | 70.0% (14/20) | +15.0% | Knowledge graph traverses multi-step chains |
| Aggregation / Comparison (N=20) | 65.0% (13/20) | 70.0% (14/20) | +5.0% | Multi-filer synthesis across documents |
| Out-of-Scope Abstention (N=20) | 80.0% (16/20) | 90.0% (18/20) | +10.0% | Strict refusal on unindexed companies |
| Recall@5 (Answerable Queries) | 72.5% | 85.0% | +12.5% | Gold evidence recovered in top-5 |
| Mean Reciprocal Rank (MRR) | 0.626 | 0.784 | +0.158 | Rank of first relevant evidence passage |
| Multi-Fact Evidence Coverage | 68.2% | 81.4% | +13.2% | Average fraction of required facts retrieved |
| Retrieval Failure Rate | 27.5% | 15.0% | -12.5% | Queries with zero relevant evidence retrieved |
| Citation Resolution Rate | 92.0% | 100.0% | +8.0% | Invariant check: all cited labels in registry |
| Latency (p50 / p95) | 6,559 ms / 66,668 ms | 7,141 ms / 46,246 ms | Comparable | Governed by upstream LLM generation |
| Measured Cost per Query | $0.000000 | $0.000000 | $0.00 | Free-tier open-access models |
- Discordant Pairs: 11 Hybrid Wins vs 7 Baseline Wins (Discordant Total: 18).
-
Exact Two-Sided McNemar Test:
$p = 0.4807$ ($p \ge 0.05$ ). -
Paired Bootstrap 95% Confidence Interval (2,000 resamples):
[-4.0%, +12.0%]. -
Statistical Conclusion: While GraphMind Hybrid demonstrates consistent double-digit gains on relational multi-hop reasoning (+15% on 2-hop and 3-hop questions), the aggregate overall accuracy gap (+4.0%) does not reach statistical significance (
$p \ge 0.05$ ). GraphMind should be understood as specialized for complex multi-entity graph queries rather than universally superior across all factual lookup queries.
- The Graph Traversal Trap: On questions where the router selects
graphbut returns partial or unhelpful graph facts (e.g., finding a founder node without executive titles), the system does not fall back to vector retrieval becausegraph_pathsis non-empty. This starves the generator of rich descriptive text found in Item 1. - Entity Ambiguity Roadblocks: Ambiguity guardrails on queries like "Mellanox Technologies" can halt generation with clarification prompts, whereas plain vector search directly surfaces the correct acquisition text.
- Lexical Strength of Dense Embeddings: On focused SEC filings, pgvector dense search is highly effective for single-hop queries; graph retrieval adds overhead without accuracy gains unless multi-hop hops span separated sections.
The standardized runner writes one self-contained artifact set per run:
| Artifact | Purpose |
|---|---|
eval/results/<run_id>/manifest.json |
Source of truth for run ID, Git SHA, dataset checksum, configuration, timestamps, and completion status |
eval/results/<run_id>/comparison.md |
Canonical human-readable report for that exact run |
eval/results/<run_id>/metrics.json |
Machine-readable aggregate metrics and paired analysis |
eval/results/<run_id>/question_results.jsonl |
Per-question, per-system records |
eval/results/<run_id>/failures.md |
Recorded failures and representative examples |
eval/results/<run_id>/reproducibility.md |
Exact reproduction instructions |
Provenance note: the numbers in the summary table below are the benchmark figures currently documented in this README. The repository tree does not currently contain a checked-in Eval v1.0 run directory, so these figures cannot yet be linked here to a specific committed run manifest. Treat them as reported results until the matching manifest.json, comparison.md, metrics.json, and per-question output are archived and linked from this section. Do not invent a run ID or claim that a run is reproducible from the README alone.
For a result to be considered fully auditable, publish or otherwise preserve the complete artifact directory and record the exact run ID and Git SHA here. The dataset manifest proves dataset-file integrity; it does not alone prove that the model evaluations were completed successfully or that every gold passage was verified against the live corpus.
Historical only; this is a separate dataset/configuration and must not be combined with Eval v1.0. See docs/benchmark_methodology.md for the methodology and discrepancy audit. The canonical report for a new 100-question run is generated at eval/results/<run_id>/comparison.md, alongside its manifest.json.
| System ID | Architecture | Accuracy | Wilson 95% CI | Citation Resolution | Latency (p50 / p95) |
|---|---|---|---|---|---|
| System A | Vector Baseline | 72.0% (18/25) | [52.4%, 85.7%] | 92.0% | 11,809 ms / 120,313 ms |
| System B | Graph-Only | 50.0% (4/8) | [21.5%, 78.5%] | 100.0% | 5,103 ms / 11,258 ms |
| System C | Hybrid Auto-Routed (GraphMind) | 88.0% (22/25) | [70.0%, 95.8%] | 100.0% | 7,339 ms / 46,246 ms |
| System D | Hybrid-Always | 87.5% (7/8) | [52.9%, 97.8%] | 100.0% | 3,222 ms / 7,147 ms |
| System E | Oracle Diagnostic | 87.5% (7/8) | [52.9%, 97.8%] | 100.0% | 2,445 ms / 6,520 ms |
Historical Statistical Note (M8): 4 Hybrid wins vs 0 Baseline wins on N=25 yielded an exact McNemar
Use the standardized runner for Eval v1.0. Do not use the legacy M8/M10 runner to reproduce the table above.
# 1. Validate the pinned dataset, checksum, and category distribution (no model calls)
python -m eval.run_benchmark --mode dry-run
# 2. Run a small smoke test (not a benchmark result)
python -m eval.run_benchmark --mode smoke
# 3. Run the full comparison: same 100 questions for vector baseline and hybrid
python -m eval.run_benchmark --dataset eval/datasets/graphmind_eval_v1.jsonl --mode full --expected-count 100
# 4. Test the benchmark runner and adversarial citation checks
pytest tests/unit/test_benchmark_100q.py tests/unit/test_adversarial_citations.py -vThe full run should create an immutable directory at eval/results/<run_id>/. Its manifest.json identifies the run, dataset checksum, Git commit and configuration; comparison.md is the canonical report for that run; metrics.json and question_results.jsonl contain the aggregate and per-question evidence. Use the exact same run directory when citing results—never infer a run ID or combine files from different runs.
flowchart TD
U(["Question"]) --> CLI["Terminal CLI / Web UI"]
CLI --> API["FastAPI / Local Engine"]
API --> R["Router and guardrails"]
R -->|graph or hybrid| G[("Neo4j knowledge graph")]
R -->|vector or hybrid| V[("PostgreSQL + pgvector")]
G --> F["Reciprocal Rank Fusion"]
V --> F
F --> E["Request-scoped evidence registry"]
E --> L["Grounded answer generation"]
L --> C["Citation resolution validator"]
C --> S["Answer with resolvable citations"]
S --> CLI
R -. low confidence / 429 .-> RR["Deterministic fallback router"]
RR --> G
RR --> V
- Route: classify the query as
vector,graph, orhybrid. Low-confidence or rate-limited routing can fall back to deterministic rules. - Retrieve: graph queries use controlled Cypher templates; semantic queries retrieve passages from pgvector; hybrid requests use both.
- Fuse and register evidence: combine ranked results and assign request-local labels to graph facts and passages.
- Generate: produce an answer grounded in the retrieved evidence.
- Validate: mechanically check that citation labels resolve to evidence available in this request. A separate model-based check assesses whether claims are supported by their citations.
- Return: provide the answer, route information, and citation references through the API.
No hosted demo or video recording is linked here yet. You can run the application locally and reproduce the documented evaluation instead of relying on screenshots or canned answers.
Requirements: Docker with Compose, Python 3.11+, and an OpenRouter API key for model-backed requests.
git clone https://github.com/Git-Kapish/GraphMind.git
cd GraphMind
cp .env.example .env
# Add OPENROUTER_API_KEY to .env
# 1. Start backend databases (PostgreSQL/pgvector + Neo4j)
docker compose up -d postgres neo4j
# 2. Query immediately with GraphMind Terminal CLI (Zero-setup / Local in-process)
uv run graphmind interactiveGraphMind features an enterprise terminal interface (graphmind or python -m kgrag.cli) powered by Click and Rich:
# 1. Interactive terminal REPL shell (with dynamic mode switching, slash commands & live status)
uv run graphmind interactive
# 2. One-shot grounded query with formatted markdown, citation tables, and routing badge
uv run graphmind query "What does Qualcomm say about customer concentration or Apple purchasing modem products?"
# 3. Side-by-side A/B comparison: GraphMind Hybrid vs Plain Vector Baseline
uv run graphmind compare "Who are the key executive officers of NVIDIA Corporation?"
# 4. Canonical entity topology & graph connection inspector
uv run graphmind entity "Apple Inc."
# 5. Authoritative SEC chunk text and extraction observations
uv run graphmind chunk "0000804328-25-000085:item1a:0003"
# 6. Live database and model runtime health check
uv run graphmind health
# 7. Machine-readable JSON output for automated scripting / CI
uv run graphmind query "What is Broadcom's principal business?" --jsonServices & Endpoints:
| Interface / Service | Access |
|---|---|
| GraphMind Terminal CLI | uv run graphmind |
| FastAPI Swagger UI | http://localhost:8000/docs |
| API health check | http://localhost:8000/health |
| Neo4j Browser | http://localhost:7474 |
Ask a question through the API:
curl -s -X POST http://localhost:8000/query \
-H "Content-Type: application/json" \
-d '{"question":"Who are the executive officers of NVIDIA Corporation?","mode":"auto"}'For a fair manual comparison, submit the same question with "mode":"vector" and "mode":"hybrid". For the dedicated vector-only baseline endpoint, use POST /query/baseline. Compare actual responses, route, evidence, citation behavior, and latency; model-backed results can vary.
Stop the services:
docker compose downThe following commands invoke the separate legacy evaluation path. They are useful for reproducing historical experiments, not for reproducing the standardized 100-question table above. Their output locations and methodology are described in docs/benchmark_methodology.md.
# Historical held-out split: vector baseline vs auto-routed hybrid
python -m kgrag.benchmarks.eval --systems A,C --split test
# Historical five-system ablation
python -m kgrag.benchmarks.eval --systems A,B,C,D,E --split test
# Historical runner's timestamped output
python -m kgrag.benchmarks.eval --systems A,C --split test --output-run-dir benchmarks/results/runsDo not interpret the two-question M10 smoke run as the 25-question M8 evaluation, and do not merge either historical result with Eval v1.0.
| Decision | Why | Cost or limitation |
|---|---|---|
| Graph + vector instead of vector-only | Graph traversal can express explicit relationships; vector search handles narrative text | More complex ingestion, entity resolution, and two stores to maintain |
| Route selectively instead of always running both | Avoid unnecessary work for simpler questions | Routing errors and fallback behavior require testing |
| Reciprocal Rank Fusion | Combines differently scored ranked lists without requiring score calibration | Adds retrieval and merge complexity |
| Request-scoped evidence registry | Citations must resolve to retrieved facts/passages | Citation formatting failures may require regeneration |
| Mechanical citation checks plus claim-support evaluation | Citation existence and citation relevance are different properties | Claim-support evaluation adds latency and relies on another model judgment |
| Free-tier model providers | Low-cost experimentation | Rate limits and model availability are external dependencies |
| Layer | Technology |
|---|---|
| Language / API | Python 3.12, FastAPI |
| Graph database | Neo4j 5.26, APOC |
| Vector database | PostgreSQL + pgvector, HNSW |
| Frontend | React 19, Vite |
| Models | OpenRouter model endpoints documented in source configuration |
| Tests and linting | pytest, Ruff, Oxlint |
| Delivery | Docker, Docker Compose, GitHub Actions |
The repository documents these commands:
pytest -v
python test_e2e.py
ruff check . && ruff format --check .
cd frontend && npm run lint && npm run buildThe GitHub Actions workflow is at .github/workflows/ci.yml. It runs backend quality checks, tests against PostgreSQL/pgvector and Neo4j service containers, the frontend production build, and Docker builds. The live end-to-end check requires configured services and model-provider credentials.
GraphMind/
├── src/kgrag/ # API, retrieval, graph, vector, and benchmark code
├── frontend/ # React + Vite dashboard
├── benchmarks/ # datasets and benchmark artifacts
├── docs/ # design, benchmark methodology, and reports
├── tests/ # unit and integration tests
├── .github/workflows/ # CI
├── Dockerfile
├── docker-compose.yml
└── PRD.md
- Evaluation provenance: the standardized 100-question result summary is documented above, but its exact generated run artifacts are not currently checked into the repository. Archive and link the matching per-run manifest and outputs before treating the table as independently reproducible.
- Gold evidence audit: dataset schema and checksum tests establish structural integrity, not independent verification that every annotated source passage exists in the ingested corpus or that each multi-hop question requires all annotated facts.
- Scoring validity: answer correctness uses automated token/entity heuristics; these are inexpensive but can mis-score paraphrases or partially correct answers. Validate a representative sample manually or with an independently calibrated judge before making stronger accuracy claims.
- Corpus scale: 789 text chunks and 331 graph entities are suitable for auditing the prototype, but do not establish behavior on a large enterprise corpus.
- Provider dependence: the deterministic fallback applies to routing; embedding and generation still depend on upstream model availability.
- Operational readiness: authentication and multi-user access controls, larger-corpus ingestion performance, and incremental graph updates remain potential next steps.
- Latency: the preliminary p95 values are high for interactive use, despite the lower hybrid p50. Measure and optimize tail latency before positioning this as a production service.
Built by Kapish, final-year Electronics and Communication Engineering student at Delhi Technological University (Class of 2027). GitHub profile.
MIT. See LICENSE.