Final Edit - #1
Draft
vardhjain wants to merge 29 commits into
Draft
Conversation
- Both GraphRAG.ipynb and Plain_RAG.ipynb are now fully self-contained (no shared_utils import dependency) with all shared constants, prompts, FuzzyEvaluator, Evaluator, and call_ollama defined inline identically - GraphRAG: credentials via Colab Secrets (ARANGO_PASS) with env var fallback - Plain_RAG: GPU-aware FAISS with faiss-gpu-cu12 swap instruction - Add Comparison.ipynb for side-by-side accuracy/F1/latency/confusion charts - Add shared_utils.py for local script runs - Add run_graphrag.py, run_plainrag.py, run_comparison.py standalone scripts
The Ollama install script requires zstd for extraction (new in recent versions). Added `apt-get install -y zstd -q` before the curl install. Also expanded ensure_ollama() to search multiple candidate paths and fall back to a filesystem find rather than hard-coding /usr/local/bin. Both Plain_RAG and GraphRAG notebooks updated identically.
Implements the agreed revamp: an importable src/kgqa package, a 4-arm ablation (plain -> plain_rr -> graph -> graph_concepts), and the fairness fixes from the audit. Science fixes: - Single shared ChunkStore (identical corpus + per-section chunking across arms) - Reranker promoted to its own arm so the graph never gets it as a hidden edge - Label leakage removed: ingestion stores no question-derived title and no final_decision; graph context uses generic "=== STUDY n ===" labels - MeSH Concepts/MENTIONS now used via a concept-hop arm (graph_concepts) - Seeded random sampling (n=200) + paired McNemar significance test - Fixed the NameError in the graph-expansion fallback Repo hygiene: - src/ package, scripts/ (ingest, run_benchmark, compare), thin Colab notebooks - tests/ (17 pytest cases, CPU-only via fakes), ruff config, GitHub Actions CI - README, requirements, .gitignore, LICENSE (MIT), .env.example - Docs (PDF/PPTX) moved to docs/; .DS_Store untracked; superseded files removed
- concept-hop AQL now ranks neighbours by shared-concept count first and reconstructs abstracts only for the top-N (was building an abstract for every candidate on every query — 200x on a full run) - remove faiss-cpu: PlainRAG now uses the shared numpy-cosine ChunkStore
- default ARANGO_HOST -> new Oasis deployment (581c546a8d66), overridable via env - notebooks request A100 GPU + High-RAM; add nvidia-smi check and a labeled-only ingest smoke test before the full run
A single Ollama 500/timeout previously aborted an entire arm, and since the server is shared across arms one crash cascaded into all four failing. - run_benchmark: per-question retry (3x) with automatic Ollama restart between attempts; checkpoint results every 25 questions; the script now owns Ollama health (health-check + (re)start + warm) instead of relying on a one-shot start - llm: cap generation via num_predict and keep the model resident (keep_alive), so a runaway reasoning chain can't stall/crash the server - config: LLM_NUM_CTX / LLM_NUM_PREDICT / LLM_KEEP_ALIVE / LLM_TIMEOUT now env-tunable (shrink on small-VRAM GPUs); defaults 4096 / 1024 / 30m / 180s - notebook: drop --no-ollama-start so the runner can self-heal; add a GPU-memory tuning hint
- clone cell now %cd /content + rm -rf before clone, so re-running can't nest a second checkout (caused a doubled results path) - benchmark secrets cell sets LLM_NUM_CTX=8192 / LLM_NUM_PREDICT=1024 for the 80GB A100: full graph context, bounded generation (~halves runtime; identical across arms so the comparison is unaffected)
n=200, seed 42, deepseek-r1:8b on A100. Parent-document expansion is the decisive win (plain_rr -> graph +22.5pp, McNemar p<0.0001); reranker +7pp (n.s.); concept-hop -2pp (n.s.) at 5x latency. plain/plain_rr fall below the majority baseline — context sufficiency, which the graph supplies, dominates.
- results/ablation.png + results/summary.md generated from the n=200 run - revert notebooks/02_benchmark.ipynb to the clean (output-free) wrapper that Colab's "Created using Colab" save had filled with execution outputs
Bring the repo up to industry/community standards for a public release: - Makefile (install/test/lint/ingest/benchmark/compare/clean) + `make help` - CONTRIBUTING, CODE_OF_CONDUCT, SECURITY, CHANGELOG, CITATION.cff - .github/ ISSUE_TEMPLATE (bug, feature, config) + PULL_REQUEST_TEMPLATE - assets/architecture.svg + README badges and embedded diagram - .pre-commit-config.yaml (ruff + hygiene hooks), .editorconfig - .gitattributes to enforce LF (Makefile-safe) and tidy linguist stats - richer pyproject metadata (authors, urls, classifiers, keywords, dev extras) Deliberately no configs/ dir: configuration lives in src/kgqa/config.py (typed + env-overridable), which is documented in CONTRIBUTING.
So anyone can run it out of the box: - default ARANGO_HOST is now http://localhost:8529 (no specific deployment baked in) - add docker-compose.yml for a one-command local ArangoDB - .env.example and README document both paths (local Docker / cloud Oasis) - notebooks read ARANGO_HOST + ARANGO_PASS from Colab Secrets (nothing hardcoded) and clone the main branch - README setup/run instructions generalized
vardhjain
marked this pull request as draft
June 13, 2026 16:06
- app/chat_app.py: live GraphRAG chat over the winning `graph` arm; answers cite source PubMed IDs (--share for a public link, --concepts for the concept arm) - app/dashboard.py: Streamlit dashboard of the ablation (bars, McNemar, per-class confusion matrices when raw results are present); reads results/ only, so it deploys to Streamlit Cloud - BaseRetriever.chat(): conversational answer + retrieved source pubids (tested) - compare.py now also emits results/summary.json (structured metrics) for the dashboard - requirements-app.txt, make chat/dashboard/install-app, app/ added to ruff + CI - README "Interactive demo & dashboard" section + app/README deployment notes
The dashboard is the hosted public demo (no LLM/DB/GPU — reads results/ only). The chat stays local/Colab (it needs a GPU + a persistent ArangoDB). - app/requirements.txt: light deploy deps (streamlit/pandas/scikit-learn). Streamlit Cloud reads the entrypoint's dir first, so this is used and the heavy root requirements.txt is ignored for the deploy — benchmark users keep their full deps. - .streamlit/config.toml: clean light theme matching the ablation figure. - dashboard: richer set_page_config (icon, About/menu links, methodology expander); per-class section degrades cleanly when raw results are absent. - README: "Open in Streamlit" badge + "Live demo" section; app/README has the exact Streamlit Cloud deploy steps.
Match the presentation style of the EthicLens repo — centered title with emoji + tagline, a badge row, a quick-nav line (live demo / results / why / setup), and emoji on section headers. Nav-target headings kept plain so anchors stay stable.
- new unit tests for data sampling, the Ollama client, evaluation report/save, and ChunkStore.from_dataset (mocking the datasets/requests boundaries) — 18 -> 24 tests - CI runs pytest --cov and uploads to Codecov; README gets a coverage badge - pytest-cov added to dev deps; coverage config in pyproject
- docs/index.md + docs/_config.yml (Jekyll Cayman theme): a landing page with the architecture diagram, results table, ablation figure, honest findings, and links to the demo / report / slides - README: docs badge + a Documentation section Enable via Settings -> Pages -> Deploy from branch -> main -> /docs.
- assets/dashboard.png: real screenshot of the running dashboard, embedded (clickable) in the README Live demo section - dashboard: group the accuracy/macro-F1 bars (were misleadingly stacked) and replace the redundant figure with a horizontal latency-by-arm chart - bump deploy pin to streamlit>=1.39 (grouped/horizontal bar options)
- assets/chat.png: the Gradio chat interface (title, description, example questions), embedded in the README chat section - chat_app.py: drop the 'theme' kwarg — gr.ChatInterface no longer accepts it on Gradio 5/6 (requirements pin gradio>=4), so a fresh install would have crashed
The app deployed to Streamlit's auto-generated subdomain rather than the custom kgqa-ablation; repoint the README badge/nav/screenshot link, the docs site links, and the About website to the live URL so nothing 404s.
Codecov rejected tokenless uploads ('Token required - not valid tokenless
upload'), so the badge stayed unknown. Use the repo upload token via the
CODECOV_TOKEN secret.
Single canonical source for every number quoted in the README, resume, and future /benchmark dashboard -- the +22.5pp parent-document-expansion lift (McNemar p<0.0001) is the real headline, not the raw graph-arm accuracy.
…dary
- kgqa/providers.py: multi-provider LLM chain (Groq/Gemini, falling back to
local Ollama) so a single free-tier provider deprecation doesn't take the
hosted agent down.
- kgqa/service.py: answer(question, graph_id) -> {answer, reasoning_path,
sources}, the one entry point a web backend needs. Shares encoder/reranker/
db connections across calls and degrades to raw chunks (rather than
crashing) when no graph is configured.
- graphrag/: thin public re-export of kgqa.service, the stable import
surface for the backend.
- retrieval/graph.py: extracted gather_studies()/format_studies() so the
service and GraphRetriever share one graph-expansion implementation
instead of two copies.
- backend/main.py: /health, /query (calls graphrag.answer), and /ingest (only resolves preloaded dataset ids -- arbitrary document upload is out of scope for v1, per the execution plan's own scope warning; returns 501). - render.yaml: Render Blueprint (free tier, health check wired). - .github/workflows/keep-warm.yml: pings /health every 10 min during daytime hours only, so Render's free-tier cold start (~30-50s) doesn't hit a cold demo. - Wired backend/ into CI (lint, pytest, coverage) and pyproject.toml.
- Chat UI (components/ChatPanel.tsx) with example questions, source links, and per-message error handling (cold-start timeout vs unreachable backend get distinct messages). - Reasoning-path graph (components/ReasoningGraph.tsx, react-flow) rendering the seed-chunk -> parent-paper -> concept-neighbour traversal the backend returns, with a legend distinguishing each edge type. - Typed API client (lib/api.ts, lib/types.ts) matching the backend's QueryRequest/QueryResponse contract, with a 60s timeout sized to Render's free-tier cold start. - Removed the create-next-app scaffold's unused prefers-color-scheme: dark block -- this app has no dark-mode support anywhere, and the leftover block silently made text near-invisible on the app's white backgrounds. .gitignore: exclude .claude/ (local dev tooling state, not project source).
- /benchmark page (frontend/app/benchmark/page.tsx): static dashboard reading results/summary.json directly at build time (lib/results.ts) -- per-arm accuracy/F1/latency table, McNemar significance table, and the honest-summary writeup. No duplicated numbers to drift from RESULTS.md. - Nav link between the chat and benchmark pages. - tests/test_results_regression.py + a dedicated "Eval regression gate" CI step: fails the build if results/summary.json regresses (graph arm accuracy/F1 floors, graph beats the reranked baseline, parent-expansion McNemar significance still holds). Doesn't re-run the LLM benchmark in CI -- that needs a GPU + live ArangoDB, which CI doesn't have -- it guards the checked-in artifact everything else points to instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.