This project has been created as part of the 42 curriculum by lyeh.
This project is a Retrieval-Augmented Generation (RAG) system built over the
vLLM codebase. It indexes the corpus
(Python source and Markdown docs), retrieves the most relevant snippets for a
question with a classic lexical method, and generates a grounded answer with
a small local model (Qwen/Qwen3-0.6B).
Beyond the 42 subject's mandatory deliverable, the actual goal of this
project is to serve as a continuous benchmarking harness for the RAG
service: build the index once, then repeatedly run search_dataset →
evaluate (or the official moulinette) as the retrieval and generation
logic evolves, tracking recall@k over time rather than measuring it once for
a single submission.
Status: this repository currently holds the project scaffold — directory layout, dependency/tooling setup, the mandatory pydantic data models, and a fully wired Fire CLI. The indexing, retrieval, generation and evaluation algorithms themselves are stubs (
NotImplementedError) pending the next implementation pass. Sections below marked_TODO_will be filled in as that work lands.
- Python 3.14 (pinned via
.python-version; the subject only requires ≥3.10) uvas the project/package manager
make install # uv sync — installs all dependencies into .venv/Optional, but recommended for the continuous-benchmarking workflow: set these
environment variables (e.g. in a local .env, untracked) to auto-trace every
LangGraph run against LangSmith's free cloud tier — no self-hosted service
needed on a small VM:
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-api-key>
export LANGSMITH_PROJECT=naive-ragThe vLLM corpus, the question datasets, and the moulinette evaluator are 42
intranet attachments and are not committed to this repository (see the
warning in .gitignore). Fetch them once per machine:
wget https://cdn.intra.42.fr/document/document/55207/vllm-0.10.1.zip
wget https://cdn.intra.42.fr/document/document/55205/datasets_public.zip
wget https://cdn.intra.42.fr/document/document/55208/moulinette.zip
unzip vllm-0.10.1.zip -d data/raw/
unzip datasets_public.zip -d data/datasets/
unzip moulinette.zip
mv moulinette-ubuntu moulinette # or moulinette-fedora, depending on your OS
chmod +x moulinetteEvery command is invoked as uv run python -m src <command> [options]:
# 1. Index the corpus once.
uv run python -m src index --max_chunk_size 2000
# 2. Search a single query.
uv run python -m src search "How to configure the OpenAI server?" -k 5
# 3. Search a whole dataset.
uv run python -m src search_dataset \
--dataset_path data/datasets/UnansweredQuestions/dataset_docs_public.json \
-k 10 \
--save_directory data/output/search_results/UnansweredQuestions
# 4. Score with the moulinette.
./moulinette evaluate_student_search_results \
data/output/search_results/UnansweredQuestions/dataset_docs_public.json \
data/datasets/AnsweredQuestions/dataset_docs_public.json \
--k 10 --max_context_length 2000
# 5. Generate answers from the search results.
uv run python -m src answer_dataset \
--student_search_results_path data/output/search_results/UnansweredQuestions/dataset_docs_public.json \
--save_directory data/output/search_results_and_answer/UnansweredQuestions
# For your own iteration (the moulinette remains the official score):
uv run python -m src evaluate \
--student_search_results_path data/output/search_results/UnansweredQuestions/dataset_docs_public.json \
--dataset_path data/datasets/AnsweredQuestions/dataset_docs_public.jsonmake run # show CLI help
make debug # run the CLI under pdb
make lint # flake8 + mypy (mandatory flags)
make lint-strict # flake8 + mypy --strict
make clean # remove caches
uv run pytest # run the (non-graded) test suite- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the original RAG paper.
- vLLM documentation — the indexed corpus.
- rank_bm25 / scikit-learn TF-IDF — candidate lexical retrieval implementations.
- Qwen3 model card — the mandatory generation model.
- Python Fire — CLI framework used here.
- Pydantic v2 documentation — data validation.
- LangGraph — used to orchestrate the index → retrieve → augment → generate stages as an explicit graph.
- LangSmith — used to trace and observe repeated benchmark runs over time; integrates natively with LangGraph runs.
- Ragas — used to compute RAG evaluation metrics (context precision/recall, faithfulness, answer relevancy) beyond recall@k.
AI usage: AI assistance (Claude Code) was used to scaffold the repository structure, generate the pydantic models from the subject's specification, and draft this README skeleton. All generated code has been reviewed and is understood by the author; the actual RAG algorithms (chunking, indexing, retrieval, generation, evaluation) are implemented and reasoned through directly by the author in the following pass, with AI used only to reduce repetitive boilerplate.
data/raw/ ──chunk──▶ data/processed/ (BM25/TF-IDF index)
│
query ─┼─▶ Retriever.search ─▶ top-k MinimalSource
│ │
└─▶ AnswerGenerator.generate ◀──┘ (Qwen/Qwen3-0.6B)
│
▼
StudentSearchResultsAndAnswer (JSON)
src/chunking/— two chunking strategies (Python code, Markdown/text).src/indexing/indexer.py— builds and persists the lexical index underdata/processed/.src/retrieval/retriever.py— loads the index and ranks chunks for a query.src/generation/generator.py— groundsQwen/Qwen3-0.6Bon retrieved sources.src/evaluation/evaluator.py— local recall@k for iteration (not the official score).src/cli.py/src/__main__.py— the Fire CLI wiring the above together.src/models.py— the pydantic models shared between every stage.
TODO: expand with the LangGraph graph topology once the orchestration layer is implemented.
TODO: document the Python (AST-aware) and Markdown (heading-aware) chunking strategies once implemented, including how --max_chunk_size (default 2000, capped by the moulinette's max_context_length) is enforced.
TODO: document which lexical method was implemented (BM25 or TF-IDF) and why, plus any hybrid/reranking added on top.
TODO: report indexing time, retrieval throughput for 200 questions, and recall@1/3/5/10 on the docs and code datasets once measured.
- uv + Python 3.14: the subject requires ≥3.10; 3.14 is pinned via
.python-versionsince PyTorch 2.13 (Jul 2026) added full support for it, giving the most headroom forlanggraph/langsmithand the ML stack (transformers/torch), whilepyproject.tomlkeepsrequires-python = ">=3.10"for grader flexibility. - LangSmith over Langfuse for tracing: LangSmith integrates natively with LangGraph (
LANGSMITH_TRACING=trueauto-traces every graph run, no manual instrumentation) and its free cloud tier avoids running extra services on a small VM. The tradeoff versus Langfuse is no self-hostable open-source fallback if the free tier ever stops being enough. - Service classes are plain Python, models are pydantic: per the subject, only the data exchanged between stages must be pydantic;
Indexer/Retriever/AnswerGeneratorare plain classes. - CLI validates, collaborators implement:
src/cli.pyhandles all defensive input validation (empty query,k<=0, missing/malformed files) and never lets an unhandled traceback escape; the actual algorithms live in single-responsibility collaborator classes. - Heavier ML dependencies deferred:
rank-bm25/scikit-learnandtransformers/torchare added in the implementation pass rather than the scaffold, to keepuv syncfast until they're needed.
TODO: add the chunking/retrieval-specific decisions once implemented.
TODO: document once the implementation pass is complete.
See Running the CLI above for the full end-to-end walkthrough (index → search_dataset → moulinette evaluation → answer_dataset).