Build a question-answering assistant over your own documents — one that answers with citations and says "I don't know" when your documents don't cover the question.
New to this? GETTING-STARTED.md builds the whole thing from an empty directory, explaining chunking, embeddings, retrieval and grounding as it goes.
$ customllm ingest ./data
Embedding with ollama:all-minilm ...
Indexed 6 chunks from 2 document(s)
$ customllm ask "how many days of annual leave do employees get"
Q: how many days of annual leave do employees get
A: Employees are granted 27 days of paid annual leave [1]. Unused days may be
carried over up to a maximum of five days [1].
Sources: handbook.md:19
$ customllm ask "what is the capital of France"
A: I don't have anything in the indexed documents that answers that.
That last answer is the important one. A system that will not admit ignorance is worse than no system at all.
People asking for "a custom LLM trained on my data" usually want one of three quite different things. Choosing wrong wastes a lot of time, so here they are honestly:
| What it does | What it costs | When it is right | |
|---|---|---|---|
| 1. RAG (this repo's default) | Finds relevant passages at question time and puts them in the prompt | Minutes. CPU is fine | You want factual answers about your documents, with citations |
| 2. Custom model definition | Bakes a system prompt and parameters into a named model | Seconds | You want to change tone, format or persona |
| 3. Fine-tuning (LoRA) | Actually updates model weights | Hours, and a GPU | You want to change how the model behaves in a way prompting cannot |
If your goal is "answer questions about my documents", you want option 1. Not because fine-tuning is hard, but because it is the wrong tool: fine-tuning teaches a model a style, not a fact table. Facts change; retraining every time a document changes is absurd when you could simply re-index in seconds.
flowchart LR
DATA[("Your data")]
DATA --> P1["<b>1 · RAG</b><br/>index it, retrieve at<br/>question time"]
DATA --> P2["<b>2 · Custom model</b><br/>ollama create<br/>with a system prompt"]
DATA --> P3["<b>3 · Fine-tune</b><br/>JSONL → LoRA"]
P1 --> R1["cites its sources<br/>refuses when unsure<br/>update = re-index<br/><br/><b>minutes, CPU</b>"]
P2 --> R2["shapes tone and format<br/><b>invents facts</b><br/>cannot cite<br/><br/><b>seconds</b>"]
P3 --> R3["changes deep behaviour<br/>still cannot cite<br/>retrain to update<br/><br/><b>hours, GPU</b>"]
R1 --> USE1["Use for<br/><b>facts about your documents</b>"]
R2 --> USE2["Use for<br/><b>persona and format</b>"]
R3 --> USE3["Use for<br/><b>behaviour prompting cannot reach</b>"]
classDef store fill:#0d3b66,stroke:#0d3b66,color:#fff
classDef good fill:#1b5e20,stroke:#1b5e20,color:#fff
classDef mid fill:#7c4a03,stroke:#7c4a03,color:#fff
class DATA store
class R1,USE1 good
class R2,R3 mid
This repository does all three, and is honest about what each delivers.
During development, create-model built a real Ollama model whose system prompt described this
coffee company's knowledge base. Asked a question that is nowhere in that knowledge base:
$ ollama run nimbus-bot "What is the boiling point of water?"
The information you're asking about can be found in document [2], which states:
"Water has a standard atmospheric pressure boiling point at exactly 100 degrees
Celsius (°C) or 212 degrees Fahrenheit (°F)." Therefore, the answer is that water
boils at 100°C or 212°F. [2]
There is no document [2] containing that. The model invented the quote and the citation. The
system prompt was stored correctly — ollama show confirms it — but instructions are not
guarantees.
The same question through the RAG path:
$ customllm ask "what is the boiling point of water"
A: I don't have anything in the indexed documents that answers that.
That is the entire argument. Retrieval can refuse, because it knows what it actually found.
Prerequisites: Python 3.11+. Optionally Ollama for real embeddings and generation — the tool runs without it.
git clone https://github.com/kuldeepcodes/custom-llm-python.git
cd custom-llm-python
python -m venv .venv
source .venv/bin/activate # Windows: .\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
python -m pytest # 103 tests
customllm ingest ./data
customllm ask "how many days of annual leave do employees get"For real semantic search and generation:
ollama pull all-minilm # ~45 MB embedding model
ollama pull phi3 # ~2.2 GB chat modelWithout Ollama it still works. ingest falls back to a deterministic hashing embedder and
ask returns the best matching passage verbatim instead of a generated answer. Retrieval quality
drops, but the pipeline is real and every test passes — which is also how CI runs it, with no
model download.
| Command | What it does |
|---|---|
customllm ingest <path> |
Chunk, embed and index a folder or file |
customllm ask "<question>" |
Retrieve, then answer with citations |
customllm search "<question>" |
Show what retrieval found, and why, without generating |
customllm info |
Describe the current index |
customllm create-model <name> |
Build a real custom Ollama model (path 2) |
customllm export |
Write JSONL training data for LoRA (path 3) |
Useful flags: --embedder {auto,ollama,hashing}, --top-k N, --show-context, --no-llm.
search is the one to reach for when an answer looks wrong. It shows the blended score plus its
vector and keyword components, so you can see whether retrieval or generation is at fault:
$ customllm search "how much holiday allowance do staff get"
[1] handbook.md:19 score=0.351 (vector=0.468 keyword=0.000)
## Time off Everyone receives 27 days of paid annual leave...
Note keyword=0.000. The question and the passage share no words at all — "holiday
allowance" never appears in the handbook, which says "annual leave". The embedding found it
anyway. That is what embeddings buy you, and running the same query with --embedder hashing
drops that passage to third place.
flowchart TD
subgraph INGEST["INGEST — once per corpus change"]
direction LR
SRC["Your files<br/><code>.md .txt .json .jsonl</code>"]
LOAD["<b>load</b><br/>walk, name<br/>flatten JSON to<br/>one doc per record"]
CHUNK["<b>chunk</b><br/>whole sentences + overlap<br/>keeps source:line"]
EMB1["<b>embed</b><br/>one batched call"]
SRC --> LOAD --> CHUNK --> EMB1
end
IDX[("<b>index.json</b> — vectors + provenance + embedder name")]
EMB1 --> IDX
subgraph QUERY["QUERY — per question"]
direction LR
Q["Question"]
EMB2["<b>embed</b> question<br/>same model as index"]
SEARCH["<b>search</b><br/>0.75 cosine + 0.25 keyword"]
Q --> EMB2 --> SEARCH
end
IDX --> QUERY
SEARCH --> FLOOR{"top score at least 0.25?"}
FLOOR -->|no| REFUSE["<b>refuse</b><br/>nothing in the indexed<br/>documents answers that"]
FLOOR -->|yes| PROMPT["<b>prompt</b> — numbered passages,<br/>cite every claim, NOT_IN_CONTEXT escape"]
PROMPT --> LLM["LLM"] --> AUDIT["<b>audit citations</b>"]
AUDIT --> ANSWER["<b>Answer</b> + handbook.md:19"]
AUDIT -.->|number never supplied| W1["warn: fabricated"]
AUDIT -.->|wording mismatch| W2["warn: mis-numbered"]
classDef store fill:#0d3b66,stroke:#0d3b66,color:#fff
classDef good fill:#1b5e20,stroke:#1b5e20,color:#fff
classDef stop fill:#7f1d1d,stroke:#7f1d1d,color:#fff
classDef warn fill:#7c4a03,stroke:#7c4a03,color:#fff
class IDX store
class ANSWER good
class REFUSE stop
class W1,W2 warn
The two phases are deliberately separate. Ingest is the slow part and runs only when your documents change. Query is fast, and touches nothing but the index — which is why updating what the system knows is a re-index measured in seconds, not a retrain measured in hours.
Every component, in the order it is actually called:
sequenceDiagram
autonumber
actor U as You
participant CLI as CLI<br/>(customllm ask)
participant EMB as Embedder<br/>(all-minilm)
participant IDX as VectorIndex<br/>(index.json)
participant GEN as Grounding
participant LLM as LLM<br/>(phi3)
U->>CLI: customllm ask "how much annual leave?"
CLI->>IDX: load(index.json)
IDX-->>CLI: chunks + vectors + embedder name
CLI->>IDX: ensure_compatible(embedder)
Note over CLI,IDX: Refuses if the index was built with a<br/>different model. Cross-model cosine is<br/>arithmetic without meaning.
IDX-->>CLI: ok
CLI->>EMB: embed_one(question)
EMB-->>CLI: 384-dim unit vector
CLI->>IDX: search(question, vector, top_k=4)
Note over IDX: 0.75 x cosine + 0.25 x keyword overlap
IDX-->>CLI: ranked passages with source:line
CLI->>GEN: answer_question(question, passages, chat)
alt top score below 0.25
GEN-->>CLI: refusal, grounded = false
Note over GEN: The corpus has no answer.<br/>Never ask the model to improvise.
else passages look relevant
GEN->>GEN: build_prompt(question, passages)
Note over GEN: [1] (handbook.md:19) Everyone receives 27 days...<br/>[2] (products.md:1) Meridian is 60 percent...
GEN->>LLM: system rules + numbered passages
LLM-->>GEN: "Employees get 27 days [1]."
GEN->>GEN: strip_template_artifacts
GEN->>GEN: check cited numbers exist
GEN->>GEN: check wording matches cited passage
GEN-->>CLI: answer + citations + any warnings
end
CLI-->>U: A: Employees get 27 days [1].<br/>Sources: handbook.md:19
Three things in that sequence are easy to miss and matter a lot:
The compatibility check happens before any work (step 4). Querying an index with a different embedder than built it produces no error — just silently meaningless scores. Failing loudly here saves an afternoon of misdiagnosis.
The model is never asked to improvise (step 11). If the best passage is too weak, the LLM is not called at all. You cannot hallucinate from a prompt you never sent.
The model's output is checked, not trusted (steps 15–17). The prompt asks for honest citations; the audit verifies them. Instructions are not guarantees.
Six decisions worth knowing about:
One document per JSON record. A top-level array becomes file.json#0, #1, … rather than one
blob, so a citation points at a record you can actually open and check.
Sentence-aware chunking with overlap. Chunks are packed with whole sentences up to a size budget, and each chunk repeats a little of the previous one. Overlap matters because a fact sitting on a boundary would otherwise be split across two chunks and retrievable in neither.
Hybrid retrieval. Score is 75% cosine similarity, 25% keyword overlap. Pure vector search is weak on rare literal tokens — part numbers, error codes, surnames — because embeddings smooth them away. Pure keyword search misses paraphrase. Together they cover each other.
A relevance floor. If the best passage scores below 0.25, the system refuses rather than letting the model improvise from thin context.
Citation auditing. Instructions are not guarantees, so citations are checked afterwards:
- invalid — the answer cited a passage number that was never supplied
- weak — the cited passage shares little wording with the claim, so the number is probably wrong even where the content is right
That second check exists because phi3 answered a question correctly from passage [1] and then cited [2]. A citation nobody verifies is decoration.
Embedder identity is recorded in the index. Querying an index built by a different model is refused outright. Cosine similarity between vectors from two different models is arithmetic without meaning, and the symptom is not an error — it is quietly terrible retrieval.
customllm ingest ~/my-notes
customllm ask "what did we decide about the pricing model"| Extension | How it is read |
|---|---|
.md, .markdown, .txt |
One document per file |
.json |
A top-level array becomes one document per element, named file.json#0, file.json#1, ... A single wrapped array ({"items": [...]}) is unwrapped. Anything else is one document |
.jsonl, .ndjson |
One document per line, named file.jsonl#1, #2, ... numbered as your editor shows them |
JSON records are flattened into readable key: value lines rather than fed in raw, because
embedding models were trained on prose and braces carry no meaning:
{"id": "TKT-1041", "customer": {"name": "Priya"}, "tags": ["hardware", "grinder"]}becomes
id: TKT-1041
customer.name: Priya
tags: hardware, grinder
Field names are kept because they are genuine context — subject: Grinder jams embeds better than
the bare value. Nulls are dropped, since a line reading resolution: None is noise that embeds.
Splitting an array into one document per element is the point of JSON support: a citation reading
support-tickets.json#3:1 sends you to a specific record you can open and check, whereas "somewhere
in tickets.json" tells you nothing.
For PDFs or Word files, convert them first (pandoc, pdftotext) — deliberately not built in, so
the dependency list stays honest. GETTING-STARTED.md
shows how to add a format in about five lines.
Tuning tips:
- Long reference documents: raise
--chunk-sizeto 1200 - Dense factual material such as FAQs: lower it to 400
- Answers missing obvious content: raise
--top-k, and usesearchto see what was retrieved
python -m pytest # 103 tests
python -m ruff check .Tests use the hashing embedder throughout, so they are deterministic and need no model, no network and no GPU. Generation is exercised through a stub chat client that returns scripted replies — including deliberately bad ones, so the citation auditing is proven to fire. Loading is covered separately: JSON array splitting, JSONL line numbering, nested flattening, malformed input, and mixed corpora where Markdown and JSON sit in the same folder.
- custom-llm-python — Python 3.11+ (you are here)
- custom-llm-dotnet — C# / .NET 10
- custom-llm-java — Java 17