AI Evals Framework — Python LLM and RAG Evaluation

Evaluate language-model answers against versioned datasets with a Python CLI and JSON reports. Use it to compare prompt changes, check RAG grounding, and inspect answer accuracy, similarity, and pattern-based PII or toxicity signals.

Try an offline evaluation

Python 3.10 or newer is required. This example uses the lexical accuracy metric; it does not need a model-provider key or cloud infrastructure.

git clone https://github.com/AcadifySolution/ai-evals-framework.git
cd ai-evals-framework
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
ai-evals --dataset examples/dataset.json --metrics accuracy --output evaluation_report.json

Open evaluation_report.json for per-sample results and aggregate scores. The included dataset is a demonstration, not a representative model benchmark.

Choose the metric for your question

Metric name	What it checks	Interpretation
accuracy	Lexical overlap against expected answers	Wording changes can affect scores even when meaning is similar
hallucination	Grounding against supplied context, with optional LLM judging	The local fallback uses overlap heuristics, not factual verification
semantic_similarity	Answer similarity, with optional provider embeddings	The offline fallback is lexical and can miss paraphrases
toxicity	Configured toxic-language and PII patterns	Pattern matches are screening signals, not comprehensive detection

Select several names with --metrics accuracy,hallucination. Provider-backed metrics require configuration and can transmit evaluation content to a provider. Review configuration and security guidance before using sensitive datasets. Scores do not establish regulatory compliance or prove that an answer is safe or true.

Explore the implementation

• Example dataset: start with the expected sample format.
• Dataset loaders: JSON, JSONL, and CSV ingestion.
• Evaluation engine: concurrent execution and ordered results.
• CLI: metric selection, concurrency, and JSON output.
• Metrics: inspect scoring behavior before setting thresholds.
• Tests: regression examples and validation boundaries.
• Infrastructure examples: Terraform scaffolding, requiring a separate deployment review; not a prerequisite for local evaluation.

Develop and contribute

ruff check .
black --check .
pytest -q

Use a small reproducible dataset when reporting a scoring issue. Include the metric, expected behavior, configuration, and package version; remove secrets and personal data. See CONTRIBUTING.md.

For retrieval implementation, explore Enterprise RAG Boilerplate. For human review records, explore ASR Feedback. If this tool is useful, star it to save it, fork it for experiments, or contribute a reproducible issue or pull request.

License

MIT.