Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasoning metrics.
-
Updated
Aug 29, 2026 - Python
Production-grade LLM Evaluation & Benchmarking Framework - GPT-4, Claude, Gemini, Mistral. Accuracy, latency, cost, hallucination, reasoning metrics.
Code for NAACL paper When Quantization Affects Confidence of Large Language Models?
Enterprise-grade LLM evaluation framework | Multi-model benchmarking, honest dashboards, system profiling | Academic metrics: MMLU, TruthfulQA, HellaSwag | Zero fake data | PyPI: llm-benchmark-toolkit | Blog: https://dev.to/nahuelgiudizi/building-an-honest-llm-evaluation-framework-from-fake-metrics-to-real-benchmarks-2b90
LLM hallucination evaluation pipeline: two Claude models scored on TruthfulQA (817 questions, 38 categories) via AWS Bedrock and DeepEval LLM-as-judge, with per-category failure analysis and model comparison
SCM is a lightweight external middleware layer that reduces LLM hallucinations by enforcing confidence and citation checks before delivering answers. Tested on TruthfulQA and HaluEval — accuracy improved from 75% to 90%. Open source
A hallucination detection pipeline for Large Language Models (LLMs).
A deterministic honesty layer for LLM agents + a reproducible TruthfulQA eval that measures calibration honestly — reporting the wins and the costs.
LOGIC-HALT: multi-signal LLM hallucination detection (DeBERTa NLI, self-consistency, entropy, NCD) tuned with Optuna — evaluated on TruthfulQA
Evaluation of Llama-3.1-8B Base vs Instruct on TruthfulQA using few-shot prompting and automatic judge models
The definitive zero-dependency AI/ML evaluation, hallucination detection, prompt injection guardrail, and reliability library in pure Python. 2000x faster than LLM-as-a-judge at $0 cost.
Does instruction tuning make language models more sycophantic? A paired causal study across Qwen, Llama, and Gemma on TruthfulQA, showing the effect is family-dependent in both magnitude and direction. 7,200 evaluations, 12 ATE estimates with paired t-tests and bootstrap CIs.
Base vs instruct calibration on TruthfulQA reverses with prompt and scoring
Evaluating how semantically equivalent prompt variations affect LLM predictions and uncertainty estimates.
Official code for "From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems" (IWSDS 2026)
TruthfulQA MC1 evaluation harness for small LLMs: conditional log-likelihood scoring, 10-bin ECE, Brier score, prompt-flip heatmaps, and post-hoc temperature scaling.
Prototype that asks several LLMs to review an answer and flags likely hallucinations, with a balanced TruthfulQA evaluation.
Transformer transfer learning for sentiment analysis and commonsense reasoning: RoBERTa, PIQA, TruthfulQA, WinoGrande, and semantic similarity. PyTorch/Hugging Face coursework, ECE NTUA (2023).
Adaptive, probe-controlled activation steering that cuts LLM hallucination rate by ~10pp (64.6%→58.5%) on TruthfulQA — steers only when a real-time risk probe flags danger, unlike fixed-strength ITI/CAA/TSV baselines reproduced here for comparison.
Your LLM judge says a model is wrong. Can it show the false sentence? Evidence-backed grading, checked against human labels.
Does inter-agent LLM disagreement signal hallucination? An 817-question TruthfulQA study with judge-based evaluation and a fair single-model baseline.
To associate your repository with the truthfulqa topic, visit your repo's landing page and select "manage topics."