To evaluate the performance of each model, I compared their generated summaries across four key metrics: Summary Quality, Accuracy, Conciseness, and Hallucinations. Each metric is scored out of 10, where a higher score indicates a stronger presence of that quality in the summary (e.g., a 9/10 in Accuracy means the summary closely and correctly reflects the original content). For Hallucinations, a score of 0 means no hallucinations were detected, while higher values indicate more fabricated details.
Based on these four metrics, each model was given an overall Overall Rating, representing its combined performance across all criteria. The model with the highest overall rating is considered the best-performing model for this summarization task.
| Model | Summary Quality | Accuracy | Conciseness | Hallucinations | Overall Rating |
|---|---|---|---|---|---|
| Gemini | 9 | 9 | 7 | 0.5(Minor Terminology Issue) | 9.5/10 |
| Claude | 8.5 | 9 | 8.5 | 0(None) | 9/10 |
| Groq – openaigpt-oss-120b | 8 | 9 | 9 | 0(None) | 8.5/10 |
| Groq – llama-3.3-70b | 7 | 9 | 9.5 | 0(None) | 8/10 |