[Hrant-Khachatrian]

Result

Adds a completed full ArmBench-LLM evaluation for google/gemma-4-31b-it:

• Overall score: 0.6141561129879723 (leaderboard display: 0.6142)
• Completeness: 2,928 / 2,928 examples with nonblank outputs across all 24 tasks
• Reasoning: disabled
• Provider: OpenRouter → DeepInfra, pinned with fallbacks disabled
• Generation: temperature 0, seed 0
• Runtime: 5,662.62 seconds
• Recorded cost: approximately $0.3869 (see the manifest note about duplicate request-ledger keys)

Reproducibility

• LightEval commit: 2e22157f66c25a476fd2e5ebf6f75d84dd6efd5e
• ArmBench dataset revision: c1a7672da0b615de6d8437c74e4aa4c2f1a681f3
• The tracked result directory contains the aggregate LightEval JSON, run manifest, exact configuration, dependency pins, and compressed runner patch.
• The attached ZIP contains all 24 per-task Parquet detail files and the same submission metadata.
• Attachment SHA-256: 2BA74F9064D87F612450632C0EE76EAE54694887965BBE293DD438EEEA76773D

This PR contains result artifacts only; benchmark-definition and endpoint-runner improvements will be proposed separately.

Full detail artifacts:
gemma-4-31b-it-openrouter-2026-08-27.zip

[bugdaryan]

Hi @Hrant-Khachatrian

Thanks, we appreciate the contribution.

New results are added to the benchmark here https://huggingface.co/spaces/Metric-AI/ArmBench-LLM

Closing this PR as we don't store results in this repo.