Skip to content

Add Gemma 4 31B ArmBench full-run results #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Hrant-Khachatrian wants to merge 1 commit into from
Closed

Add Gemma 4 31B ArmBench full-run results #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions results/armbench/gemma-4-31b-it-openrouter-2026-08-27/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# ArmBench-LLM full-run submission: Gemma 4 31B IT

This is a standalone result submission for a complete ArmBench-LLM run. It is intentionally separate from proposed changes to the ArmBench task definition and the LiteLLM/OpenRouter runner.

## Submitted result

| Field | Value |
|---|---|
| Model | `google/gemma-4-31b-it` |
| OpenRouter model | `openrouter/google/gemma-4-31b-it` |
| Provider | DeepInfra, pinned; fallbacks disabled |
| Reasoning | Disabled |
| Temperature | 0 |
| Seed | 0 |
| ArmBench examples | 2,928 / 2,928 with nonblank outputs |
| Overall ArmBench score | **0.6141561129879723** (display: **0.6142**) |
| Runtime | 5,662.62 seconds |
| Recorded cost | approximately $0.3869 |

The cost is marked approximate because the auxiliary request ledger used request-derived keys; identical requests can overwrite earlier ledger entries. This does not affect the saved 2,928 sample outputs or benchmark score.

## Category scores

| Category | Score |
|---|---:|
| NER | 0.7168503431 |
| POS | 0.8000000000 |
| Reading | 0.5842618319 |
| Classification | 0.6428571429 |
| MCQA | 0.8733333333 |
| Generation | 0.1086606355 |
| Translation | 0.2838959010 |
| Exams | 0.5000000000 |
| Text processing | 0.8369071472 |
| MMLU-Pro | 0.7947947948 |

The overall score is the unweighted mean of the ten ArmBench category scores.

## Reproducibility

- Metric-AI LightEval commit: `2e22157f66c25a476fd2e5ebf6f75d84dd6efd5e`
- ArmBench dataset revision: `c1a7672da0b615de6d8437c74e4aa4c2f1a681f3`
- Configuration: `armbench_gemma4_31b_nothink.yaml`
- Dependency snapshot: `armbench_requirements_lock.txt`
- The exact uncommitted safety/reproducibility changes used by the run are included in `runner.patch.gz`.
- Standard LightEval aggregate JSON and all 24 per-task Parquet detail files are included under `artifacts/`.

No API key or other credential is included.
22 changes: 22 additions & 0 deletions results/armbench/gemma-4-31b-it-openrouter-2026-08-27/armbench_gemma4_31b_nothink.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
model_parameters:
model_name: "openrouter/google/gemma-4-31b-it"
provider: "openrouter"
max_model_length: 262144
concurrent_requests: 4
api_max_retry: 3
api_retry_sleep: 2.0
api_retry_multiplier: 2.0
timeout: 180
caching: false
reasoning_enabled: false
max_cost_usd: 2.0
extra_body:
reasoning:
enabled: false
provider:
order:
- "DeepInfra"
allow_fallbacks: false
generation_parameters:
temperature: 0
seed: 0
6 changes: 6 additions & 0 deletions results/armbench/gemma-4-31b-it-openrouter-2026-08-27/armbench_requirements_lock.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Compatibility pins verified in WSL Ubuntu-22.04 on 2026-08-27.
# Install the ArmBench fork editable with its litellm extra first, then apply
# these exact overrides until a complete lock is generated for the safe runner.
litellm==1.66.0
pydantic==2.13.4
xxhash==3.5.0
Loading