Skip to content

StirrupAgentRunner's default model id never matches a price-table key, so est_cost silently returns None #522

Description

@AmirF194

Background

_estimate_cost (src/evaluation/metrics.py) looks up a model's per-token
rate in _PRICE_PER_1M after normalizing the model id through
_normalize_model. For StirrupAgentRunner's own default model,
watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8
(src/agent/stirrup_agent/runner.py), the normalized id never matches the
existing llama-4-maverick table entry, so _estimate_cost returns None
and any report built on it (including aggregate_ops's
est_input_cost_usd_total/est_output_cost_usd_total, added in #518) is
silently missing a dollar figure for runs using the default model.

src/evaluation/tests/test_metrics.py only exercises single-segment ids
(granite-13b, one date-suffix case), so this gap isn't caught by the
existing suite.

Steps to Reproduce

import sys
sys.path.insert(0, "src")
from evaluation import metrics

default_model = "watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
print("normalized:", metrics._normalize_model(default_model))
print("cost:", metrics._estimate_cost(default_model, 1000, 1000))
print("control (bare table key):", metrics._estimate_cost("llama-4-maverick", 1000, 1000))

Observed

normalized: llama-4-maverick-17b-128e-instruct-fp8
cost: None
control (bare table key): 0.00112

Expected

llama-4-maverick-17b-128e-instruct-fp8 resolves to the existing
llama-4-maverick entry, same as the control case.

Environment

  • Repo HEAD: 3a7bd23c7421f6ec3c0f215f018d7a8c2bcd8e37
  • Python 3.12.14, python:3.12-slim Docker image, pydantic from PyPI

Reproduced end to end in a clean container against current main, running
the real module (not a mock). Happy to send a PR if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions