Skip to content

Extend evidence-strength notes to the cost objective - #354

Merged
dipeshbabu merged 2 commits into
mainfrom
feat/cost-noise-guidance
Nov 19, 2026
Merged

dipeshbabu merged 2 commits into
mainfrom
feat/cost-noise-guidance

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

The noisy-baseline planning note only measured latency, so cost comparisons got no warning about run-to-run variation.

  • For `--objective cost`, `improve start`/`compare` compare estimated cost between repetitions of the same task (the same `trace.report()` measurement the cost gates use) and flag a noisy baseline at a median spread of at least 10% that exceeds `--minimum-improvement`. The note now records its `objective`.
  • New `cost_unmeasurable` note: completed baseline attempts with unknown model pricing or estimated token counts, whose cost gates will be indeterminate.
  • Latency behaviour is unchanged; notes still never change decisions.

Tests: noisy cost, quiet cost cases, unmeasurable pricing / estimated tokens (and that the latency objective ignores them); guided suites pass locally (72 passed). Checked against real captured traces. Docs and changelog updated.

🤖 Generated with Claude Code

dipeshbabu and others added 2 commits November 19, 2026 02:00
The noisy-baseline note only measured latency. For the cost objective it
now compares estimated cost between repetitions of the same task, using
the same report the cost gates use, and separately reports completed
baseline attempts whose cost cannot be measured because of unknown
pricing or estimated token counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_015c2d14895e4bce runs/research_agent_baseline.json yes agentloop_demo
candidate run_53caaee7c02c4138 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_a6f11737f4ab: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu merged commit 63c6d6b into main Nov 19, 2026
48 checks passed
@dipeshbabu
dipeshbabu deleted the feat/cost-noise-guidance branch November 19, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant