fix(eval): add online logs and include batch evaluation errors in get

[Hweinstock]

Superseded by #2588, authored by @jariy17, with the same changes. Review history is retained here.

What is this PR doing?

Adds agentcore eval online-eval logs so customers can stream or search evaluation logs using an online evaluation configuration ID. It follows the existing log command conventions: tail by default, search with --since / --until, filter with --level / --query, limit search results with --limit, and emit JSON Lines with --json.

Also expands agentcore eval batch-evaluation get to return evaluation errors alongside scores, including failures that have no evaluator ID. Batch results and errors are retrieved through get; this PR does not add a separate batch-evaluation logs command.

# Follow an online evaluation's logs
agentcore eval online-eval logs --id <online-evaluation-id>

# Search recent online evaluation logs as JSON Lines
agentcore eval online-eval logs --id <online-evaluation-id> --since 1h --limit 20 --json

# Read a batch job's results, including evaluation errors
agentcore eval batch-evaluation get --id <batch-evaluation-id> --json

Why is this PR doing this?

Online evaluations run continuously. Customers need to inspect their output from the CLI without looking up the CloudWatch log group themselves, whether results go to a dedicated group or the configured source group.

Batch evaluations already expose results through get, but the parser dropped failures without an evaluator name and omitted the standard error.message and error.type fields. A failed or partially successful job could therefore hide the information needed to diagnose it. Returning scores and errors together keeps the output tied to the requested job's exact log stream; searching a shared output group could mix records from different jobs.

How is this PR doing this?

• Adds an online-specific logs handler under online-eval/logs and reuses the existing shared log handler and CloudWatch search/tail implementation.
• Resolves the online evaluation's explicit output group, default dedicated group, or single source group for SOURCE_LOG_GROUP. Returns actionable errors for unsupported output configurations and ambiguous source groups. The resolver stays specific to online evaluations.
• Extends the batch result parser to read error.message and error.type, retaining legacy gen_ai.evaluation.error support and scope identifiers. Unnamed errors must be evaluation-result records, so unrelated application errors are excluded.
• Preserves batch retrieval from the job's exact output stream, --disable-cw-results, and nonfatal warnings when CloudWatch results cannot be read.
• Adds recorded AWS golden tests for online text/JSON search output and batch success, unnamed errors, missing log events, mixed results with custom output, and metadata-only output. Keeps focused unit coverage for streaming, required IDs, source resolution, and parsing.

Testing

Verified the built Node CLI against live AWS in us-east-1 on October 13, 2026:

• Online logs: searched a retained real online evaluation result with a fixed time window in text and JSON formats. Both returned the same record with the expected Builtin.Correctness score. Recorded these calls for golden tests that replay offline.
• Batch get: ran fresh jobs using complete arithmetic-agent traces and deliberately incomplete traces. The evaluation service produced the error records. Compared each returned score, error type/message, evaluator ID, and session ID with CloudWatch events from that job's exact stream.

Batch case	Job status	Scores	Errors	CloudWatch comparison
Complete runtime traces	COMPLETED	2	0	Matched
No spans available for the sessions	FAILED	0	2	Matched
Spans missing corresponding log events	FAILED	0	2	Matched
Complete and incomplete traces, custom output group	COMPLETED_WITH_ERRORS	1	1	Matched

Rechecked all four batch cases after the final test changes. Confirmed metadata-only output for each case and failures for a missing --id, a nonexistent job, and the removed batch logs command. The first mixed-job attempt hit log ingestion delay; rerunning after ingestion produced the mixed outcome above.

All local checks passed: bun test (4,041 passed, 0 failed), bun run typecheck, bun run lint:check, bun run format:check, bun run secrets:check, and bun run build. New recordings are sanitized and replay offline. The live checks exercise the changed AWS interaction; no deployment E2E suite was run locally.

Captured output

These screenshots render captured CLI stdout. jq selects the fields shown; job and span identifiers are redacted.

Successful evaluation

Evaluation errors

Mixed results in a custom output group

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

The new eval online-eval logs and eval batch-evaluation logs subcommands are a clean reuse of createLogsHandler and mirror the existing harness-logs pattern. The log-source resolution in EvalClient.evaluationLogSource correctly layers (1) explicit CloudWatch output log group, (2) rejection of non-CloudWatch output configs, (3) the service default log group, and (4) the SOURCE_LOG_GROUP fall-through — including the one-hop indirection through onlineEvaluationConfigSource for batch evaluations spawned by an AB test. Error types (InputValidationError vs. MalformedServiceResponseError) are used appropriately.

Nothing I'd block the merge on. A couple of optional follow-ups the author may want to consider:

• src/core/eval.tsx — evaluationLogSource has several branches (default/explicit/source-group, with and without the AB-test indirection) but no unit tests. The manual verification in the PR description covers the paths well; a small table-driven test around this method would make future refactors safer. Not a blocker — the file has no existing unit test to extend.
• src/handlers/eval/logs/index.tsx — typing resolveLogSource as CoreEvalClient["resourceBatchEvaluationLogSource"] works only because both resolver signatures are identical today. A named shared type (e.g., (id: string, options: CoreOptions) => Promise<LogSource>) would read more clearly, but it's just style.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.52%. Comparing base (e5a71d6) to head (7cc950b).
⚠️ Report is 7 commits behind head on refactor.

Additional details and impacted files

@@            Coverage Diff            @@
##           refactor    #2582   +/-   ##
=========================================
  Coverage     97.52%   97.52%           
=========================================
  Files           634      635    +1     
  Lines         47260    47328   +68     
=========================================
+ Hits          46090    46158   +68     
  Misses         1170     1170           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Do we need batch-evaluation logs? batch-evaluation get already returns the log results?
• get reads the job's own stream (logGroupName + logStreamName) and parses each result (src/core/eval.tsx:518-560).
However, it doesn't return errors from the logGroup like this one:
message: {"name":"gen_ai.evaluation.result","attributes":{"error.type":"LogEventMissingException","error.message":"Session span data is incomplete. Span with ID: <span-id> and name: invoke_agent Strands Agents is missing a corresponding log event. Please retry the request with the complete span data"}}
We should expand this command to return errors like this.

[jariy17]

This will show every job in that log group. You need to add a filter in the cloudwatch query or filter by logStream because each job get their own logstream.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)