Skip to content

Refactor metrics recording to invocations and add tool/workflow duration metrics - #615

Open
lmolkova wants to merge 21 commits into
lmolkova-infer-retrieval-attrsfrom
lmolkova-metrics-refactor
Open

Refactor metrics recording to invocations and add tool/workflow duration metrics#615
lmolkova wants to merge 21 commits into
lmolkova-infer-retrieval-attrsfrom
lmolkova-metrics-refactor

Conversation

@lmolkova

@lmolkova lmolkova commented Sep 4, 2026

Copy link
Copy Markdown
Member

What does this change do?

Refactors metric recording directly into invocation classes (removing InvocationMetricsRecorder, instruments.py, and metrics.py), and adds duration metrics for tool and workflow invocations.

Semantic convention changes

Added metrics:

  • gen_ai.execute_tool.duration (s): Measures tool execution duration.
    • Attributes: gen_ai.tool.name, gen_ai.tool.type, gen_ai.agent.name, error.type.
  • gen_ai.invoke_workflow.duration (s): Measures workflow execution duration.
    • Attributes: gen_ai.workflow.name, error.type.

Changed metric attributes & buckets:

  • gen_ai.invoke_workflow.duration: Updated advisory bucket boundaries to workflow-specific scale ([1, 5, ... 7200]s).
  • Removed gen_ai.operation.name from tool and workflow duration metrics in compliance with the semconv metrics spec.

Why?

Aligns tool and workflow metrics with GenAI semantic conventions (metrics spec) and removes an unnecessary indirection layer.

lmolkova added a commit that referenced this pull request Sep 4, 2026
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Sep 4, 2026

Copy link
Copy Markdown

Pull request dashboard status

Waiting on maintainers · refreshed 2026-09-04 23:17 UTC

Resolve merge conflicts, then merge when ready.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It removes public metric APIs without deprecation and repeats histogram creation in the streaming hot path.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Refactors GenAI metric recording into invocation classes and introduces semconv-aligned tool duration metrics.

Changes:

  • Replaces InvocationMetricsRecorder usage with direct Meter access.
  • Adds dedicated tool metrics and aligns workflow metric attributes/buckets.
  • Updates metric tests and changelog fragments.
File summaries
File Description
.changelog/615.added Documents metric changes.
.changelog/615.removed Documents removed helpers.
_agent_invocation.py Uses shared client metric method.
_embedding_invocation.py Uses shared client metric method.
_fetch_response_invocation.py Uses shared client metric method.
_inference_invocation.py Passes Meter directly.
_invocation.py Creates and records client metrics.
_retrieval_invocation.py Uses shared client metric method.
_tool_invocation.py Adds tool duration recording.
_workflow_invocation.py Aligns workflow duration recording.
handler.py Passes Meter to invocations.
instruments.py Removes histogram factories.
metrics.py Removes the metrics recorder.
test_handler_metrics.py Updates tool metric assertions.
test_handler_workflow.py Updates workflow attributes.
Review details

Suppressed comments (2)

util/opentelemetry-util-genai/src/opentelemetry/util/genai/metrics.py:1

  • Deleting this module removes InvocationMetricsRecorder, which was exposed via __all__ from a non-private module. Existing consumers importing it will fail immediately; the util compatibility policy requires public objects to be deprecated before removal (.github/instructions/util-genai.instructions.md:36-40). Keep a compatibility module/object with a deprecation note for a release cycle, or defer removal to a breaking release.
    util/opentelemetry-util-genai/src/opentelemetry/util/genai/instruments.py:1
  • These helpers are public imports from a non-private module, and repository release notes explicitly advertised create_duration_histogram and create_token_histogram for reuse (instrumentation/opentelemetry-instrumentation-genai-openai/CHANGELOG.md:114-116). Removing the module breaks downstream imports; retain deprecated compatibility wrappers first or remove them only in a breaking release.
  • Files reviewed: 15/15 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread util/opentelemetry-util-genai/src/opentelemetry/util/genai/_invocation.py Outdated
Comment thread util/opentelemetry-util-genai/.changelog/615.added Outdated
- Align GenericPart with semconv by removing the value field
- Extract non-text Bedrock system content blocks as GenericPart
- Check system is None in Anthropic message extractors
- Retain list[MessagePart] in CompletionHook signature
- Add tests for GenericPart serialization and system instruction span attributes
lmolkova added a commit to lmolkova/opentelemetry-python-genai that referenced this pull request Sep 4, 2026
lmolkova added a commit that referenced this pull request Sep 4, 2026
@lmolkova
lmolkova force-pushed the lmolkova-metrics-refactor branch from 7e63ff9 to 93acaa8 Compare September 4, 2026 05:56
@lmolkova
lmolkova force-pushed the lmolkova-metrics-refactor branch from 93acaa8 to b23f3f7 Compare September 4, 2026 06:09
@lmolkova
lmolkova marked this pull request as ready for review September 4, 2026 06:17
@lmolkova
lmolkova requested a review from a team as a code owner September 4, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants